Skip to content

PCAP import always fails: importer INSERT is rejected by the node's read-only SQL validator (11.0.336) #993

Closed

Description

@seriouslysahid

Summary

POST /api/v4/imports/pcap always fails with a 400. The importer builds an
INSERT, sends it to the storage node's HTTP /query endpoint, and that
endpoint rejects any statement that is not SELECT/WITH/SHOW/DESCRIBE/EXPLAIN/PRAGMA.
The PCAP import feature therefore cannot work in any configuration.

Reproduced on 11.0.336 (ghcr.io/sipcapture/homer:latest, commit 93fdb78),
all-in-one deployment with local DuckLake storage (sqlite catalog, no S3).

Reproduction

curl -X POST http://localhost:8080/api/v4/imports/pcap \
  -H "Authorization: Bearer $TOKEN" \
  -F "file=@capture.pcap" \
  -F "override_to_current_time=true"

Response:

{"error":{"type":"https://api.homer/errors/invalid_argument","title":"Bad Request","status":400,
"detail":"query HTTP error: status=400 body=\"{\\\"success\\\":false,\\\"error\\\":\\\"SQL validation failed: query must start with SELECT, WITH, SHOW, DESCRIBE, EXPLAIN, or PRAGMA (got \\\\\\\"INSERT\\\\\\\")\\\"}\\n\""}}

The same happens from the UI import dialog.

Cause

  1. importPcapSIP builds an INSERT via BuildInsertMultiValues and executes it
    with flight.QueryFirstConnected — src/coordinator/handlers/pcap_sip_import.go:167-172.
  2. QueryFirstConnected is documented as "Intended for INSERT/DDL that must not be
    duplicated across a multi-node cluster"
    — src/coordinator/services/flight_service.go:283-285.
  3. It delegates to queryNode, which is HTTP-only and always posts to the node's
    /query endpoint on FlightServer.Port + 1 — src/coordinator/services/flight_service.go:445-453.
    There is no gRPC/Airport write path to fall back to.
  4. Node.handleQuery calls sqlvalidator.ValidateRawSQL(req.SQL) unconditionally,
    before any branching — src/node/node.go:1215. allowedStatementStarts permits
    only SELECT/WITH/SHOW/DESCRIBE/EXPLAIN/PRAGMA, so INSERT is rejected with 400
    (src/coordinator/sqlvalidator/validator.go:276).

The validator was added for GHSA-rm5w-rqr7-2h54 (per the comment on node.go:1214).
It appears the import path was not accounted for.

Supporting evidence that DML was intended on this endpoint: immediately after the
validator, handleQuery still contains a non-SELECT branch that calls
db.ExecContext(...) (src/node/node.go:1230). That branch is now unreachable —
nothing can get past ValidateRawSQL and still be a non-SELECT statement.

Impact

  • PCAP import is non-functional for all users on this release, via API and UI.
  • No workaround via configuration: there is no allow_write / read_only toggle
    for the node, and queryNode has a single hard-coded HTTP transport.
  • Normal HEP ingest is unaffected — the writer holds its own DuckLake connection
    and does not go through the node's /query endpoint.

Suggested fix

Give internal writes a path that bypasses the raw-SQL validator, rather than
relaxing the validator itself. For example an authenticated internal write
endpoint/RPC used by QueryFirstConnected, keeping /query read-only for
user-supplied SQL. The dead ExecContext branch in handleQuery should then be
removed or moved to that path.

Environment

Version 11.0.336 (built 2026-08-28, commit 93fdb78, go1.27.0, linux/amd64)
Image ghcr.io/sipcapture/homer:latest
Deployment all-in-one (ingest + storage + node + coordinator)
Storage local DuckLake, sqlite catalog, no object storage

Activity

  1. self-assigned this
    on Sep 4, 2026
  2. adubovikov commented on Sep 4, 2026

    @adubovikov
    Member

    Thanks for the excellent write-up — the diagnosis was correct.

    POST /query is intentionally read-only (GHSA-rm5w-rqr7-2h54), so the PCAP importer could never land an INSERT there. The fix keeps /query as-is and sends coordinator writes through a separate authenticated POST /exec. That path only accepts:

    INSERT INTO <catalog>.main.hep_proto_<proto>_<name> (...) VALUES ...

    SELECT/FROM, system catalogs, quoted identifiers, and stacked statements are still rejected.

    Shipped in 11.0.337 via #994. After upgrading, POST /api/v4/imports/pcap (and the UI import dialog) should work again.

  3. added a commit that references this issue on Sep 8, 2026
    3048019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions