Wave 4 / Search: query latency floor — full posting-blob CRC32 on every index open (reader.rs)

[dean0x]

Wave: wave4-search-followups
Depends on: #380

flowchart LR
  T372["1. #372 lexical AND-intersect"]
  T373["2. #373 FileId skew"]
  T374["3. #374 AST verify gate"]
  T375["4. #375 remove index positional"]
  T377["5. #377 weights notice"]
  T380["6. #380 binarize + positional"]
  T376["7. #376 CRC marker"]
  T379["8. #379 worktree staleness"]
  T378["9. #378 risk volume weighting"]
  T381["10. #381 index-location docs"]
  T372 --> T373 --> T374 --> T375 --> T377 --> T380 --> T376 --> T379 --> T378 --> T381
  T373 -. requires .-> T374
  T373 -. requires .-> T380
  T380 -. requires .-> T376
  T375 -. requires .-> T377
  classDef cur fill:#dcffe4,stroke:#1a7f37,stroke-width:3px;
  class T376 cur;

Loading

Solid arrows = integration order into wave/wave4-search-followups. Dotted = hard dependency. One branch per ticket; this ticket highlighted.

Found via dog-fooding round 2 (2026-06-27, branch wave/wave4-search), tracked under #174 Wave 4. Priority: P1.

Problem: Query latency is median 57 ms / p90 77 ms (only 2/36 samples under the <50 ms target). The floor is a fixed per-open cost independent of result count.

Evidence: latency scales with .skpost size across corpora (townhall 9 ms / devflow 15 ms / skim-search 48 ms). Literal trigram: skim 70 ms wall vs ripgrep <10 ms, git grep 30 ms.

Root cause: crates/rskim-search/src/index/reader.rs:130-132 (added by #364) CRC32-hashes the entire post_mmap (40.7 MB here) on every open(), before any query runs.

Best fix: stop re-hashing the whole blob per query. Options: (a) verify CRC once at build / first-open and persist a validity marker keyed on (path, mtime, size); (b) segment the checksum and verify only segments actually touched; (c) make full verification opt-in (--verify-integrity) and rely on mmap otherwise. Restores <50 ms.

Refs: #358, #364, #174.

[dean0x]

Plan + resolved decisions (dynamic-plan, 2026-06-28)

Query latency floor eliminated via validity-marker caching: after one verified open, sidecar index.skverify records (len, mtime-ns, header.checksum); subsequent opens skip full CRC if marker proves byte-identity to prior verification, else full verify + rewrite marker. Preserves ADR-006 desync guard while moving it off per-query hot path.

Resolved design decisions

• Validity-marker caching approach (AD-376-1): Skip full-payload CRC32 when persisted validity marker proves both index files are byte-identical to a previously verified state (path+len+mtime-ns+checksum). Restores sub-50ms query floor by moving one-shot integrity check off per-query hot path. CRC still runs on any marker miss, preserving ADR-006 desync/mis-rank guard. Refs Wave 4 / Search: query latency floor — full posting-blob CRC32 on every index open (reader.rs) #376; applies ADR-006, ADR-001.

• Marker trust boundary (AD-376-2, THREAT-MODEL RESOLVED to ACCEPT): Validity keyed on (idx_len, idx_mtime_ns, post_len, post_mtime_ns) AND verified header.checksum. mtime+len detects any rewrite; carrying the checksum means marker can never validate file with different header.checksum. Marker compares header's checksum FIELD (cheap to read), NOT a recompute. TRUST BOUNDARY (accepted, ADR-006): marker's purpose is FileId consistency across opens, NOT corruption detection; the full CRC remains the desync integrity guard and the marker only caches a prior successful verification. KNOWN LIMIT: a content byte-flip that simultaneously preserves len AND mtime AND header.checksum field is served unverified (silent mis-rank); this is rare and explicitly out of scope. A content-SHA sidecar would be a separate follow-up if ever justified. AC1 pins this exact skip (PF-007), so the boundary is measurable, not implicit. Refs Wave 4 / Search: query latency floor — full posting-blob CRC32 on every index open (reader.rs) #376; applies ADR-006.

• Marker on-disk format (AD-376-2): Fixed little-endian binary record. Marker is read on hot path of every query (lexical and --ast); binary record is trivially cheap and adds no dependency. Human-inspectability does not justify hot-path parse cost. Refs Wave 4 / Search: query latency floor — full posting-blob CRC32 on every index open (reader.rs) #376.

• mtime source portability (AD-376-2): STD-ONLY in production (Metadata::modified()); filetime as DEV-ONLY dependency for AC1 test (set_file_mtime). Standard engineering judgment; filetime is the known tool for mtime manipulation in tests. Refs Wave 4 / Search: query latency floor — full posting-blob CRC32 on every index open (reader.rs) #376.

• AST index reader scope (AD-376-5, RESOLVED to INCLUDE): Identical full-payload CRC32 in ast_index/store/reader.rs::open sits on --ast query hot path and has identical fixed-cost defect. The dual lexical+AST index is one coherent unit, so desync guard must cover BOTH readers. Extend validity-marker fix to AST reader IN THIS TICKET (AC11 option a); same marker mechanism with AC1/AC2/AC5/AC6 analogues passing. Sibling ticket (co-change reader, ADR-004) with referencing code comment at cochange/reader.rs:71. Refs Wave 4 / Search: query latency floor — full posting-blob CRC32 on every index open (reader.rs) #376; applies ADR-006, ADR-001.

• Rebuild invalidation (AD-376-4): On --rebuild/--force AND incremental/staleness rewrites, new files get fresh mtime/len/checksum so surviving marker auto-invalidates; also unlink index.skverify defensively so partial/aborted rebuild cannot leave marker validating wrong bytes. Refs Wave 4 / Search: query latency floor — full posting-blob CRC32 on every index open (reader.rs) #376; applies ADR-006.

• Build-time stamp (AD-376-3): After atomic_write of both index files + verify-back NgramIndexReader::open, stamp index.skverify for freshly written files so first post-build query skips redundant full CRC (bytes verified microseconds ago in verify-back open). Refs Wave 4 / Search: query latency floor — full posting-blob CRC32 on every index open (reader.rs) #376.

### Acceptance criteria

AC1 (FUNCTIONALITY + TRUST-BOUNDARY PIN, fast-path skip)
Given a freshly built lexical index and successful first NgramIndexReader::open(dir) (stamps index.skverify), when a single byte of index.skpost is corrupted WITHOUT changing byte-length AND WITH mtime held identical to marker AND WITHOUT changing header.checksum field, then second NgramIndexReader::open(dir) MUST return Ok(_) (full CRC32 skipped). This fails if optimization deleted. Explicit, falsifiable pin of accepted (len, mtime, header.checksum) trust boundary (ADR-006, PF-007).

AC2 (FUNCTIONALITY/INTEGRITY, marker-miss re-verify)
Given a built index, when index.skpost is corrupted AND mtime bumped (OR index.skverify deleted/absent), then NgramIndexReader::open(dir) MUST return Err(SearchError::IndexCorrupted(_)) with exact substring "checksum mismatch". Pre-existing test_open_corrupt_skpost_detected MUST pass verbatim.

AC3 (NEGATIVE, no query-result regression)
Given a built index over known fixture, when same query run twice (cold open then marker-fast open) via binary dispatch path, then ranked result set (file order, scores) MUST be byte-identical between runs AND identical to result from build at HEAD without this change.

AC4 (NEGATIVE, on-disk format compatibility)
Change MUST NOT bump FORMAT_VERSION (stays 4), alter SkidxHeader layout (SKIDX_HEADER_SIZE stays 62), or encoded bytes of index.skidx/skpost. Index pair written by current-HEAD MUST open under new code; pair written by new code MUST open under current-HEAD code (.skverify sidecar purely additive, ignored by older binaries).

AC5 (FUNCTIONALITY, rebuild/rewrite invalidation)
Given a built index with valid index.skverify, when index rebuilt with DIFFERENT content (via --rebuild/--force or incremental self-heal), then (a) new content MUST open successfully and return results reflecting NEW content, AND (b) stale marker MUST NOT cause new bytes served unverified (corrupting post-rebuild .skpost MUST catch with "checksum mismatch").

AC6 (FUNCTIONALITY, derived-cache robustness)
Given uncorrupted index, when index.skverify is truncated, garbage, zero-length, or unreadable, then NgramIndexReader::open(dir) MUST return Ok(_) (fall through to full CRC) and MUST NOT error attributable to bad marker. Failed marker WRITE during open MUST NOT fail open() (best-effort; next open re-verifies).

AC7 (API CONTRACT, signatures unchanged)
Public signatures NgramIndexReader::open(dir: &Path) -> Result and open_with_config(dir: &Path, config: BM25FConfig) -> Result MUST be unchanged (no new params, no new error variant). No new CLI flag or subcommand.

AC8 (FUNCTIONALITY, build-time stamp)
Given fresh NgramIndexBuilder::build() over any corpus (including zero-file), then index.skverify MUST exist immediately after build() returns, AND immediately-following NgramIndexReader::open(dir) of uncorrupted index MUST return Ok(_).

AC9 (SECURITY/PERMS, sidecar)
index.skverify MUST be written via crate::io_util::atomic_write so on Unix it carries 0o600 permissions and leaves NO temp/partial file on success or mid-write failure.

AC10 (PERFORMANCE, explicitly NO wall-clock threshold per ADR-003)
MUST NOT have brittle numeric latency assertion (e.g., 'open() < 50ms') in test suite. Grounded performance guard is AC1 (full-blob CRC provably skipped on fast path) -- the deterministic mechanism behind latency win and regression sentinel.

AC11 (FUNCTIONALITY -- AST reader floor, RESOLVED to option (a), in-scope)
Identical full-payload CRC32 in ast_index/store/reader.rs::open MUST be moved off per-open hot path by SAME marker mechanism (tag AD-376-5), with AC1/AC2/AC5/AC6 analogues passing for AST reader (fast-path skip on flipped-but-(len+mtime+header.checksum)-held byte; corruption caught on mtime-bumped or marker-absent miss; rebuild invalidation; bad-marker fall-through). Dual lexical+AST index is one coherent unit (ADR-006), so build MUST NOT leave AST per-open CRC as known residual floor. Co-change reader (cochange/reader.rs::open) is OUT of scope, covered by real sibling ticket filed up-front (ADR-004, no #NEW); code comment at co-change CRC site MUST reference that ticket.

Build context

Wave-4 search follow-ups. Sequence position 7 of 10. Depends on: #380 (manifest format locked first). AD-376-N rustdoc at decision sites (ADR-004); falsifiable ACs (PF-007); cites ADR-006.

[dean0x]

Resolved on wave/wave4-search via PR #386 (Wave-4 dog-food round-2). Residual partial-scope was ticketed as #388–#391 + #392 and resolved. Not yet on main — ships with PR #368 (wave→main).