Phase 2 (#380): positional phrase / --near search

[dean0x]

Context

Phase 1 of #380 delivered the lexical positional index (v4: PostingEntry { doc_id, field_id, position }, delta+varint codec). Phase 2 adds phrase (contiguous multi-term) and --near (proximity) query support.

Design: .devflow/docs/adr-380-phase2-positional-search.md (ADR-380-P2).

Pivotal finding (dual-sourced research)

The v4 position is a document-absolute BYTE offset with zero tokenization, emitted for every trigram window (no dedup / cap / drop), one entry per (trigram, doc, position), and already decoded in the query hot path (reader.rs:461-462, 518, 536). This splits the scope cleanly:

• Exact phrase + byte/char-distance --near → v4 query-only (no format bump, no rebuild, no size-guard change).
• Word/token-distance --near N → v5 re-index (indexer must tokenize + store token positions; FORMAT_VERSION 4→5; automatic cold rebuild; size guard re-measured).

Scope decision: PENDING sign-off

The v4-query-only vs v5 choice is a product/risk decision — see the ADR. Implementation is gated on that sign-off.

Acceptance criteria

• AC-P2-1 (positive): a quoted phrase "alpha beta" returns files where those terms appear contiguously (byte-adjacent), ordered.
• AC-P2-2 (negative, falsifiable): the same query must NOT return files where the terms appear only non-adjacently or reversed — must fail if reverted to bag-of-trigrams. (--near N accepts within-window pairs, rejects beyond-window.)
• AC-P2-3 (no regression): non-phrase / non-near queries return byte-identical results to Phase 1; if v5, a v4 index cold-rebuilds under the v5 binary and serves correctly.
• AC-P2-4 (perf/size): phrase/near query latency < ~50 ms on the corpus-class benchmark; index size re-measured only if v5.

Implementation notes (from research)

• New dispatch branch modeled on search_exact_intersection (reader.rs:499-661); the positional filter runs AFTER trigram intersection and BEFORE the truncation gate (verify-then-truncate-last, AD-355-2 / AD-372-4).
• New offset-preserving query extractor — current extract_query_ngrams (ngram.rs:353) discards per-trigram query offsets.
• Size guard: crates/rskim-search/src/index/reader_tests.rs:1103 (LEXICAL_SIZE_RATIO_CEILING = 5.0) — corrected ref (the AST guard < 2.2x is separately at ast_index/store/reader_tests.rs:550).
• Atomic-save ordering (if v5): crates/rskim/src/cmd/search/index.rs:380-436 (ADR-006) — corrected ref (there is no crates/rskim-search/src/index.rs).
• UTF-8 caveat: byte-distance --near != codepoint-distance on multi-byte text.

Refs #380, #174. Blocks the merge of PR #386 per the wave-4 close-out plan.

[dean0x]

Phase 2 implemented in PR #386 — CI-green.

• a009a2c — lexical format v5: PostingEntry.token_position (word-token ordinal of a trigram's first byte) + a 4th tail varint in the posting codec; automatic v4→v5 cold-rebuild via the existing v < LEXICAL_INDEX_FORMAT_VERSION self-heal (no --rebuild needed).
• 6ba3167 — --phrase / --near N query: search_positional AND-intersects within-word trigram posting lists, then filters by whole-word token_position (phrase = contiguous & ordered; --near = within-N word-tokens, unordered). CLI flags + pure-logic unit tests.
• 83cbb4a — AC-P2-1..4 end-to-end acceptance tests + v4→v5 self-heal test + v5 size-guard re-baseline comment.

AC-P2-1..4 pass on CI (Linux): phrase matches contiguous-ordered and rejects reversed / non-adjacent; --near N respects the word-token window; plain queries are byte-unchanged (no regression); a v4 index cold-rebuilds under the v5 binary. Snyk clean; size ratio stays under the 5.0× guard.

Design note: uses within-word trigrams only (excludes cross-word/border trigrams) so --near isn't forced into adjacency; the query token_off reuses the indexer's word_token_indices for query/doc symmetry.

Resolved on wave/wave4-search-followups (→ wave/wave4-search).

[dean0x]

Resolved in PR #386 (merged to wave/wave4-search @ ea99703): targeted test/perf hardening + #380 Phase 2 positional search. Not yet on main — ships with PR #368 (wave→main).