🔴 P0: --phrase/--near return silent-empty on short words and false positives on trigram-containment (add token-exact verify at the CLI gate)

[dean0x]

Source: Wave-4 skim search dog-food campaign, round 3 (2026-07-02) — a read-only, multi-agent real-world usage sweep of the wave/wave4-search branch. No code was changed. Each finding was adversarially re-reproduced in a fresh cache by an independent verifier and (for confirmed items) root-caused by a code-trace agent. Tracking issue: #174.

Binary under test: skim 2.10.0 @ wave/wave4-search ea99703. All repros use an isolated SKIM_CACHE_DIR and --root; the corpus is this repo unless noted. Ground truth via git grep/rg + file reads.

Two correctness defects in the Phase-2 positional engine that share one recommended fix: make the CLI candidate-then-verify gate the token-exact authority for --phrase/--near (the trigram reader stays a recall-oriented candidate generator, mirroring the #355/#372 lexical architecture). Symptom 1 = any query word <3 bytes makes the whole positional query bail to empty (so --phrase "in the loop", --phrase "at the end", --phrase "fn main" all silently return nothing while git grep finds them). Symptom 2 = each query word matches any doc token containing its trigrams, so adjacent superstring identifiers register as a phrase and rank tied with true literal phrases (--phrase "encode varint" returns 3 files where the adjacent pair exists nowhere).

Symptom: Phrase/near silently returns empty when any query word is < 3 bytes — misses phrases that literally exist ("in the loop")

Verdict: CONFIRMED · Severity: P0 · Confidence: high

Repro

SKIM_CACHE_DIR=$C $BIN search --phrase "in the loop" --root /Users/dean/Sandbox/autobeat; echo exit=$?  (then ground truth: SKIM_PASSTHROUGH=1 git -C /Users/dean/Sandbox/autobeat grep -c "in the loop" README.md → 1)

Expected

README.md returned (it contains 'No human in the loop.'), or at minimum a loud diagnostic that 'in'/'the' cannot be positioned so the phrase query is degenerate (fail-loud is a stated design constraint). The lexical path handles short words via all-files+substring-verify fallback; the positional path could do the same for the short words and align only the >=3-byte ones, or verify the raw phrase string.

Actual

'no results for "in the loop"', exit 0, zero stderr output (same for "no loop" --near 3). Any stopword-containing phrase — extremely common in prose corpora — silently returns nothing while git grep finds it. Trust-breaking missing results with no warning.

Verifier reproduction & evidence

SETUP (fresh isolated cache, this session): BIN=/Users/dean/Sandbox/skim-search/target/release/skim (2.10.0); C=/private/tmp/claude-501/-Users-dean-Sandbox-skim-search/caae0020-fbf7-4356-bae6-a4f3a86b9c89/scratchpad/verify-phrase-near-silently-returns-e-cache. SKIM_CACHE_DIR=$C $BIN search --build --root /Users/dean/Sandbox/autobeat -> "indexed 429 files (5 skipped) in 1.2s".

REPRO (byte-exact): SKIM_CACHE_DIR=$C $BIN search --phrase "in the loop" --root /Users/dean/Sandbox/autobeat -> stdout: no results for "in the loop" ; exit=0 ; stderr=0 bytes. Ground truth verified with Read tool: /Users/dean/Sandbox/autobeat/README.md:8 = "One goal in. Finished work out. No human in the loop." (git grep -c "in the loop" README.md -> 1; dozens more hits repo-wide). SKIM_DEBUG=1 rerun -> still 0 bytes stderr. --json rerun -> {"total":0,"results":[]} with no warning/diagnostic field, exit 0. Silence is total on every surface.

SCOPE BOUNDING (controlled lab: 2-file git repo under scratch; prose.md:3="No human in the loop.", prose.md:5="alpha beta gamma delta", main.rs:1="fn main() {"; indexed in same fresh cache):

• CONTROL --phrase "alpha beta" -> prose.md:5 hit, score 1.00 (phrase machinery works when all words >=3 bytes)
• --phrase "in the loop" -> silent empty (phrase exists in the SAME file the control matched)
• --phrase "human in" -> silent empty (ONE <3-byte word kills the whole query)
• CONTROL "alpha delta" --near 5 -> hit (near path works with long words)
• "no loop" --near 9 -> silent empty although span no->loop = 4 <= 9 in prose.md:3 (unambiguous missing result on the near path)
• --phrase "fn main" -> silent empty although main.rs:1 is fn main() {.
  REAL CODE CORPUS: built /Users/dean/Sandbox/skim-search index (676 files, 6.8s); SKIM_CACHE_DIR=$C $BIN search --phrase "fn main" --root /Users/dean/Sandbox/skim-search -> "no results", exit 0. Ground truth: SKIM_PASSTHROUGH=1 git grep -l "fn main" | wc -l -> 64 files.

ROOT CAUSE (read in source): crates/rskim-search/src/index/reader.rs:768-773 — search_positional() bails return Ok(Vec::new()) when qtokens.iter().any(|t| t.trigrams.is_empty()) (any word <3 bytes); comment: "A <3-byte word cannot be positioned; bail to empty (documented limitation)". Both --phrase and --near share this path (phrase/near branching at lines 811-812 comes after the bail). The CLI prints only the generic message (crates/rskim/src/cmd/search/query.rs:981 no results for {:?}); grep of crates/rskim/src finds NO short-word/positioning warning. README.md and CHANGELOG.md contain zero mentions of "phrase" or "--near" — the "documented limitation" exists only as a source comment, invisible to users.

Root cause

search_positional in crates/rskim-search/src/index/reader.rs:768-773 hard-bails to Ok(Vec::new()) if ANY query word is < 3 bytes: extract_query_positional_tokens (crates/rskim-search/src/ngram.rs:437-443) gives words < 3 bytes an empty trigrams list (they have no within-word trigram to look up token_position from), and the guard qtokens.iter().any(|t| t.trigrams.is_empty()) treats one unpositionable word as fatal for the whole query — silently (no error, no stderr, exit 0; the CLI just prints no results for ... at crates/rskim/src/cmd/search/query.rs:981). The reader-dispatch comment at reader.rs:1052-1054 wrongly assumes short-word queries were already routed to short_query_fallback — that guard (reader.rs:1048) only fires when the WHOLE query text yields zero trigrams; a mixed query like "in the loop" produces cross-word covering trigrams and reaches the positional path, then dies at the bail. Confirmed by controlled experiment: --phrase "human in the loop" (two >=3-byte words) is silent-empty while --phrase "alpha beta" matches and the plain lexical path finds "in the loop" in the same file. This violates the stated fail-loud constraint and loses recall on any stopword-containing phrase.

Code locations

• crates/rskim-search/src/index/reader.rs:768-773 (the silent bail — the defect)
• crates/rskim-search/src/ngram.rs:419-450 (extract_query_positional_tokens: <3-byte word => empty trigrams, token_off preserved)
• crates/rskim-search/src/index/reader.rs:114-138 (count_phrase_alignments: assumes contiguous base+k, must become token_off-gap-aware)
• crates/rskim-search/src/index/reader.rs:140-179 (near_match: operates only on positioned words)
• crates/rskim-search/src/index/reader.rs:1048-1057 (dispatch: ngrams.is_empty() guard does NOT cover mixed short/long phrase queries; positional routing)
• crates/rskim-search/src/index/reader.rs:432-469 (short_query_fallback — existing all-files+verify precedent AD-355-7/AD-372-4 to reuse for all-short-word phrases)
• crates/rskim-search/src/index/builder.rs:169-190 (token_position = word_token_indices ordinal counting ALL words incl. short — proves token_off-gap alignment is exact)
• crates/rskim/src/cmd/search/query.rs:375-428 (CLI positional branch, verify-then-truncate gate)
• crates/rskim/src/cmd/search/snippet.rs:179-181 + crates/rskim-search/src/types.rs query_substring_present (AND-of-tokens verify; phrase queries need a contiguous token-sequence verify)
• crates/rskim/src/cmd/search/query.rs:981 ("no results for" print — the only user-visible surface today)

Fix options

• Option A — Fail-loud only: keep the empty result but detect degenerate tokens in the CLI (before or after engine.search) and emit a loud stderr diagnostic ("phrase word 'in' is <3 bytes and cannot be positioned; positional search cannot answer this query"), plus a warning field in --json.
  • Trade-offs: Satisfies the fail-loud design constraint and restores trust, but recall stays zero — the P0 'misses phrases that literally exist' is not fixed. No perf or index-format impact. Acceptable only as a stopgap.
  • Effort: Small (~20-40 lines + tests)
• Option B (RECOMMENDED) — Skip-unpositionable-words alignment + authoritative verify: (1) in search_positional, drop tokens with empty trigrams from the intersection/alignment set instead of bailing; make count_phrase_alignments gap-aware using token_off deltas (doc token_position and query token_off share the same word_token_indices ordinal, counting short words too, so 'human'@base and 'loop'@base+3 is EXACT contiguity semantics — builder.rs:169-190 proves the coordinate match); near_match runs over positioned words only. (2) If ALL words are unpositionable, reuse short_query_fallback (all-files score-0, existing AD-355-7 precedent) so the CLI verify gates. (3) Strengthen the CLI verify for --phrase from AND-of-substrings to a contiguous word-token-sequence check on the already-read file content (same [A-Za-z0-9_]+ tokenizer) — this is the authoritative ground truth that kills filler-wo …[truncated]
  • Trade-offs: Restores exact phrase recall (verify is ground truth; positional index is just candidate narrowing — same defense-in-depth as the lexical path). NO index-format change: no v5→v6 bump, no rebuild/self-heal churn. Perf: candidate sets slightly larger (intersection over fewer words) but same asymptotics, well within <500ms; all-short fallback is O(file_count) reads with the same measured SLA as the existing 'fn' short-query path (2s/5k files documented). --near semantics for short words is relaxed (presence not proximity) — disclosed loudly on stderr. Moderate blast radius, all in the positional path added this wave.
  • Effort: Medium (~100-150 lines across reader.rs, ngram alignment fns, CLI verify + tests)
• Option C — Index positions for short tokens (format v6): record token_position posting entries for 1-2 byte words (padded keys or a separate short-token dictionary) so short words are first-class positionable.
  • Trade-offs: Exact semantics for both phrase and near including short words. But stopwords ('in','the','of') are the most frequent tokens — posting lists and index size blow up, directly fighting the v5 size-guard work from Wave 4 search follow-ups (#374–#381) #386; requires v5→v6 format bump forcing a full rebuild for every user via self-heal; build-time and query-time perf risk. Not justified when Option B achieves exact phrase semantics without any of it.
  • Effort: Large (builder + format + reader + version bump + self-heal + size guards)

Recommended fix (incl. test coverage)

Option B. It fixes the actual P0 (recall), not just the silence: phrase results become exact because the CLI verify gate — strengthened to a contiguous word-token-sequence check on file content the CLI already reads for snippets — is the authoritative predicate, with the positional index reduced to candidate narrowing (identical defense-in-depth architecture to the existing lexical AND-intersect→verify→truncate-last design, AD-355/AD-372). The key enabler is already in the data model: doc-side token_position (builder.rs:182) and query-side token_off (ngram.rs:435) are the SAME word-ordinal coordinate counting short words, so skipping unpositionable words while aligning on token_off gaps preserves exact contiguity distances — only count_phrase_alignments needs to consume (token_off, positions) pairs instead of assuming base+k. It requires no index-format bump (v5 unchanged, no rebuild for users), reuses the short_query_fallback precedent for the all-short degenerate case, and folds in Option A's stderr notice for the one place semantics genuinely relax (--near proximity of short words). Accompanying tests: unit — gap-aware count_phrase_alignments (offsets 0,3 matches human..loop with 2 fillers; rejects wrong gap); reader_tests — mixed short/long phrase returns candidate, all-short phrase falls back to full candidate set, near with short word returns candidates; CLI query_tests — --phrase "in the loop" over a fixture containing 'No human in the loop.' returns the file (text + --json), --phrase "human in the loop" over a doc containing 'human za zb loop' returns EMPTY (verify gate kills the positional false positive), "no loop" --near 3 returns the file plus asserts the stderr proximity notice, regression controls --phrase "alpha beta" and all-long --near unchanged, and exit codes stay 0/1 per contract.

---

Symptom: Phrase word matching is trigram-containment (substring-like), returning files/lines with no token-exact phrase and ranking them equal to true phrase hits

Verdict: CONFIRMED · Severity: P1 · Confidence: high

Repro

SKIM_CACHE_DIR=$C $BIN search --phrase "encode varint" --json --limit 500 --root /Users/dean/Sandbox/skim-search → 3 files, while SKIM_PASSTHROUGH=1 git grep -lE "encode[^A-Za-z0-9_]+varint" → NOTHING (matches are 'encode_varint / decode_varint' format_tests.rs:605, 'encode_header, encode_postings_varint' builder.rs:17, 'encode_postings_varint / decode_postings_varint' reader_tests.rs:1125). Also the smoke query: --phrase "token position" returns ngram_tests.rs + format.rs (no token-exact pair) at score 3.00, TIED with reader.rs which has the real literal phrase.

Expected

--phrase billed as 'exact token adjacency' should require the doc token to BE the query word (or at least rank exact-token phrases above containment matches). A strict phrase engine returns 0 for 'encode varint' here.

Actual

A query word matches ANY doc token containing all its trigrams, so adjacent long identifiers ('tokens extract_query_positional_tokens', 'eval_type, loops' in autobeat RELEASE_NOTES_v1.3.0.md table) produce phrase hits; reversed probe 'position token' returned 7 files vs 3 with true token adjacency. Upside: plural/stem tolerance ('token positions', 'encoded varint') is sometimes wanted — but there is no exactness tier in scoring and no way to opt out, and the CLI verify gate cannot catch it.

Verifier reproduction & evidence

Setup (fresh isolated cache, index rebuilt this session): BIN=/Users/dean/Sandbox/skim-search/target/release/skim; C=/private/tmp/claude-501/-Users-dean-Sandbox-skim-search/caae0020-fbf7-4356-bae6-a4f3a86b9c89/scratchpad/verify-phrase-word-matching-is-trigra-cache; SKIM_CACHE_DIR=$C $BIN search --build --root /Users/dean/Sandbox/skim-search -> "indexed 676 files ... in 6.9s".

(1) REPRO, exact: SKIM_CACHE_DIR=$C $BIN search --phrase "encode varint" --json --limit 500 --root /Users/dean/Sandbox/skim-search -> total=3 (format_tests.rs score 3.0, reader_tests.rs 2.0, builder.rs 1.0; 24ms). Ground truth: SKIM_PASSTHROUGH=1 git grep -lE "encode[^A-Za-z0-9_]+varint" -> no output, exit=1 — ZERO true whitespace/punct-separated "encode ... varint" phrases exist in the repo. All 3 hits are adjacent fused identifiers verified in-file: builder.rs:17 "encode_header, encode_postings_varint, lang_to_id," and format_tests.rs:605 "// v4 varint codec (encode_varint / decode_varint)". A strict phrase engine (and grep/rg) returns 0 here; skim answers "3 files contain this phrase".

(2) RANKING TIE, reproduced and stronger than claimed: SKIM_CACHE_DIR=$C $BIN search --phrase "token position" --json --limit 500 --root /Users/dean/Sandbox/skim-search -> 5 files: reader.rs 3.0, ngram_tests.rs 3.0, format.rs 2.0, cmd/rewrite/mod.rs 2.0, cmd/security.rs 1.0. Ground truth: SKIM_PASSTHROUGH=1 git grep -nE "token[^A-Za-z0-9_]+position" -> exactly ONE site in the entire repo (crates/rskim-search/src/index/reader.rs:115 "doc token positions"). So 4 of 5 results have no token-exact phrase, and containment-only ngram_tests.rs (3x "let tokens = extract_query_positional_tokens(" at lines 523/565/582 — token "tokens" adjacent to a token containing "positional") TIES the one true-phrase file at 3.0. No exactness tier in scoring.

(3) CONTROLLED LAB (different corpus, md prose; bounds scope): 3-file repo at .../verify-phrase-word-matching-is-trigra-lab: exact.md "the alpha beta protocol", contain.md "the alphabet betamax recorder", fused.md "the alpha_beta identifier". After --build (3 files indexed): SKIM_CACHE_DIR=$C $BIN search --phrase "alpha beta" --json --root $LAB -> contain.md score=1.0, exact.md score=1.0 — the containment false positive ("alphabet betamax", which contains neither word) scores IDENTICAL to the true phrase and sorts FIRST. fused.md correctly absent (token-level adjacency is genuinely enforced; a lone fused identifier does not phrase-match, matching the finder's quokka_wombat observation). SKIM_CACHE_DIR=$C $BIN search "alpha beta" --near 1 --json --root $LAB -> all three files at 1.0, i.e. the same containment D_k feeds --near too.

(4) NOT BY-DESIGN: not on the known/by-design list. Source confirms mechanism: /Users/dean/Sandbox/skim-search/crates/rskim-search/src/index/reader.rs:835 comment "Per query word: D_k = ∩ over its trigrams of {token_position at doc}" with no token-boundary/length/equality constraint (lines 841-876); count_phrase_alignments (reader …[truncated]

Root cause

Phrase/near matching never verifies token identity — only trigram containment plus ordinal adjacency. In NgramIndexReader::search_positional (crates/rskim-search/src/index/reader.rs:841-875), each query word k is reduced to its within-word trigrams (extract_query_positional_tokens, ngram.rs:419-450 — the word's text is discarded), and the word's doc-position set D_k is computed as the intersection, over those trigrams, of the token_position values in the doc's postings. Because the builder assigns every trigram inside a fused identifier the same word ordinal (builder.rs:182-191), any doc token that merely CONTAINS all of a query word's trigrams (e.g. encode_header ⊇ trigrams of "encode", encode_postings_varint ⊇ trigrams of "varint") places its ordinal into D_k. count_phrase_alignments (reader.rs:117-138) then only checks ordinal adjacency, so two adjacent superstring identifiers register as a phrase hit. Scoring (reader.rs:881-891) is the bare alignment count with no exactness tier, so containment hits tie with true literal phrase hits. The CLI verify gate cannot catch it because the positional path (cmd/search/query.rs:379-397) feeds results into the same query_substring_present predicate (snippet.rs:179-181 → rskim-search/src/types.rs:660), which is AND-of-substrings-anywhere-in-file — phrase-blind by design. Confirmed by controlled experiment: a 3-file corpus where use crate::{encode_header, encode_postings_varint, ...} scores 1.0 on --phrase "encode varint", tied with the file containing the literal "encode varint"; a lone fused encode_varint token correctly does NOT match (both words map to one ordinal), which pins the mechanism precisely to adjacent-superstring-token containment.

Code locations

• crates/rskim-search/src/index/reader.rs:841-875 (search_positional D_k = trigram-containment intersection over token_position — the root cause; no token-equality check)
• crates/rskim-search/src/index/reader.rs:117-138 (count_phrase_alignments — ordinal adjacency only)
• crates/rskim-search/src/index/reader.rs:881-891 (score = alignment count; no exactness tier)
• crates/rskim-search/src/ngram.rs:419-450 (extract_query_positional_tokens — query word text discarded, only within-word trigrams kept)
• crates/rskim-search/src/index/builder.rs:182-191 (token_position = word ordinal of trigram start byte; all trigrams of a fused identifier share one ordinal)
• crates/rskim/src/cmd/search/query.rs:379-397 and 417-428 (positional path routes results to the generic substring verify gate)
• crates/rskim/src/cmd/search/snippet.rs:179-181 and crates/rskim-search/src/types.rs:660 (query_substring_present — AND-of-substrings, phrase-blind verify predicate)
• crates/rskim-search/src/lexical/tokenize.rs:21-45 (word_token_indices — canonical word-token semantics the verifier should reuse)

Fix options

• OPTION A (verify-gate fix): Make the CLI verify gate phrase/near-aware and token-exact. Add pure predicates in rskim-search/src/types.rs next to query_substring_present — phrase_tokens_present(content, query) and near_tokens_present(content, query, n) — that tokenize content with the SAME word_token_indices semantics ([A-Za-z0-9_]+ runs) and require each doc token to EQUAL the query word (case-sensitive) at consecutive ordinals (phrase) or within n ordinals (near). Thread the phrase/near mode through SnippetVerifyParams into extract_snippet_and_verify so the positional path uses the new predicate instead of query_substring_present; re-anchor the snippet to the first verified phrase occurrence found during the scan (fixes snippets currently pointing at containment trigram sites). Reader stays a recall-oriented candidate generator — false candidates are simply dropped, so 'encode varint' r …[truncated]
  • Trade-offs: Correctness: fully strict (grep-parity via the same ordinal-adjacency semantics the verifier's ground-truth regex uses). Perf: zero new I/O — the candidate file is already read once at the verify gate; an O(n) token scan replaces the O(n) substring scan, well within <500ms query / <50ms targets; large files keep the existing MAX_VERIFY_SCAN_BYTES bounded-read tradeoff. Index-format: NO v5 bump, NO rebuild/self-heal. Blast radius: CLI verify path only; library callers of reader.search() still get containment candidates (documented limitation, same as the existing exact-symbol path which also relies on the CLI gate per AD-355-1). Loses the plural/stem tolerance upside by default — could later …[truncated]
  • Effort: Small-medium: ~2 pure functions + tests in rskim-search/types.rs, plumbing in cmd/search/query.rs + snippet.rs; no reader/builder/format changes.
• OPTION B (engine fix, format v6): Enforce token-exactness inside search_positional. Store token_len (varint) per posting (format v5→v6 bump with the existing self-heal rebuild pattern), then a doc token matches query word w (len L) iff token_len == L AND w's trigrams appear at consecutive byte offsets within that token (byte position is already in postings, decoded in the reader loop). Consecutive-run + equal-length ⇒ token equality with no file I/O.
  • Trade-offs: Correctness at the engine layer — benefits library users and removes reliance on the CLI gate; no per-candidate file read needed for exactness. Costs: index grows (~1 varint/posting over hundreds of MB-scale corpora), hot varint decode path gets more work, format bump forces a full rebuild on every existing v5 index (self-heal exists, but it is a one-time multi-second cost per root), and the change touches the highest-risk code (builder/codec/reader) right after the v4→v5 migration. Snippet mis-anchoring and the phrase-blind verify predicate still need the Option A snippet/near plumbing anyway for --near over stale-guarded files.
  • Effort: Medium-large: builder + format codec + reader changes, v6 version bump, staleness/self-heal wiring, migration tests.
• OPTION C (ranking tier, keep fuzziness): Keep containment matching for recall but add an exactness tier: in-reader, use the existing byte position field to detect whether the query word appears as a CONTIGUOUS byte run inside the doc token (consecutive positions p0..p0+L-3), scoring contiguous-substring alignments above scattered-trigram ones, and let the verify gate re-rank verified-literal-phrase files above the rest. Optionally expose --phrase-exact / --phrase-loose.
  • Trade-offs: Preserves the sometimes-wanted stem/plural tolerance ('token positions', 'encoded varint') and fixes the tie, but does NOT satisfy the billed contract — 'encode varint' would still return 3 files instead of 0, contradicting 'exact token adjacency' and the P1 report's expected behavior. Contiguous-byte check alone still can't prove token equality (substring 'varint' inside 'encode_varint' passes), so full exactness still needs Option A or B. Adds scoring-semantics complexity users must learn.
  • Effort: Small-medium in-reader change + verify re-rank; no format bump.

Recommended fix (incl. test coverage)

OPTION A — strict token-exact phrase/near verification at the CLI verify gate. Rationale: (1) it matches the architecture the codebase deliberately built across #355/#372/#392 — the trigram reader is a recall-oriented candidate generator and the CLI candidate-then-verify-then-truncate-LAST gate is the single correctness authority (AD-355-1/AD-355-2/AD-372-3); phrase verification is the missing member of that family, not a new pattern. (2) Zero index-format impact: no v6 bump, no self-heal rebuild, no builder/codec churn immediately after the risky v4→v5 migration — smallest blast radius of the three. (3) Zero perf cost against the <50ms/<500ms targets: each surviving candidate file is already read exactly once at the gate; a word-token scan is the same O(n) as the current substring scan, and the reader's containment intersection keeps candidate sets tiny (3-7 files in the repro). (4) It fixes both reported symptoms at once: false hits are dropped ('encode varint' → 0, grep parity) and the ranking tie disappears because non-exact files never reach output. Implement the predicates in rskim-search/src/types.rs beside query_substring_present so the rskim-bench harness measures the same verified surface (AD-355-1 parity requirement), reuse word_token_indices semantics for ordinal distance so --near N means the same thing at index and verify time, and re-anchor snippets to the first verified occurrence. If loose/stemmed phrase matching is genuinely wanted later, add an explicit --phrase-loose flag as a follow-up ticket rather than keeping it as silent default behavior. Test coverage to accompany the fix: unit tests for phrase_tokens_present/near_tokens_present (positive: 'token position', 'token(position)', 'token::position'; negative: fused single token 'token_position', adjacent superstrings 'encode_header, encode_postings_varint', plural superstring 'positions', case mismatch; near-N ordinal distance incl. N=0; words <3 bytes path unchanged); gate-level tests that --phrase 'encode varint' on a fused-identifier fixture returns 0 while the literal fixture returns 1 with the snippet anchored on the literal line; a reversed-order probe (phrase 'position token' ⊂ near results); the large-file MAX_VERIFY_SCAN_BYTES bounded-verify path with a phrase query; and a bench-parity assertion that rskim-bench filters positional output with the identical predicate. Note these tests must run against the built binary/integration harness carefully — the rskim-search lib test binary is known to hang at startup on this machine (see memory), so prefer predicate unit tests in rskim-search plus CLI integration tests in rskim --bins.

---

Note: These are filed together because the recommended fix for both is the same — strict token-exact phrase/near verification at the CLI verify gate (reusing word_token_indices semantics), with no index-format change. Fixing one without the other leaves the family half-broken.

[dean0x]

Resolved on wave/wave4-search (merge 0a71892), not yet on main — ships via PR #368. Verified: all four dynamic-build gates passed, zero surviving findings, and the branch build gate is green (fmt, build, clippy --all-targets -D warnings on both crates, and 1011 rskim-search unit tests passing).