Wave 4 / Search: standalone --ast resolves matches to the WRONG files (FileId→path ordering skew)

[dean0x]

Wave: wave4-search-followups

flowchart LR
  T372["1. #372 lexical AND-intersect"]
  T373["2. #373 FileId skew"]
  T374["3. #374 AST verify gate"]
  T375["4. #375 remove index positional"]
  T377["5. #377 weights notice"]
  T380["6. #380 binarize + positional"]
  T376["7. #376 CRC marker"]
  T379["8. #379 worktree staleness"]
  T378["9. #378 risk volume weighting"]
  T381["10. #381 index-location docs"]
  T372 --> T373 --> T374 --> T375 --> T377 --> T380 --> T376 --> T379 --> T378 --> T381
  T373 -. requires .-> T374
  T373 -. requires .-> T380
  T380 -. requires .-> T376
  T375 -. requires .-> T377
  classDef cur fill:#dcffe4,stroke:#1a7f37,stroke-width:3px;
  class T373 cur;

Loading

Solid arrows = integration order into wave/wave4-search-followups. Dotted = hard dependency. One branch per ticket; this ticket highlighted.

Found via dog-fooding round 2 (2026-06-27, branch wave/wave4-search), tracked under #174 Wave 4. Priority: P0.

Problem: Standalone --ast prints the wrong file paths for matches — results include files that cannot contain the pattern. Acting on these sends an agent to the wrong file (worse than an empty result).

Evidence:

$ skim search --ast "for_expression > block" --limit 40
   ... crates/rskim-contract/Cargo.toml
   ... crates/rskim-contract/clippy.toml
   ... crates/rskim-core/README.md      # none can contain a Rust for-loop

Reproduced decisively in an isolated 2-file repo: function_item > block returned foo.rs (no function) instead of foo/bar.rs (the only file with a function).

Root cause: FileIds are assigned in PathBuf order at index time (walk.rs:428) but resolved through a String/BTreeMap-ordered path list (manifest.sorted_paths) at query time. Path-separator vs string-collation ordering differ, so the Nth match resolves to a different file's path.

Best fix: make FileId assignment and path resolution use the same canonical ordering (pick byte-wise string order, apply on both sides). Add a round-trip test asserting resolve(assign(path)) == path over a corpus with nested directories.

Refs: #174.

[dean0x]

Plan + resolved decisions (dynamic-plan, 2026-06-28)

Planned via /devflow:dynamic-plan; decisions resolved at the command boundary. Full plan + test plan in the local gitignored design dir; this comment is the durable record.

Resolved design decisions

• Force-rebuild on upgrade = bump manifest FORMAT_VERSION 2->3 (AD-373-3: correctness-on-upgrade, reuses the existing version-mismatch rebuild path; behavior of THIS ticket, no #NEW)
• Canonical FileId ordering = byte-wise comparison of normalized rel-path string (AD-373-1: walk sort uses the same byte-wise order as manifest BTreeMap resolution to eliminate skew)
• Single source of truth for rel-path -> manifest-key normalization (AD-373-2: three confirmed sites walk.rs:428, index.rs:615, index.rs:737; temporal.rs:112 left with rustdoc note intentionally NARROW scope)
• Platform-agnostic collision test validates the fix (feed ./foo and foo which normalize-collide on any OS, not Windows-specific; proves round-trip without Windows CI)

Acceptance criteria

Expand AC list

1. AC-1 (Functionality - Discriminating Repro): For corpus [foo.rs, foo/bar.rs, foobar.rs], run_ast_standalone("function_item > block") MUST emit foo/bar.rs and MUST NOT emit foo.rs or foobar.rs (FAILS on current code)
2. AC-2 (Functionality - Round-Trip Identity): After build_index over nested-dir corpus, manifest.sorted_paths()[n] MUST be byte-identical to rel-path assigned FileId(n) during walk (FAILS on current code)
3. AC-3 (Functionality - Lexical Consumer Fixed): Token unique to foo/bar.rs MUST resolve to foo/bar.rs and MUST NOT return foo.rs (FAILS on current code where verify gate masks skew as recall drop)
4. AC-4 (API Contract - Signature Unchanged): run_ast_standalone retains exact 10-arg signature, returns ExitCode::SUCCESS; manifest::sorted_paths retains Vec<&str> return; no new CLI flag/command
5. AC-5 (Functionality - Assignment Key Equals Lookup Key Byte-Level): Byte strings used to ORDER FileIds, STORE as manifest key, and LOOKUP in lexical SHA-cache MUST all be byte-identical (falsifiable: unit test on separator-variant paths)
6. AC-6 (NEGATIVE - No Regression on Flat Corpora): Flat single-directory corpus FileId->path resolution and result ordering MUST be byte-identical to pre-change; existing flat-corpus tests MUST pass unmodified
7. AC-7 (Functionality - Force-Rebuild on Upgrade FORMAT_VERSION 2->3 PF-007): manifest::FORMAT_VERSION MUST be 3; lexical FORMAT_VERSION=4 and AST format v2 unchanged; v2 manifest MUST auto-rebuild on next query via existing FORMAT_VERSION-mismatch path (no new staleness mechanism; FAILS if reverted)
8. AC-8 (Documentation Correctness PF-008): index.rs guard comment MUST NOT claim walk output is deduped nor rely on PathBuf order; MUST cite AD-373-1 byte-wise normalized-String invariant; manifest::sorted_paths and walk_metadata docs MUST state FileId assignment uses same byte-wise order as BTreeMap resolution
9. AC-9 (NEGATIVE - Commit-Boundary Guard Preserved ADR-006): manifest_count != file_count guard MUST be preserved; two walk entries normalizing to same key MUST return Err before save() (falsifiable: constructed collision scenario)
10. AC-10 (Performance - Grounded Guard ADR-003): Sort comparator MUST NOT call filesystem (no std::fs/canonicalize per comparison); operates only on in-memory PathBuf string work
11. AC-11 (Correctness-on-Upgrade PF-007): Pre-existing v2 manifest MUST trigger auto-rebuild on next skim search query under fixed v3 binary, no manual --rebuild; on-disk manifest MUST be rewritten at v3 after rebuild with correct FileId ordering

Build context

Wave-4 search follow-ups. Sequence position 2 of 10. Depends on: none. Unblocks #374 and #380. AD-373-1/2/3 rustdoc at decision sites (ADR-004); falsifiable ACs (PF-007); cross-plan note: manifest version timeline is 2->3 (#373, stays JSONL) then 3->4 (#380, JSONL->binary); #380 rebases onto v3.

[dean0x]

Resolved on wave/wave4-search via PR #386 (Wave-4 dog-food round-2). Residual partial-scope was ticketed as #388–#391 + #392 and resolved. Not yet on main — ships with PR #368 (wave→main).