Skip to content

Wave 4 / Search: re-baseline #358 <30% index-size target + binarize the JSONL field-map sidecar #380

Closed
Closed
Wave 4 / Search: re-baseline #358 <30% index-size target + binarize the JSONL field-map sidecar#380
Labels
searchCode search featuretech-debtTechnical debt backlog itemwave-4Wave 4: Compound queries

Description

@dean0x
dean0x
opened on Jun 28, 2026

Wave: wave4-search-followups
Depends on: #373

flowchart LR
  T372["1. #372 lexical AND-intersect"]
  T373["2. #373 FileId skew"]
  T374["3. #374 AST verify gate"]
  T375["4. #375 remove index positional"]
  T377["5. #377 weights notice"]
  T380["6. #380 binarize + positional"]
  T376["7. #376 CRC marker"]
  T379["8. #379 worktree staleness"]
  T378["9. #378 risk volume weighting"]
  T381["10. #381 index-location docs"]
  T372 --> T373 --> T374 --> T375 --> T377 --> T380 --> T376 --> T379 --> T378 --> T381
  T373 -. requires .-> T374
  T373 -. requires .-> T380
  T380 -. requires .-> T376
  T375 -. requires .-> T377
  classDef cur fill:#dcffe4,stroke:#1a7f37,stroke-width:3px;
  class T380 cur;
Loading

Solid arrows = integration order into wave/wave4-search-followups. Dotted = hard dependency. One branch per ticket; this ticket highlighted.

Found via dog-fooding round 2 (2026-06-27, branch wave/wave4-search), tracked under #174 Wave 4. Priority: P2.

Problem: On-disk index is ~300% of source (skim-search 65 MB / 21 MB; devflow 697%, townhall 684%) — 10–23x over the #358 <30% target. The target is unachievable for a positional trigram index; zoekt-class engines land ~200–300%. skim is ~30–50% heavier than zoekt mainly due to a 13 MB JSONL field-map sidecar and a 1-byte field_id per posting.

Best fix: (1) re-baseline the #174 size target to a realistic ~200% (or define it against a compressed/non-positional variant if <100% is a hard product need); (2) replace the JSONL field-map sidecar with a binary encoding; (3) evaluate dropping/segmenting positions for the common case. Also: --stats "index size" (42 MB) undercounts on-disk by ~23 MB (omits skfiles+temporal+ast) — report true on-disk size.

Refs: #358, #174.

Activity

  1. dean0x
    added
    searchCode search feature
    wave-4Wave 4: Compound queries
    tech-debtTechnical debt backlog item
    on Jun 28, 2026

  2. dean0x commented on Jun 28, 2026

    @dean0x
    dean0x
    on Jun 28, 2026
    OwnerAuthor

    Plan + resolved decisions (dynamic-plan, 2026-06-28)

    Durable record of #380 scope, phasing, and resolution (full plan/test plan local, gitignored). Comprehensive scope: four workstreams split Phase 1 (mechanical, low-risk) and Phase 2 (research-gated, higher-risk).

    Resolved design decisions

    Acceptance criteria

    Phase 1 (binarize sidecar + re-baseline + accurate stats)
    1. FUNCTIONALITY (binarize field-map, round-trip): FileManifest with field_map triples whose end byte-offset exceeds 65535 (u16::MAX) MUST survive save()->load() with every field byte-identical. NEGATIVE: on-disk index.skfiles MUST NOT contain ASCII JSON-array encoding (no literal '],[' substring) after save().
    2. API CONTRACT (format version + self-describing header): FORMAT_VERSION bumped to 4 (u32), rebasing onto Wave 4 / Search: standalone --ast resolves matches to the WRONG files (FileId→path ordering skew) #373's v3. Binary body MUST begin with 4-byte magic (b"SKFM") + version + u32 entry count, mirroring ast_index.skcache. load() returns Ok(empty manifest) when magic absent, version != 4, count > MAX_MANIFEST_ENTRIES, or body truncated.
    3. API CONTRACT (bounded + saturating, observable): Binary encoder MUST be SATURATING — length/count exceeding u32::MAX written as u32::MAX, never panic. Decoder MUST REJECT BEFORE allocating when declared count > MAX_MANIFEST_ENTRIES (60000) or file size > MAX_MANIFEST_FILE_BYTES (256 MiB). NEGATIVE: forged u32::MAX count + oversized declared per-entry length yields Ok(empty) without panic.
    4. FORMAT COMPATIBILITY / SELF-HEAL (central negative): After 3->4 bump, v3 JSONL index.skfiles (immediate predecessor, post-Wave 4 / Search: standalone --ast resolves matches to the WRONG files (FileId→path ordering skew) #373) and older v2 JSONL MUST trigger full automatic rebuild on next skim search <query> or skim search index WITHOUT --rebuild and WITHOUT error, FOR BOTH git AND non-git roots. NEGATIVE: system MUST NOT silently accept below-current (< v4) manifest as Current. check_staleness MUST detect below-current FORMAT_VERSION independent of git HEAD state.
    5. FUNCTIONALITY (FileId alignment under corruption): On corrupt/truncated binary manifest body, load() MUST reject WHOLE file (Ok(empty)) and MUST NOT return partially-recovered manifest shorter than written, so sorted_paths()[n]==path-for-FileId(n) invariant never violated. Falsifiable: valid header with count=3 but only 1.5 entries of body yields entry_count()==0.
    6. API CONTRACT + FUNCTIONALITY (--stats reports TRUE on-disk size): skim search --stats (text and --json) MUST report total-on-disk size = sum of metadata().len() over present index artifacts in cache_dir (index.skidx, index.skpost, index.skfiles, ast_index.skidx, ast_index.skpost, ast_index.skcache, temporal.db). NEGATIVE: total MUST NOT report lexical-only when other artifacts exist.
    7. API CONTRACT (stats fail-soft + complexity bound): --stats total MUST treat missing artifact as 0 bytes, no error. MUST iterate FIXED known filename list via metadata() (O(1)), MUST NOT recursive directory walk. NgramIndexReader::stats.index_size_bytes unchanged (skidx+skpost only). Falsifiable: with only three lexical files present, reported total equals exactly their summed sizes.
    8. GROUNDED REGRESSION GUARD (ADR-003, replaces baseless <30% target): NEW non-#[ignore] test MUST build real index via build_index over >= 8 KiB source corpus, measure skfiles bytes / source bytes, eprintln! ratio, assert < ceiling = measured post-binarization ratio plus fixed headroom. DISCRIMINATING (PF-007): test MUST FAIL if field-map encoding reverts to JSONL (JSONL ratio exceeds ceiling).
    9. GROUNDED REGRESSION GUARD (whole-footprint, optional): IF whole-on-disk-footprint guard added, MUST exclude temporal.db from numerator (scales with git history) while temporal.db MAY still be reported under --stats. Ceiling = measured sum(6 deterministic artifacts)/source_bytes plus headroom, recorded as real number.
    10. KNOWLEDGE DRIFT (PF-008): .devflow/features/cmd-search/KNOWLEDGE.md MUST describe manifest as binary (v4), remove 'JSONL' descriptor (lines 76/168/451-454/686), collapse duplicate 'manifest.jsonl' listing (lines 522-523) into single index.skfiles entry. index.json keyword blob updated consistently. NEGATIVE: CLAUDE.md edit NOT required; grep for JSONL must stay empty.
    11. NO QUERY-PATH REGRESSION (negative, performance): Same query against binary-manifest index MUST return identical ordered SearchResult set (same paths, ranking, scores within tolerance) as prior JSONL-manifest build for same source tree. NEGATIVE: field_map MUST NOT be read at query time. Falsifiable: diff ordered results from parent-commit (JSONL) vs branch (binary) build.
    12. ADR-006 ORDERING PRESERVED (negative): Binary save() MUST remain atomic temp-file-then-rename (readers never observe partial manifest). Build pipeline MUST persist ast_index.skcache and lexical/AST indexes BEFORE new_manifest.save(), manifest saved LAST. NEGATIVE: skcache or index write failure aborts BEFORE manifest saved so old index survives. Falsifiable: existing atomic-replace and ADR-006 abort tests pass against binary writer.
    Phase 2 (positional / segment postings codec — research-gated)

    AC-P2-1 (FUNCTIONALITY — positions round-trip & decode, positive PF-007): After lexical postings codec extended, posting list for known file with known token offsets MUST survive encode->decode with every position byte-identical (positions delta+varint-decoded back to exact ascending offsets, none truncated). Lexical FORMAT_VERSION MUST bump v4->v5, positional postings file MUST begin with v5 version marker. Carries // AD-380-9: rustdoc at codec call site.

    AC-P2-2 (FALSIFIABLE PF-007 — positional/proximity adjacency correctness, the required negative): Corpus: file A contains adjacent token sequence alpha beta; file B contains NON-adjacent (or reversed/distant) alpha … beta. Phrase/proximity query ("alpha beta" or --near 1) MUST return file A, MUST NOT return file B. NEGATIVE (fails when feature deleted/broken): if positions ignored (codec reverted to bag-of-trigrams) same query returns BOTH A and B — test MUST FAIL. Pinned that positional/proximity queries enforce token adjacency from decoded positions.

    AC-P2-3 (NO-REGRESSION — non-positional queries byte-identical, negative): Plain (non-phrase, non-proximity) trigram query against v5 positional index MUST return identical ordered SearchResult set (same paths, order, scores within tolerance) as Phase-1 (v4, non-positional) index for same source tree. NEGATIVE: v4 postings file under v5 binary MUST cold-start (not mis-parsed as positional); check_staleness MUST flag below-current v4 lexical version independent of git state.

    AC-P2-4 (PERFORMANCE + RE-GROUNDED SIZE GUARD, ADR-003): Positional queries MUST stay within query budget (<50ms per 1000-line file class; phrase/proximity adds only bounded post-verify adjacency check). Phase-1 grounded size ceiling MUST RE-MEASURED and RE-SET against positional build: new index.skidx/skpost (and total) ratio recorded as real measured number + fixed headroom in test comment (NOT arbitrary, NOT 0.30x). Falsifiable: re-grounded guard FAILS if positions persisted without re-measuring (old v4 ceiling breached) AND FAILS if codec unbounded.

    Build context

    Wave-4 search follow-ups. Sequence position 6 of 10. Depends on: #373 (normalize_rel_path). Unblocks: #376 (manifest format locked). AD-380-N rustdoc at decision sites (ADR-004, ADR-003, ADR-006). Falsifiable ACs (PF-007).

  3. dean0x
    added 6 commits that reference this issue on Jul 1, 2026

    feat(rebaseline-binarize): binary v4 manifest, true on-disk --stats, …

    0131004

    Merge ticket/380-rebaseline-binarize into wave/wave4-phase2-staging (#…

    f31095c

    feat(search): add token_position to postings, format v5 for positiona…

    a009a2c

    feat(search): --phrase / --near positional query over v5 token_positi…

    6ba3167

    test(search): AC-P2 phrase/near acceptance + v4→v5 self-heal + v5 siz…

    83cbb4a

    merge(#386): wave-4 search follow-ups (#373-#381) + #380 Phase 2 posi…

    ea99703

  4. dean0x commented on Jul 2, 2026

    @dean0x
    dean0x
    on Jul 2, 2026
    OwnerAuthor

    Resolved on wave/wave4-search via PR #386 (Wave-4 dog-food round-2). Residual partial-scope was ticketed as #388–#391 + #392 and resolved. Not yet on main — ships with PR #368 (wave→main).

  5. dean0x
    closed this as completedon Jul 2, 2026
  6. dean0x
    added a commit that references this issue on Sep 24, 2026

    Wave 4: 3-layer code search — intersection / ranking / formatting / C…

    b8a0a79
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    searchCode search featuretech-debtTechnical debt backlog itemwave-4Wave 4: Compound queries

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions