perf(deflate): relax each match length at its own nearest distance

[justin13888]

Stacked on #499 (perf/478-deflate-u64-compare); this PR's base is that branch so its diff shows only this change. Merge #499 first, then retarget or merge this one.

This is an unattended run: no human approved the plan. The decision record below is what a human reads afterwards.

Summary

• Matcher::find_sublen runs the existing nearest-first hash-chain walk and also fills a caller-owned [u16; MAX_MATCH + 1] in which sublen[len] is the nearest distance achieving at least len (zopfli's sublen): a candidate records exactly the lengths it is the first to reach, so the first to reach a length is the nearest that does. find keeps its signature and delegates to the same walk with no table, so the lazy parser (Level::Fast/Default) pays nothing and is byte-identical.
• parse_dp (the Level::Best shortest-path pass) relaxes each length 3..=longest at sublen[len] instead of at the longest match's distance, re-deriving the distance cost only when the distance changes along the length axis. The table is allocated once per span, not per position.
• Why: a short match was forced to pay the far match's distance code — the largest remaining ratio gap to zopfli named in gamut-deflate: relax each match length at its own nearest distance #479. Result on the bench corpus (Level::Best, zlib streams): rfc1951.txt 10 767 → 10 664 (gap to zopfli +2.1% → +1.1%), lz77.rs at the gamut-deflate: compare 8 bytes at a time in longest_match #478 head 8 055 → 8 003 (+1.3% → +0.7%); text.x300/ramp20k/pseudo20k unchanged (their short lengths' nearest distance is the longest match's). tests/default_bytes.rs: two Best pins moved, both down — skewed 1061 → 1048, mixed 21643 → 21601 — and no row grew at any level; every Fast/Default pin is unchanged.
• Cost: Level::Best on the rfc1950.txt throughput input 22.88 → 24.68 ms fastest (+7.9%; median +7.0%) in one back-to-back session where the matcher-free Store row also moved 7%, so that is an upper bound. Numbers and invocation recorded in STATUS.md; README.md table updated.
• Two inline tests: find_sublen_records_each_length_at_its_nearest_distance (nearer candidate reaches 3, farther reaches 5 → sublen[3] = near, sublen[4..=5] = far, rest 0, longest unchanged) and optimal_parse_takes_a_short_match_at_its_nearer_distance (hand-set costs under which relaxing at the longest match's distance emits Match{5, 44} and per-length relaxation must emit Match{3, 3} + two literals).

Validation

All commands ran in the lane worktree on commit 18c058f (this tree); crate-scoped runs with CARGO_BUILD_JOBS=2, workspace runs inside systemd-run --user --scope --slice=agents.slice -p MemoryMax=16G -p MemorySwapMax=0 -- env CARGO_BUILD_JOBS=2 CMAKE_BUILD_PARALLEL_LEVEL=2 sh -c 'ulimit -v 12000000; …'.

• cargo test -p gamut-deflate --all-features — pass: 49 unit tests (incl. the two new ones), default_bytes 1, oracle 11 (incl. best_beats_zlib_9), 1 doctest. Before re-pinning, the same run reported exactly two moved pins (skewed/Best 1048, mixed/Best 21601), both smaller.
• __CARGO_TEST_ROOT=<worktree root> mise run fmt then … mise run fmt-check — exit 0 (the prefix is the documented nested-worktree artefact; no manifest changed).
• mise run check-tests — pass ("module docs, pinned proptest seeds and oracle filenames all conform").
• convco check origin/perf/478-deflate-u64-compare..HEAD — "no errors in 1 commit".
• mise run lint (capped scope) — exit 0 (cargo clippy --workspace --all-targets --all-features -- -D warnings, 8m30s).
• mise run test (capped scope) — exit 0, no failing test binary.
• cargo bench -p gamut-deflate --bench compression size table, before (at 4f2c2a4) and after; the like-for-like lz77.rs row is the after binary run directly over the base revision of the file (my copy restored and SHA-256-verified afterwards). Throughput: cargo bench -p gamut-deflate --bench compression -- gamut, base code then this code, back to back. Numbers in the Summary and in STATUS.md.
• GAMUT_MUTANTS_BASE=origin/perf/478-deflate-u64-compare mise run mutants-diff (capped scope) — "Found 34 mutants to test … 34 mutants tested in 49s: 27 caught, 7 unviable"; 0 missed, 0 timeouts, no exclusion added.
• After the independent review's two Markdown-only repairs (second commit): cargo test -p gamut-deflate --all-features re-run — pass (same counts); __CARGO_TEST_ROOT=<worktree root> mise run fmt-check, mise run check-tests, convco check origin/perf/478-deflate-u64-compare..HEAD — pass. lint/test/mutants-diff were not re-run: the second commit touches only STATUS.md/README.md, neither of which is compiled or included in crate docs (lib.rs has no include_str!).
• Not run, with reason: check-release-deps/check-ffi-features/check-ffi-header (no Cargo.toml or C surface touched); coverage (no new module; the new code is exercised by the existing suite and two inline tests).
• Required checks at 862c799 (first two commits): Format & Metadata, Clippy & Doctests, Coverage (test gate), Incremental (PR diff) 0–3 — all passed.
• Third commit (docs-only, decision 12: README.md/STATUS.md wording): __CARGO_TEST_ROOT=<worktree root> mise run fmt-check, mise run check-tests, convco check origin/perf/478-deflate-u64-compare..HEAD — pass; lint/test/mutants-diff not re-run (Markdown only).

Risks and rollout

• Output bytes at Level::Best change (smaller on every measured input; Fast/Default/Store byte-identical). Any consumer pinning exact Best output must re-pin; inside the workspace only gamut-deflate's own default_bytes.rs does, and it is re-pinned here.
• Level::Best is up to ~8% slower on prose-like input (the per-position table clear and fill); Default/Fast unaffected. Reverting is one commit.
• No public API change, no new dependency, no unsafe.

Issue

Closes #479

Decisions taken

Issue 479 - gamut-deflate: relax each match length at its own nearest distance
Plan:     v1
Branch:   perf/479-deflate-per-length-distance
Base:     head of PR #499 (origin/perf/478-deflate-u64-compare, 4f2c2a4) - stacked; PR opened against that branch
Cause:    -
Touches:  gamut-deflate lz77.rs (Matcher::find returns a per-length nearest-distance table; parse_dp relaxes each length at its own distance), tests/default_bytes.rs (re-pinned), STATUS.md/README.md ratio numbers
Will not: change the hash chain, WINDOW, MIN/MAX_MATCH, the lazy parser (parse_range), the cost model, or the public API; add a dependency
Lane:     serialised behind E-478 (stacked); #493 branches from this head
Settled:  S1 no new dependency; S3 docs/testing.md; #499's appended decision 5 (the always-true guard in front of the prune is gone; do not reintroduce it)

Decisions taken.
1. Deliverable boundary
   Taken:    the whole issue, contained to Matcher::find and parse_dp as the issue states
   Rejected: also applying per-length distances to the lazy parser (parse_range) - outside the issue's two named functions; the DP is where the cost model prices distances
   Reverses: drop the parse_dp change
   Filed:    -
2. Table shape
   Taken:    a caller-owned `[u16; MAX_MATCH + 1]` (or a small struct wrapping it) filled by `Matcher::find_sublen(&mut sublen, ...)`, where `sublen[len]` is the nearest distance achieving at least `len`, 0 where none; `find` keeps its signature by delegating (the lazy parser still needs only the longest) - consequence: no allocation per position, lz77.rs only
   Rejected: `Vec<(len, dist)>` per position - allocation in the hot loop; changing `find`'s signature - touches parse_range, which is out of scope
   Reverses: replace the fixed array with a Vec
3. Relaxation
   Taken:    in parse_dp, for each length l in MIN_MATCH..=max_len relax with `dist_cost[code(sublen[l])]`, where sublen is filled nearest-first so the first candidate reaching a length wins (matching Zopfli's semantics); the longest-match result is unchanged
   Reverses: relax every length at the single longest-match distance again
4. Acceptance
   Taken:    `best_beats_zlib_9` still passes; the zopfli ratio table (bench) is recorded before/after in STATUS.md with the invocation; every corpus row in default_bytes.rs must be <= its previous size at every level, else the change is a negative result: freeze at the last pushed commit and report which rows grew and by how much (do not merge a regression into the stack)
   Rejected: accepting a mixed result - the issue frames this as the largest remaining ratio gap, so a row that grows means the implementation is wrong
5. Mutation
   Taken:    every new operator must be caught (mutants-diff in the capped scope); a test that pins, for a constructed input, that a short length takes a nearer distance than the longest match's is the natural killer; no exclusion added
6. Shared walk (appended)
   Taken:    `find` and `find_sublen` share one private `walk(..., Option<&mut [u16; MAX_MATCH + 1]>)`; the table is optional so the lazy parser's two `find` calls per position do not clear a 518-byte table they never read
   Rejected: `find` allocating a stack table and calling `find_sublen` - a memset per lazy-parser call for nothing; duplicating the chain loop in two functions - two copies of the prune invariant to keep in step
   Reverses: inline the walk into `find_sublen` and have `find` call it with a scratch table
7. Distance-cost re-derivation (appended)
   Taken:    in `parse_dp` the distance code and its cost are recomputed only when `sublen[len]` differs from the previous length's distance (the table is non-decreasing in `len` and piecewise constant), with the sentinel `dist = 0` guaranteeing the first length recomputes because every filled entry is >= 1
   Rejected: calling `symbols::distance_code` (a 30-entry linear scan) for every one of up to 256 lengths per position - up to ~7.7k iterations per position, comparable to the chain walk itself; a 32 769-entry distance-cost lookup table per span - 64 KiB of extra working set for a cost that changes a handful of times per position
   Reverses: drop the `!=` guard and compute the cost inside the loop unconditionally
8. Like-for-like bench row (appended)
   Taken:    the bench's `lz77.rs` row compresses the source file this change edits, so STATUS.md reports that row over the file at the #478 head for both columns (new binary run directly over the base revision of the file; the file was restored and hash-verified); README.md's table shows the current file
   Rejected: reporting the printed after-row (raw 30 610 vs 25 031) as a before/after comparison - different inputs
   Reverses: none needed; documentation only
9. Commit before the independent review reported (appended)
   Taken:    the first commit was made and pushed while the review sub-agent was still running, after every other gate (crate tests, fmt-check, check-tests, workspace lint and test) had passed on the identical tree; the review's findings landed as a second, docs-only commit
   Rejected: idling until the review returned - the lane contract makes a pushed commit the durable state and forbids ending the turn early; a review finding never needs history rewriting, only a further commit
   Reverses: nothing to reverse; ordering only
10. Unnamed kill of the `*` mutant in the distance cost (appended)
   Taken:    the `+` -> `*` mutant of `dist_cost[dsym] + dbits` is killed by `tests/default_bytes.rs` (Best output moves), not by a test naming that operator; accepted, because the pin file exists precisely to catch the optimiser silently changing, and `mutants-diff` reports 0 missed
   Rejected: a third fixture candidate at distance 5..=8 with dist cost 17 and 1 extra bit so that `*` (17) ties and `+` (18) loses - the kill would rest on a strict-`<` tie order, which is the kind of fixture the workspace's own mutation notes warn against
   Reverses: add that candidate to `optimal_parse_takes_a_short_match_at_its_nearer_distance` if a named kill is wanted
11. Per-position table clear kept (appended)
   Taken:    `find_sublen` zeroes all 259 entries at every position so "every other entry is 0" is a contract a test can assert; `parse_dp` reads only `3..=max_len`, which the walk always writes
   Rejected: zeroing only `..=previous best_len` or dropping the contract - saves part of the measured <= 8% Best cost but makes `sublen[len]` for `len > max_len` stale, which decision 2 rules out ("0 where none")
   Reverses: replace `sublen.fill(0)` with a documented "entries above the returned length are unspecified" contract and update the inline test
12. Decided by the orchestrator on review (appended; docs-only commit)
   Taken:    1a - keep the full 259-entry clear per position and the "0 where none" contract (decision 11 stands; no code change). 2a - keep the README `lz77.rs` row pinned to the file at 4f2c2a4, and make the table's reproducibility claim exact: every row except that one is reproduced by `cargo bench -p gamut-deflate`, that row is historical and says so in the note. Also STATUS.md's Throughput section now names `Matcher::walk` (behind `find` and `find_sublen`) as the Best-level loop instead of `Matcher::find`
   Rejected: 1b - clearing only `..=previous best_len` / dropping the contract to trim the measured <= 8% Best cost - trades a testable contract for an unmeasured fraction of a noise-level figure; 2b - regenerating the `lz77.rs` row from the current file as the last edit - self-referential, drifts with every comment edit, and cannot be reproduced by a reader either
   Reverses: 1b is decision 11's Reverses; 2b is one table row plus deleting the note

Unresolved review notes

Independent review (a separate adversarial reading of the diff, reported after the first commit) raised four findings; dispositions:

• STATUS.md said ramp20k's matches are "all at distance 1"; the fixture is i % 256, so distance 256. Repaired (wording; the conclusion — one period, so the nearest distance for a short length is the longest match's — stands).
• README's lz77.rs row was self-referential (the file changed again under mise run fmt after the bench). Repaired: the row is pinned to the file at 4f2c2a4, as STATUS.md already does, with a note.
• The +→* mutant in dcost = dist_cost[dsym] + dbits is killed by the default_bytes.rs pins, not by a named inline test. Not changed — see decision 10; mutants-diff confirms it is caught.
• find_sublen clears the whole table per position to honour the "0 where none" contract. Not changed — decision 2 fixes that contract; see decision 11.

Nothing is left for a human reviewer to decide from this pass.