Rewrite determinization as a Laurikari TDFA for correct capture semantics

[hhugo]

Fixes wrong sub-match positions for as bindings (#199) by rewriting determinization as a tagged DFA where each NFA path carries its own registers. Based on master (#203, the plumbing, is merged). Follow-ups stacked on this PR: #207 (register optimizations), #208 (no copy of the capture cells in mark/backtrack), #209 (TDFA(1)).

The bug. One flat tag list per transition, one register per tag shared by all live paths. When a tagged node is reachable from several paths at once, e.g. Star 'a', (('a', Plus 'b') as x), the loop path re-fires the write on every iteration and clobbers the position of the path already inside the capture: wrong, empty or out-of-bounds sub-matches.

The fix. A DFA state is an ordered list of configurations (NFA node, tag → register). Order is priority: the epsilon closure is a DFS along eps lists and the first path to a node wins, i.e. leftmost-greedy among the parses of the longest match. States are looked up modulo register renaming; reaching an existing state emits the moves realigning the registers, as a parallel move. An accepting state copies the accepting path's registers into the canonical cells read by the bindings, just before Sedlexing.mark. This is Laurikari's construction, a TDFA(0) in Trofimovich's terms. Patterns without captures generate byte-identical code.

Every tag write gets a fresh register and every tag is copied on accept: the simplest correct scheme, two cells per tag. #207 brings ordinary patterns back to one cell and no copy.

Commits

1. Determinize over ordered configurations. Flips the loop_before_capture and capture_before_loop tests pinned in Pin known bugs with regression tests #200.
2. Bounded repetition is greedy, like Star/Plus.
3. Keep only nodes that matter in DFA states (no epsilon-only nodes): smaller keys, a few states merge.
4. Simplify Seq lowering.
5. Opt is greedy, so Opt p and Rep (p, 0 .. 1) agree.
6. Plus is greedy when the body's first alternative is nullable (r, Star r).
7. Codegen tests: counterparts of the as-binding tests whose positions cannot be computed statically.
8. Remove a duplicate test.
9. Changelog.
10. Current/Goal notes of the codegen tests updated to the new cell counts.

opt_greedy and plus_nullable_first_alternative regress at 1 and are restored by 5 and 6. Every commit builds and passes dune runtest.

One change that is a decision, not a fix

rep_nullable_body (pinned in #200) flips: (Rep ((Opt 'd' | 'b'), 0 .. 1) as x), Star 'b' on "b" gave x="b", now gives x="". Both parses are maximal; under leftmost-greedy the body's first alternative, Opt 'd', matches empty and wins, exactly as for the spelled-out (Opt 'd' | 'b' | ""). All other flips replace captures that no parse can produce.

Verification

The #202 oracle, run locally against this tip (it needs small interpreter changes for Copy/accept, so it is not part of this PR): the 15 pinned disagreements clear, and 300k random cases agree.

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🟡 Changes recommended

Transition seed accumulation introduces avoidable quadratic compilation cost, and several regression comments remain outdated.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Avoid quadratic seed-list copying during overlapping moves · New
• Update test documentation to reflect fixed behavior · New

What changed in this PR

Reworks determinization into a Laurikari TDFA to preserve correct capture positions and leftmost-greedy semantics.

Changes:

• Tracks per-path tag registers and realigns them across DFA states.
• Corrects greedy Opt, Plus, and bounded repetition behavior.
• Expands capture regression and code-generation coverage.

File	Description
src/​compiler/​sedlex.ml	Implements tagged DFA construction and register handling.
src/​syntax/​ppx_sedlex.ml	Makes Opt consume-first.
test/​basic.ml	Updates capture and repetition regression expectations.
test/​codegen/​test_gen.ml	Adds and updates generated-code expectations.
test/​codegen/​test_realistic.ml	Updates realistic lexer register expectations.
CHANGES.md	Documents the capture fix.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[hhugo]

@pmetzger, this is ready to go. Let me know if you want to give it some review. Otherwise, I think I'll merge

[pmetzger]

If you feel it is ready, do it. I do still encourage eventually looking at the work done for re2c btw; they took Laurikari's work and advanced the state of the art significantly. They have papers explaining it.

[hhugo]

If you feel it is ready, do it. I do still encourage eventually looking at the work done for re2c btw; they took Laurikari's work and advanced the state of the art significantly. They have papers explaining it.

This work is inspired by re2c already