Repository navigation
Conversation
This was referenced Sep 18, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two independent implementations of sedlex's matching semantics are compared on the same (rules, input) pairs; a disagreement is a compiler bug. - A brute-force backtracking matcher over Ir.t enumerates every parse in leftmost-greedy priority order (alternation prefers the left branch, repetition prefers one more iteration) and keeps the first parse among those of maximal length; across rules the longest lexeme wins and the lowest-numbered rule breaks ties. End of input is a marker after the input, read at most once and adding no length. This is the specification of capture semantics. - An interpreter for Sedlex.compile_ir output runs it the way the generated code does: transition operations record the position before the code point that selected them, final operations run on accept before the mark, end of input is fed once, and bindings are extracted through the discriminator cells as gen_binding_code does. Hand-written cases cover the capture shapes, or-patterns, greedy disambiguation, the loop-before-capture patterns of ocaml-community#199 and end of input. Two deterministic random sweeps over the alphabet a..d plus eof print nothing when every case agrees. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
hhugo
marked this pull request as ready for review
October 1, 2026 10:29
This was referenced Oct 1, 2026
Merged
This was referenced Oct 4, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tests only: a differential test harness for the compiler, in a new
test/oraclelibrary that usessedlex.compilerdirectly and involves no PPX.Two independent implementations of sedlex's matching semantics run on the same (rules, input) pairs and their results are compared; a disagreement is a compiler bug.
ref_matchintest/oracle/oracle.ml): a brute-force backtracking matcher overIr.t. It enumerates every parse in leftmost-greedy priority order (alternation prefers the left branch,Star/Plus/Repprefer one more iteration) and keeps the first parse among those of maximal length; across rules the longest lexeme wins and the lowest-numbered rule breaks ties. End of input is a marker after the input, read at most once and adding no length (Read end of input once, and let priority decide its ties #211). This is the specification of capture semantics and the part to review. It is deliberately naive so that it is correct by inspection.dfa_match): runs the output ofSedlex.compile_irthe way PPX-generated code does: memory cells start at -1, a transition's operations record the position before the code point that selected it (TDFA(1): delay the tag writes of a closure to the outgoing transitions #209), the final operations run on accept before the mark (Stop copying the capture cells in Sedlexing.mark and backtrack #208), end of input is fed once as the code point -1 without advancing, and on a dead end the last mark wins. Bindings are extracted asgen_binding_codedoes, discriminator cells included.Entry points:
oracleprints one line per input, prefixed withERRORand both results on disagreement;qcheckis a deterministic random sweep (fixed seed, no shrinking) over sequences of up to four elements on the alphabeta..dpluseof, with at least one capture, plus or-pattern rules binding the same name in both branches, on inputs of length 0 to 5. Failing cases print throughoraclewith the pattern, so an empty expect block means full agreement.The hand-written cases cover the basic capture shapes, or-patterns, greedy disambiguation, the loop-before-capture patterns of #199 and end of input (capture before
eof,eofadding no length so rule priority decides, a tie within a rule,Plus eof/Star ('a' | eof)/eof, eof). Every expect block is clean on master; the harness is what was used to check #204, #207, #208, #209, #210 and #211 as they were written (millions of random cases locally, the sweeps here are sized for CI: about 1.5 s).The interpreter is only as faithful as its imitation of the PPX glue, so a bug in that glue rather than in the compiler can escape it.
Commits:
qcheck-coredeclared as awith-testdependency indune-projectandsedlex.opam.oracle.ml), the expect tests (test_oracle.ml), and the dune library, gated on OCaml >= 4.14 like the other expect tests.🤖 Generated with Claude Code