Skip to content

Compare the oracle reference matcher with ocamllex - #194

Open
hhugo wants to merge 3 commits into
ocaml-community:pr3-oraclefrom
hhugo:oracle-testing
Open

hhugo wants to merge 3 commits into
ocaml-community:pr3-oraclefrom
hhugo:oracle-testing

Conversation

@hhugo

@hhugo hhugo commented Apr 2, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #202 (the GitHub base is a copy of its branch): ocamllex as a second reference for the oracle harness. Tests only.

  • Vendored ocamllex (test/ocamllex_vendored, from OCaml 5.4.0, sync.sh to refresh): Lexgen.make_dfa and the modules it needs, unchanged. These files are LGPL 2.1 with the OCaml linking exception; they are only linked into the test runner.
  • Bridge (test/ocamllex_oracle/ocamllex_oracle.ml): a rule set over Ir.t is translated to ocamllex's syntax (alternatives in order, repetition written iterate-first), compiled with Lexgen.make_dfa, and the automaton is run on the input the way ocamllex's engine does. The result is compared with the reference matcher of Add an oracle test harness: reference matcher vs compiled tagged DFA #202.
  • Harness change (test/oracle/oracle.ml): sweep takes the check to run and the printer for failures, and the generators take ~eof so a comparison that cannot model end of input can leave it out of the patterns.

What the comparison found:

  • On the CI-sized sweeps (2000 single-rule and 1000 two-rule cases, no eof) and on 300k local cases, ocamllex and the reference agree on rule and length everywhere.
  • Captures differ in one way only: ocamllex gives the earlier sub-pattern the longest match where sedlex takes the left alternative (('c' | 'd' ['a'-'b']) (['b'-'d']* as y) on db: sedlex y="b", ocamllex y=""). The comparison accepts ocamllex's bindings when they are another parse of the same lexeme and prints them after the reference's; a hand-written case shows two such pairs.
  • ocamllex 5.4.0 itself miscomputes a capture preceded by an overlapping loop, the same shape as sedlex's Rewrite determinization as a Laurikari TDFA for correct capture semantics #199: ('a' | ['b'-'d']*) (('c' | "bb") as y) (['a'-'d'] ['a'-'d'])* (['a'-'b'] as w) on ccdb gives y="", which no parse admits (y="c" is the only one). Reproduced with the real ocamllex binary, not just the vendored simulation. Pinned as an ERROR line in a hand-written case.

End of input is not translated: ocamllex can read it repeatedly, sedlex reads it once.

Commits (on top of #202):

  1. Vendor ocamllex's DFA compiler from OCaml 5.4.0.
  2. Let the oracle's sweep and generators serve a second comparison.
  3. Compare the reference matcher with ocamllex.

🤖 Generated with Claude Code

@pmetzger

Copy link
Copy Markdown
Member

@hhugo You should have the ability to commit such things on your own at this point...

@hhugo hhugo changed the title Extract sedlex.compiler library and add NFA/DFA oracle tests add NFA/DFA oracle tests Apr 13, 2026
@pmetzger

Copy link
Copy Markdown
Member

@hhugo Do you intend to commit this?

@hhugo

hhugo commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator Author

@hhugo Do you intend to commit this?

I need more time to work on this. The current implementation on master is not working properly. I've a fix locally, but still need to work on the specification of the behavior and work on testing (such as this PR).

hhugo and others added 3 commits October 1, 2026 13:44
Copy lex/{cset,lexgen,syntax,table}.{ml,mli} verbatim from the OCaml
compiler sources. These provide an independent TDFA implementation
used as a reference oracle for testing sedlex's tagged DFA.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
`sweep` takes the check to run and the printer for failing cases;
`qcheck` is the reference-vs-DFA instance. The generators take `~eof`
so a comparison that cannot model end of input can leave it out of
the patterns.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A rule set over Ir.t is translated to ocamllex's syntax, compiled with
the vendored Lexgen.make_dfa, and the automaton is run on the input;
the result is compared with the reference matcher. Alternatives keep
their order and repetition is written iterate-first, so the two agree
on every case of the random sweeps.

Two kinds of difference remain and are told apart: ocamllex gives the
earlier sub-pattern the longest match where sedlex prefers the left
alternative, which the comparison accepts when ocamllex's bindings are
another parse of the same lexeme (shown after the reference's); and
ocamllex 5.4.0 itself miscomputes a capture preceded by an overlapping
loop, pinned as an ERROR line in a hand-written case.

End of input is left out of the patterns: ocamllex can read it
repeatedly, sedlex reads it once.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@hhugo hhugo changed the title add NFA/DFA oracle tests Compare the oracle reference matcher with ocamllex Oct 1, 2026
@hhugo
hhugo changed the base branch from master to pr3-oracle October 1, 2026 11:49
@hhugo
hhugo marked this pull request as ready for review October 1, 2026 11:51
@hhugo hhugo mentioned this pull request Oct 1, 2026
@hhugo hhugo added the testing label Oct 1, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants