Repository navigation
Conversation
3 tasks done
This was referenced Apr 6, 2026
Member
|
@hhugo You should have the ability to commit such things on your own at this point... |
Member
|
@hhugo Do you intend to commit this? |
Collaborator
Author
I need more time to work on this. The current implementation on master is not working properly. I've a fix locally, but still need to work on the specification of the behavior and work on testing (such as this PR). |
This was referenced Sep 18, 2026
hhugo
marked this pull request as draft
September 21, 2026 00:04
Copy lex/{cset,lexgen,syntax,table}.{ml,mli} verbatim from the OCaml
compiler sources. These provide an independent TDFA implementation
used as a reference oracle for testing sedlex's tagged DFA.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
`sweep` takes the check to run and the printer for failing cases; `qcheck` is the reference-vs-DFA instance. The generators take `~eof` so a comparison that cannot model end of input can leave it out of the patterns. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A rule set over Ir.t is translated to ocamllex's syntax, compiled with the vendored Lexgen.make_dfa, and the automaton is run on the input; the result is compared with the reference matcher. Alternatives keep their order and repetition is written iterate-first, so the two agree on every case of the random sweeps. Two kinds of difference remain and are told apart: ocamllex gives the earlier sub-pattern the longest match where sedlex prefers the left alternative, which the comparison accepts when ocamllex's bindings are another parse of the same lexeme (shown after the reference's); and ocamllex 5.4.0 itself miscomputes a capture preceded by an overlapping loop, pinned as an ERROR line in a hand-written case. End of input is left out of the patterns: ocamllex can read it repeatedly, sedlex reads it once. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
hhugo
marked this pull request as ready for review
October 1, 2026 11:51
Merged
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #202 (the GitHub base is a copy of its branch): ocamllex as a second reference for the oracle harness. Tests only.
test/ocamllex_vendored, from OCaml 5.4.0,sync.shto refresh):Lexgen.make_dfaand the modules it needs, unchanged. These files are LGPL 2.1 with the OCaml linking exception; they are only linked into the test runner.test/ocamllex_oracle/ocamllex_oracle.ml): a rule set overIr.tis translated to ocamllex's syntax (alternatives in order, repetition written iterate-first), compiled withLexgen.make_dfa, and the automaton is run on the input the way ocamllex's engine does. The result is compared with the reference matcher of Add an oracle test harness: reference matcher vs compiled tagged DFA #202.test/oracle/oracle.ml):sweeptakes the check to run and the printer for failures, and the generators take~eofso a comparison that cannot model end of input can leave it out of the patterns.What the comparison found:
eof) and on 300k local cases, ocamllex and the reference agree on rule and length everywhere.('c' | 'd' ['a'-'b']) (['b'-'d']* as y)ondb: sedlexy="b", ocamllexy=""). The comparison accepts ocamllex's bindings when they are another parse of the same lexeme and prints them after the reference's; a hand-written case shows two such pairs.('a' | ['b'-'d']*) (('c' | "bb") as y) (['a'-'d'] ['a'-'d'])* (['a'-'b'] as w)onccdbgivesy="", which no parse admits (y="c"is the only one). Reproduced with the realocamllexbinary, not just the vendored simulation. Pinned as anERRORline in a hand-written case.End of input is not translated: ocamllex can read it repeatedly, sedlex reads it once.
Commits (on top of #202):
🤖 Generated with Claude Code