Do not clear the capture cells at every token

[hhugo]

Sedlexing.__private__init_mem was called on entering every match%sedlex that has as bindings, and filled all the memory cells of the rule with -1. On a port of the OCaml compiler's lexer that is 23 cells per token for the main rule, and 4 and 10 cells per character for the string and comment rules, which consume one character per call. The fill cost more than the tag operations themselves.

The function now only grows the array, and is renamed __private__ensure_mem to say so. The generated code is otherwise unchanged.

Why the fill is not needed

An action only reads cells that were written on the path of its own match:

• the position cells of a binding are written whenever the branch holding that binding matches, and are only read on that branch;
• an or-pattern has one discriminator cell that every branch writes with its own value, so the matching branch always overwrites it; nested discriminators are tested as a conjunction, so a stale inner cell is only looked at once the outer test has already failed;
• register copies and the shift done by refill may touch stale cells, but nothing reads the result.

How this was checked

• The whole test suite passes with the cells poisoned (filled with a large garbage value instead of -1 on every rule entry).
• The oracle of Add an oracle test harness: reference matcher vs compiled tagged DFA #202 passes its random sweeps with the DFA interpreter starting from garbage cells, for six poison values including 0 to 3, which collide with discriminator values. As a control, skipping the write to one cell in the interpreter makes it report errors.
• A new test, stale_cells_between_tokens, drives one lexbuf through two rules with different numbers of cells and or-patterns whose branches alternate between consecutive tokens.
• A second test, poisoned_cells, lexes about 1100 tokens read through a generator, so that the buffer is refilled, with rules covering nested or-patterns, a loop before an overlapping capture, static offsets and a capture ended by the end of input. Before every token it overwrites the cells with -1, with plausible positions, or with encoded discriminator values; the submatches are the same in every case. Skipping a tag write in the runtime makes it fail.

The poisoned runs were one-off checks and are not part of this PR.

Trade-off

Until now, a compiler bug reading a cell that was never written gave a negative position, and usually an exception. It would now return a stale position silently.

Numbers

The OCaml compiler lexer (about 50 as bindings) ported to sedlex, on 8 MB of compiler sources, Latin1 lexbuf, OCaml 5.4.0, release profile, median ns per byte over 11 rounds. "manual" is the same lexer without as, slicing the lexeme by hand; "kinds only" extracts no text. Four alternating runs of master (with #219) and this PR.

	captures	manual	kinds only	captures / manual
master	27.5 / 27.5 / 26.8 / 26.9	21.0 / 20.8 / 19.9 / 21.3	12.9 / 13.2 / 12.7 / 12.8	1.27 to 1.35
this PR	21.7 / 21.7 / 22.3 / 22.4	21.6 / 20.3 / 20.5 / 21.9	12.7 / 12.7 / 12.7 / 12.9	1.00 to 1.09

So the lexer with captures is about 19% faster (5 ns per byte), and captures cost 0 to 9% over slicing by hand instead of 27 to 35%. For reference, the same lexer generated by ocamllex -ml, with its captures, takes 23.3 to 23.8 ns per byte.

This lexer is a favourable case: its string and comment rules have captures and are entered once per character, so the fill was paid per character there. A lexer whose rules with captures match whole tokens saves one fill of a few cells per token.

🤖 Generated with Claude Code