Count end of input as one more symbol read (3.8.1)

[hhugo]

3.8 made eof tie with the accept of the state it is read from (#211), so an
earlier rule matching s beat a later rule matching s, eof. With s empty,
that is a rule matching the empty string listed before the eof rule, a common
shape for a lexer loop:

let rec token buf =
  match%sedlex buf with
  | '@' -> ...
  | Star (Sub (any, '@')) -> ...; token buf
  | eof -> ()
  | _ -> assert false

At end of input the Star rule now won with an empty lexeme, forever. Found
through the opam-repository CI of the 3.8 release
(ocaml/opam-repository#30869): pds has this lexer, and every package running
pds at build time was killed (hll, merlin-of-pds, revops,
lua_pattern).

This goes back to what every release before 3.8 did: end of input counts as
one more symbol read. A match that reads it is longer than one that stops just
before it, between rules and within a rule; rule order only decides between
matches of the same length under that count. The rest of #211 stays: end of
input is read once, so Plus eof and Star ('a' | eof) terminate and
eof, eof does not match.

Changes:

• Sedlex.transitions: every configuration of a state may read end of input,
  not only those preceding its first final one. The ppx and the runtime are
  unchanged: an eof arm still returns its rule directly, which is sound
  since that match is preferred over the state's own accept.
• test/basic.ml: eof_rule_priority becomes eof_longest_match and gains
  the nullable-rule-before-eof loop, two eof rules in order, and a capture
  overridden by the eof rule; eof_zero_width_tie_within_rule gains a
  branch reading eof after the leftmost one.
• README: the longest-match section states the rule.
• The second commit bumps the version to 3.8.1 and dates the changelog.

Not covered here: the oracle of #202 still implements the 3.8 rule.

🤖 Generated with Claude Code