Prepare the 3.8 release

[hhugo]

Bump the version to 3.8 and date the changelog, as for 3.6 and 3.7:
dune-project, the regenerated sedlex.opam, and the CHANGES.md header.

Nothing else references the version. The Unicode version strings in
examples/ are the old/new pair used by the regression test and already
mention 18.0.0.

The date is today's; adjust it if the release slips.

🤖 Generated with Claude Code

[hhugo]

Here is a proposed anoncement post

---

[ANN] Sedlex 3.8

Sedlex 3.8 is out! 🎉

Sedlex is a unicode-friendly lexer generator for OCaml originally written by
Alain Frisch at LexiFi and now maintained under the
ocaml-community umbrella.

You can read the notes for this release here:
https://github.com/ocaml-community/sedlex/releases/tag/v3.8, and an updated
opam package should be available soon (https://github.com/ocaml/opam-repository/pull/NNNNN).

The headline feature is named capture groups. Sub-patterns can be bound
with OCaml's as syntax and read back in the action:

match%sedlex buf with
| (Plus ('0'..'9') as num), '.', (Plus ('0'..'9') as frac) ->
    let n = Sedlexing.Utf8.of_submatch num in
    let f = Sedlexing.Utf8.of_submatch frac in
    Printf.printf "integer=%s fractional=%s\n" n f
| _ -> ()

Captures are implemented with a tagged DFA, so the generated lexer remains a
single pass with no backtracking, and lexers that don't use as generate the
same code as before. Sub-matches are leftmost-greedy within the longest match.
as is not allowed under Star, Plus, Opt, Rep, Compl, Sub or
Intersect.

This release also brings:

• Unicode 18.0.0
• Nested let .. in definitions of [%sedlex.regexp?]
• Clearer errors for bare Star, Plus, Opt and friends, and for empty
  character sets, which used to crash the ppx or silently never match
• Fixes around eof: a rule matching s, eof no longer beats an earlier rule
  matching s, and patterns like Plus eof no longer loop forever at end of
  input

The eof priority fix is the one most likely to change the behaviour of an
existing lexer, so check any rule that ends in eof when upgrading.

Many thanks to everyone who reported issues and reviewed the capture-group
work.

[hhugo]

@pmetzger, @toots any thought ? I think master is in a good enough state for a release. The group capture works with no known bugs. There is room for improvement regarding memory/register sizes and number of op to track capture but it doesn't need to come now. This is no perf impact for lexer without group capture.

[hhugo]

#202 and #194 add 2 different oracles to check the implementation using qcheck. I'm not sure we want to merge them, the code has not been review carefully and is quite big (194 vendors ocamllex), but they were heavily used while checking recent change in the codebase.

[pmetzger]

I think de facto at the moment you are the person who understands this code the best and has the most ownership. Submatch capture is a really important feature, I think it warrants a release, but I don't have a good sense of what remaining performance issues etc. might exist.

If you feel it would be a good idea, how about comparing the performance of re2c in OCaml mode and Sedlex on a bunch of representative stuff? I suspect that could be done in an hour or two with the aid of our robot friends, and might yield some interesting information.

But do not feel obligated, this is a suggestion for your benefit, not a requirement! I think you should do what you feel is right. Or, in other words, if you think it's a good time for a release, let's do one!

[hhugo]

ocaml/opam-repository#30869