perf(uring): gather held buffers into one Mode A forward write (+30%)

[brayniac]

Fourth of four, split out of #417. The runtime change — review this one hardest.

+30% on the forwarding path, instructions per byte 1.236 → 0.906 — past run_direct_echo's 0.985, within 5% of the ~0.86 kernel floor.

	instr/byte	Gbit/s (3 interleaved reps)
baseline	1.236	11.23 (12.03, 11.19, 11.23)
gathered	0.906	14.60 (15.14, 14.58, 14.60)

Ranges do not overlap. Two X710 guests, 2 workers so the proxy is the bottleneck.

What changed

Mode A wrote one held buffer per write; direct echo has coalesced a drain's worth into one send since #397. A forward write now takes up to 16 held buffers as one sendmsg (sockets — keeping MSG_WAITALL, which a bare writev would lose) or writev (files). Ordering is unchanged: iovecs in hold order, still one write in flight per connection.

A NO-GO on the way, kept in the journal

The first theory was that the cost was the task wake per buffer that direct echo avoids. Moving the forward state into the driver and submitting from the completion handler is correct (proxy gate 0/60) and was ~3% slower, with instructions/byte unchanged at 1.243. That eliminated the scheduler round-trip and left the completion count as the only remaining explanation — which gathering then confirmed. The driver-side state that attempt introduced is kept, because a batch cannot be assembled from inside a future polled once per buffer.

Together they decompose the per-completion cost cleanly: scheduler round-trip ≈ 0, completion count ≈ all of it.

The risky parts, and what guards them

SQE memory must outlive the operation, and here that is three things — the buffers, the iovec array, and the msghdr pointing at it. All live in ForwardWriteState, and the state is installed before pointers into it are taken: moving it afterwards would leave msg_iov's owner dangling even though the iovec heap block survives.

Bid release is now batch-wide — on completion, on submit failure, and in fail_forward_write. This is the new way to break "exactly one replenish per bid", the invariant that cost #415 five bugs, so the tests assert it directly:

• gathered iovec counts (batched, not one-per-write)
• after a short write, the rebuilt iovecs cover exactly the bytes still owed
• a fully-written buffer drops out of the iovec array (the mid-buffer rebuild)
• no bid returns until the batch completes

The 16-buffer cap is not a round number: a batch's bids all return together, so a larger batch delays them and coarsens the hold-cap throttle — and 16 × 16 KiB is already past the ~100 KB where a socket stops having more queued (#416).

It closes #416's one exception

At equal memory the default now measures 14.60 Gbit/s against 15.13 at 64 KiB and 14.22 at 256 KiB — within 3.5% of best, the same order as its deficit on every echo workload. 256 KiB is now worse than the default. So forward_to's sizing advice is retired rather than revised.

Verification

GitHub CI does build and run the io_uring backend (has_io_uring is set on
ubuntu-latest), including the unit tests this change rewrites — so the green
checks are real coverage, not a formality. What it cannot give is repetition
under load on real io_uring hardware, which is what the rack adds.

Authoritative gate, exclusive anvil guest (56 cores, io_uring):

check	result
clippy -D warnings, io_uring	0
clippy -D warnings, mio	0
forward_to_conn proxy regression test	0/100
forward_to_file (the two new tests)	0/100
forward drop-cancel test	0/20 (was 20/20 red — see below)
cargo test --all, io_uring	0/5, 1,122 tests per run
cargo test --all, mio	0/5

100 reps rather than 60, per this PR's own earlier bar: #415's last hang took
3/100 to surface and this change moves bid release from per-write to per-batch.

The spawn_with_handle_abort failure seen in an early gate is not this PR.
A/B against b0e0605 across three environments — 100 baseline full-suite runs
and 100 branch runs — produced no forward-path or file-sink failure on either
arm; the only failures were pre-existing flakes (spawn_with_handle_abort,
and two connect-path tests) at equal rates on both sides.

A regression this PR introduced, found in review and fixed here

Gathering silently voided drop-cancellation on io_uring. Submission moved
from ForwardToFuture::poll into handle_forward_write, so the driver now
advances a forward on its own. Before that, dropping the future stopped the
relay for free. After it, a select! that loses or a timeout that fires
leaves bytes streaming to a sink nobody awaits, with the connection still in
the segmented recv domain — so the caller's next with_data parks while its
data goes to the sink.

Nothing caught it: the only test covering a dropped forward was
#[cfg(not(has_io_uring))], gated off on the premise this change falsified
("on io_uring the writes are driven by polling the future, so dropping it
stops them by itself"). The mio Drop doc carried the same stale claim.
Ungated, the test failed 20/20 on io_uring with
no echo: the bytes went to the sink instead of the handler, and passes
20/20 with the fix.

The fix mirrors mio's, plus one thing the index-keyed driver state forces:

• The in-flight write is deliberately untouched. Its backings are under a
  sendmsg the kernel is still reading; returning those bids to the provided
  ring in Drop would let an arriving packet overwrite a send in progress.
  Its CQE releases them exactly once, and with the progress cleared
  advance_forward submits no successor — landing on the contract mio already
  documents: what is queued completes, no further bytes are taken.
• Held-but-unwritten bytes go to the accumulator via the same
  settle_forward_end the normal end of a forward uses, so a cancel loses no
  data the peer already sent.
• ForwardProgress gains an epoch, so a future dropped after its own
  forward resolved cannot cancel a later one that armed on the same
  connection — the generation-tagging discipline forward_write already uses.

Both stale comments are corrected, and the journal entry records the general
lesson: moving work from a future into a completion handler moves its
cancellation semantics too, and a cfg-gate on a test is an assertion about
behaviour that gets quieter as it gets wronger.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FDHdYwqsnPN4EkhRBvbDgE

[brayniac]

Validation complete on io_uring hardware, rebased onto the merged #418 (21f764f) so it tests what would actually land.

Correctness

check	result
clippy -D warnings (io_uring)	0
proxy regression test	0/100
full suite, repeated	0/15
GitHub checks	18/18

100 reps rather than 60 because #415's last hang surfaced at 3/100, and this change touches the bid lifecycle — the invariant those five bugs were about. The 15 repeated full-suite runs cover parallel contention, which single-test repetition structurally cannot reach (that is how the RSS bug in #419 escaped a rack gate and was caught by GitHub CI).

Performance

Five interleaved reps per arm, two guests, 2 workers so the proxy is the bottleneck:

	reps (Gbit/s)	median
baseline	12.03, 11.19, 11.23, 10.98, 10.95	11.19
gather	15.14, 14.58, 14.60, 14.41, 15.46	14.60

+30.4%, ranges entirely non-overlapping (10.95–12.03 vs 14.41–15.46), and instructions per byte 1.236 → 0.906.

The instructions/byte figure is the one that matters for believing this: it is what falsified the direct-forward attempt (1.236 → 1.243, no change, despite a 3% throughput wobble that could have been read as a win).

Ready for review from my side.