Adopt recv bundles (IORING_RECVSEND_BUNDLE) + incremental provided-buffer rings for large responses

[brayniac]

Problem

When a single response spans multiple provided buffers, the multishot recv path delivers one
buffer per CQE, and — because parsing clients need the reply contiguous — the segments are
memcpy-stitched back together in the per-connection accumulator. Two costs:

1. CQE amplification — a 64 KiB response over 16 KiB buffers = 4 CQEs + 4 buffer consumptions.
2. Accumulator stitching — fragments copied into one contiguous region for the parser.

Under concurrency the provided-buffer ring is also easily exhausted (N concurrent large responses
need N × ceil(resp/buf) buffers), dropping to the slow ENOBUFS fallback path.

Measured impact (cachecannon → Datomic valcache, 64 KiB GET, 10GbE, kernel 6.12)

• Default geometry (256 × 16 KiB): throughput dropped as connections rose (64c 8.1 Gbps →
  256c 7.0 Gbps) — buffer-ring starvation; client CPU was 321% system (kernel recv path).
• Enlarging only the ring (16 KiB buffers, ring 512/1024): barely helped (8.15 Gbps) — so the cost
  is the per-response stitching, not just ring exhaustion.
• One buffer per response (value-sized buffer, ring 256): 8.9 Gbps, cliff gone — but this wastes
  memory (one big buffer per response) and only helps clients that know their response size upfront.

Proposal

Use the modern io_uring recv facilities so small, packed buffers deliver large responses with one CQE
and no stitching:

• IORING_RECVSEND_BUNDLE (kernel ≥ 6.10): a bundled multishot recv consumes as many ring
  buffers as the data needs and reports the run in one CQE (total length + the span of buffer
  ids). Since a provided-buffer ring is one contiguous mmap, consecutive ids are adjacent in memory,
  so the bundled region is already contiguous — the parser reads it with no stitch.
• IOU_PBUF_RING_INC (incremental consumption, kernel ≥ 6.11): the kernel advances the ring head
  only by the bytes used, so a large recv can partially fill a buffer without wasting the tail.

Together: the memory efficiency of small packed buffers and one-CQE / zero-copy delivery —
removing both the CQE amplification and the stitching.

Sketch

• submit_multishot_recv / submit_multishot_recvmsg set the bundle flag (feature-probed; fall back
  to one-buffer-per-CQE on older kernels or the mio backend).
• Completion handling consumes a run of bids per CQE (contiguous region) instead of a single bid.
• Register the buf ring with the INC flag where supported; extend the existing kernel probe
  (error.rs) for bundle + incremental.

Benefits every large-response protocol (valcache, large Redis values, the Mode-A recv-forward proxy
path), not just memcache.

Motivation and measurements from adding memcache-binary support to cachecannon; see the
memcache-binary client work.

🤖 Generated with Claude Code