flaky: buffer_ring_exhaustion_recovers times out on GitHub runners (echo never arrives for 1-2 of 8 connections)

[brayniac]

buffer_ring_exhaustion_recovers (ringline/tests/echo.rs, io_uring only) fails intermittently on GitHub-hosted runners with one or two of its eight client connections reading nothing:

assertion `left == right` failed: conn 4 echo mismatch
  left: []
 right: [99, 111, 110, 110, 101, 99, 116, 105, 111, 110, 45, 52, ...]

The echo binary finishes in ~5.7 s, i.e. the client's 5 s read timeout expired (set_read_timeout surfaces as WouldBlock, which the read loop treats as end of data). The server did not close the connection; the echo simply never arrived within 5 s.

Occurrences

• 2026-09-11 16:23Z, main @ fd7c8e6 (refactor(conn): split RecvMode into recv_arm / read / lifecycle #373), job Test (tls-unbuffered): run 34621698481
• 2026-09-12 07:4xZ, PR feat(executor): send-capacity FIFO for bounded sends (series PR 5, no callers yet) #377 (executor-only change), job Test (release), failed twice in a row on the same commit: run 34681419746 attempts 1 and 2
• The same commits passed the same jobs on other attempts / other configurations; main @ 00c69bb passed all jobs at 07:24Z.

Not reproducible on our VMs

Debian 13 (kernel 6.12) anvil guests via systemslab, cargo test --release --test echo buffer_ring_exhaustion_recovers:

host	vCPUs	branch	single-test runs	full echo binary runs
delta	1	main @ 00c69bb	10/10	10/10
delta	1	#377 branch	10/10	10/10
hv01	4	main @ 00c69bb	20/20	5/5
hv01	4	#377 branch	20/20	5/5

Runner image was ubuntu-24.04 version 20260907.300.1 for both the passing (#376) and failing (#377) release runs, so the image is not the variable either.

Test shape

recv_buffer(4, 4096) and 8 concurrent connections, so the provided ring is exhausted by design and the multishot recvs terminate with ENOBUFS, park on recv_starved, and are re-armed by flush_replenish_and_rearm once bids are replenished. A connection that stays parked because nothing replenishes for it would show exactly this symptom. Candidates worth ruling out: a parked connection that is only re-armed "if replenished" in the same pass (a replenish that lands in a later iteration with no starved-connection wake), and the interaction with the runner's kernel scheduling under 4 vCPUs.

Suggested next steps

1. Make the test's failure informative: on timeout, dump the server-side recv_starved, recv_park_count, BUFFER_RING_EMPTY/RECV_PARKED metrics and the provided-ring free count via a test hook, so the next CI failure says whether the connection was still parked.
2. Run the test in a loop on a runner-like environment (a 4-vCPU guest with the Ubuntu 24.04 kernel; anvil supports swapping the guest kernel) to get a local reproduction.
3. Until then, treat a solo failure of this test as a rerun candidate, like signal_wait_on_signal_shutdown.

[brayniac]

Two more occurrences, both Test (release) on release/0.6.x, 2026-09-29:

run	time	commit	connections
36532679267	06:44Z	9119b0df	conn 3, 4, 6, 7
36584818981	14:43Z	4eeb5284	conn 6

Same signature as recorded: left: [] against the expected payload, 73 passed / 1 failed, whole binary finishing in ~5.3 s against the client's 5 s read timeout.

A second test shows the same signature

Worth connecting: run 36592806033 (main @ 91e9df99, job Test) failed chain_guard_send_below_threshold_small_gather in ringline/tests/guard_send_threshold.rs:226 with

read timed out after 5s with 0/144 bytes

Different test, different file, unrelated feature — but the identical failure shape: a 5 s client read timeout expiring with zero bytes received, on io_uring on a GitHub runner. That is what this issue infers indirectly from left: [], stated outright.

Two readings, and this does not distinguish them:

• one root cause (responses not arriving within 5 s under parallel test load on a 2–4 vCPU runner), surfacing in whichever test happens to be reading, or
• two independent problems that share a timeout value.

If it is the former, the interesting number is not the buffer ring at all — it is that a 5 s timeout is too tight for a saturated runner, and every test using it is exposed. chain_guard is the more useful witness because its assertion says what actually happened; #349 is the request to make this test do the same.

Filed separately as #(chain_guard issue) so the two can be compared without merging them prematurely.

[brayniac]

(The separate filing referenced above is #513.)

[brayniac]

Fixed by #415's re-arm gate on main, and by #516 on release/0.6.x (0.6.8). Closing on measurement.

The fix is one condition in flush_replenish_and_rearm: a connection parked on ENOBUFS (recv_starved) is now re-armed if replenished || self.driver.provided_bufs.free() > 0, where before it was re-armed only if replenished in that same pass.

Occurrence history

This covers every ci.yml and release.yml run since 2026-07-01, every attempt including ones later re-run green, and every failed io_uring test job's log. Commits are classified by whether the gate line is in event_loop.rs, not by date.

Code	io_uring job-attempts	failures
park logic without the gate (#245 → #415 on main; release/0.6.x until #516)	1,251	29 (2.3 %/job)
with the gate	833	0

• All 29 occurrences are on gate-less code, including the two on release/0.6.x on 2026-09-29 (9119b0df, 4eeb5284) and three tag releases.
• At 2.3 %/job, 0/833 has probability ≈ 3×10⁻⁹. On main alone: 13/520 before, 0/278 after.
• The test body is byte-identical across the whole window, so no timeout change is involved.
• The runner image was the same for today's failing 0.6.x job and today's clean main jobs: the environment didn't change, the code did.

Mechanism

The failure fits this test's configuration: recv_buffer(4, 4096) shared by 8 one-shot senders.

1. The loop's fast path runs tasks, then ring.flush(). Under DEFER_TASKRUN, that io_uring_enter(GETEVENTS) is where the kernel posts an -ENOBUFS for a connection whose data arrived while the ring was empty.
2. The next pre-block flush commits the bids those tasks returned (replenished = true) before that connection is parked.
3. The following drain parks it.
4. No further bid ever returns, so every later pass has replenished == false and the connection is never re-armed.

The result is zero echo and a connection that is never closed, exactly conn N echo mismatch, left: [] after the 5 s read timeout.

(This is not #415's commit-message path, a fallback completion re-parking the connection. That path is ineligible here: a 20-byte message with an empty accumulator.)

A/B: put the gate back, and the failure comes back

The history can't separate the gate from the rest of #415 (a single squash) or from #414, merged four hours earlier. So:

• both arms built from main;
• the ungated arm changes only that condition back to if replenished {;
• the whole echo binary ran per iteration, release profile, under taskset at 2 and 4 CPUs;
• runner-like Debian 13 guests, 300 iterations per arm per CPU budget.

arm	buffer_ring_exhaustion_recovers	any test failing
ungated (gate reverted, nothing else)	4 / 600, each conn N echo mismatch, left: [] on 1–4 connections	347 / 600
gated (main)	0 / 600	0 / 600

• 4/600 against 0/600 is modest on its own (Fisher one-sided p ≈ 0.06). Here it serves as the positive control: the failure appears only with the gate removed, with everything else in feat(mio): forward_to_conn, and five io_uring forward-path fixes it found #415 and feat(uring): forward_to_conn — name the forward sink as a connection #414 still present.
• The lab rate (≈0.7 %/suite run) is below CI's (≈2.3 %/job), so runners trigger it more readily than these guests.
• Without the gate, main's forwarding tests also fail in most iterations: forward_to_conn_proxies_both_directions 244×, forward_to_file_writes_a_large_body_byte_exact 128×. Forwarding drives the ring into the same park state much harder than this test does, so the gate is load-bearing on that path too.

What rests on what

• Measured: the occurrence table, the A/B.
• Read from code, not observed directly: the exact interleaving above (fast-path flush() posting the ENOBUFS between a replenish and the park).
• Unexplained, and not this bug: flaky: chain_guard_send_below_threshold_small_gather times out reading 0/144 bytes — same 5s signature as #378 #513 (chain_guard, 0/144 bytes) failed once on main with the gate. It stays open separately. There is no evidence here for the "5 s timeout too tight under load" reading. About 19 tests share that timeout. In the logs searched for any zero-byte timeout (every failed job on main and release/0.6.x since 07-06, plus PR runs since feat(mio): forward_to_conn, and five io_uring forward-path fixes it found #415), this test produced every such failure except flaky: chain_guard_send_below_threshold_small_gather times out reading 0/144 bytes — same 5s signature as #378 #513's, and they all stopped with the fix.

If this signature reappears on code that has the gate, it's a new bug; please open a fresh issue rather than reopening this one.

_{🤖 Analysis by Claude Code. Occurrence history independently recomputed by a second agent.}