flaky (CI): Test (release) aborts with io_uring_setup ENOMEM on GitHub runners

[brayniac]

Symptom

A test binary aborts before running any test:

ringline: cannot create the test event loop on this host, aborting the test binary so this is reported once:
  ring setup: io_uring_setup(2): Cannot allocate memory (os error 12) (ENOMEM). Fix the underlying
  error, or build with the `force-mio` cargo feature to use the mio backend instead.
error: test failed, to rerun pass `-p ringline --lib`

Every test up to that point passes; the run dies at the point a new test event loop is created. Pre-existing on main and on PRs alike, most often on Test (release).

Ruled out

RLIMIT_MEMLOCK — measured and eliminated. The original version of this issue led with it ("characteristically RLIMIT_MEMLOCK exhaustion"), and that is wrong. The whole uring test suite passes with the soft limit at 8 KiB on kernel 6.12, so the ring is not charged against memlock there. This was recorded only in a comment in .github/workflows/ci.yml, not here, and as a result it has been re-proposed as the cause since. It is not the cause.

Host memory pressure — measured and eliminated. This was the surviving hypothesis, and the reason #474 added a resource-capture step. Those diagnostics have now fired. Run 36534300805, at the moment of the abort:

max locked memory  (kbytes, -l) 8192
Mem:  15988 total, 1041 used, 11511 free, 14947 available
Swap:  3071 total,     0 used

11.5 GiB free, 14.9 GiB available, zero swap used, memlock 8 MiB. Nothing was short of host memory, and no OOM kill appeared in dmesg.

Still open, and now instrumented (#505)

free -m reads host-wide /proc/meminfo, so it is structurally blind to the two remaining candidates. #505 captures both on failure:

• A cgroup memory limit. io_uring charges ring memory to the memcg on modern kernels, so a cgroup limit produces precisely this signature — ENOMEM at io_uring_setup while the host reports gigabytes free. Now capturing memory.max / high / current / peak / events for the root and for the job's own cgroup, with a v1 fallback. memory.events and memory.peak are the load-bearing fields: they are retrospective and survive the death of the process that failed, which memory.current does not.
• vm.max_map_count. io_uring_setup mmaps its rings, and the ceiling appears in neither ulimit -a nor free. ci: capture cgroup memory and the map-count ceiling on failure (#426) #505 records the limit only — the failing process is already gone when the step runs, so its own map count cannot be recovered post-hoc. If the cgroup numbers come back clean, the next step is capturing the map count in-process on the abort path.

/proc/pressure/memory (PSI) is also captured now, since it shows memory stall without an OOM kill.

Note: Test (release) is red for two independent reasons

Of three Test (release) failures on 2026-09-29, one was this ENOMEM, one was buffer_ring_exhaustion_recovers (#378 / #349), and one was both. Fixing this issue alone will not make that job trustworthy.

Why it matters beyond the noise

The abort is indistinguishable from a real failure at a glance — it exits 101 with error: test failed. A reviewer seeing a red Test (release) has to open the log to discover nothing actually failed, and the reflex becomes "rerun it", which is the same reflex that let #423 hide for months behind an opaque timeout.

The runtime's message here is good (it names the cause and the workaround, from #361). The gap is that CI treats an infrastructure failure as a test failure.

[brayniac]

Another occurrence, and an unusually clean one for arguing this is environmental rather than
ours.

• 2026-09-29 22:29Z, Test (release), release/0.6.x (PR release: v0.6.9 #527,
  run 36639638640)
• Same signature: ring setup: io_uring_setup(2): Cannot allocate memory (os error 12),
  after ~600 lib tests had already passed.

The PR it failed on is a version bump. #527 is release: v0.6.9 — ringline/Cargo.toml
0.6.8 → 0.6.9, the matching Cargo.lock line, and a CHANGELOG heading move. It changes no
code at all, so nothing in the diff can affect io_uring_setup. That is worth recording
because it removes the "maybe the change under test provoked it" reading that any normal PR
leaves open.

Two things this occurrence also tells us:

1. It is not main-only. Every prior occurrence in this issue is from main; this is
   the first on a maintenance branch. The 10 most recent release/0.6.x CI runs before this
   one were all green, so it is rare there rather than absent.
2. The failure diagnostics from ci: capture cgroup memory and the map-count ceiling on failure (#426) #505 are not on this branch. release/0.6.x's ci.yml
   has no Resource state (on failure) step, so this occurrence captured no cgroup memory,
   PSI or vm.max_map_count data — exactly the data added to main to characterise this.
   Backporting ci: capture cgroup memory and the map-count ceiling on failure (#426) #505 would make future occurrences on the release line diagnosable; it is
   test-infrastructure only and would fit the maintenance-line rules.

Job re-run to get the release PR green. Recording it here rather than just re-running,
since the whole point of this issue is the occurrence count.

[brayniac]

Another occurrence. This one is the first with the on-failure capture from #505 / #530, so it narrows the cause.

• 2026-10-01, Test (release), main + PR test(runtime): the stale-teardown test checks the sink; set_recv_sink doc above Safety #543 (test and doc only), job 110242021497.
• Where: the ringline lib test binary, after ~600 tests had passed, while one more test was creating its event loop: ringline: cannot create the test event loop on this host … io_uring_setup(2): Cannot allocate memory (os error 12) (ENOMEM).

Measured by the capture step:

	value	reading
cgroup memory.max	max	no cgroup limit
cgroup memory.events	oom 0, oom_kill 0, max 0	the cgroup never hit a limit
cgroup memory.peak	3.1 GB
host	15.9 GB total, 15.0 GB available	not short of memory
PSI memory	all zero	no pressure
kernel log	no OOM lines
ulimit -l (max locked memory)	8192 KB
vm.max_map_count	262144	ceiling only; the captured maps = 0 is the capture step's own shell, not the test process, so the test's map count at failure is unknown

That rules out a cgroup limit and memory pressure for this occurrence.

Correction. An earlier version of this comment named memlock as the remaining candidate. That was wrong: ci.yml already records that memlock was measured and ruled out. On kernel 6.12 the whole uring suite passes with the soft limit at 8 KiB, so the rings aren't charged against it.

Together with the capture above, that leaves both recorded explanations ruled out. Memlock was ruled out on 6.12, and the capture rules out memory pressure: 15 GB available, PSI zero, no OOM.

Hypothesis, not measured: fragmentation. The SQ and CQ rings need high-order (physically contiguous) allocations. Those can fail with ENOMEM while plenty of memory is free, and with __GFP_NOWARN they leave nothing in the kernel log. A failure late in a long, parallel run of many short-lived rings fits that.

Ways to test it:

• add cat /proc/buddyinfo and uname -r to the on-failure capture. Buddyinfo shows free blocks per order; the kernel version also closes the gap that the memlock measurement was on 6.12, while the runner kernel isn't recorded;
• optionally, raise memlock to unlimited in the test jobs (sudo prlimit --pid $$ --memlock=unlimited:unlimited). That's cheap, and a recurrence afterwards would settle memlock on the runner kernel too.