Skip to content

fix(pipeline): charge Grain's pending deletions to the worker plan - #123

Merged
mahdi-shafiei merged 1 commit into
mainfrom
fix/shm-backlog-in-the-read-plan
Nov 18, 2026
Merged

mahdi-shafiei merged 1 commit into
mainfrom
fix/shm-backlog-in-the-read-plan

Conversation

@mahdi-shafiei

@mahdi-shafiei mahdi-shafiei commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

 worker plan (read_plan._worker_plan)
-  RAM:      M_main + O + C + (d + 1) E + k (F + P + r E + S) + k (2o + 2) E
-  /dev/shm: k (2o + 2) E + (d + 1) E
+  RAM:      M_main + O + C + (d + 1) E + ceil(M / L) E + k (F + P + r E + S) + k (2o + 2) E
+  /dev/shm: k (2o + 2) E + (d + 1) E + ceil(M / L) E
  • M (DELETION_BACKLOG_SEGMENTS): the segments of dropped units Grain lets wait for its
    deletion thread to close and unlink, read from SharedMemoryArray.enable_async_del's
    signature (max_outstanding_requests=50). Grain applies the first call in a process and makes
    it with the defaults; datarax never sets it.
  • L (HostTerms.unit_segments): the segments one unit takes, one per leaf Grain copies to
    shared memory. A pipeline's is the spec's non-empty leaves and the three identity arrays
    (host_unit_segments), and the build's is its first unit's (sent_segments): its columns, or
    none for a decoded copy.
  • Both refusals name the term.

Evidence

  • Before: test_steady_state_segments_are_the_units_the_queues_hold failed on main
    3901626 with (67, 65) and on the 2026-11-17 scheduled run with (69, 65). Sampling
    /dev/shm continuously over 400 batches (2 workers, o = 2, d = 0, 5 leaves):

    Grain's async deletion samples over 65 highest
    on 417 20 113
    off 692 0 65

    In 18 of the 20 over-bound samples, 47 to 50 deletions were pending, and the training process
    alone mapped the extra names. Two had none pending, 1 and 48 over the bound; the 48 is not
    explained (the names and Grain's pending count are separate reads).

  • After: the lifetime bound is the plan's: (k (2o + 2) + d + 1) L + M segments. The test now
    checks the plan's L against the leaves of a real unit, which is 5: 2 data leaves and 3 identity
    arrays.

  • tests/pipeline: 2272 passed, 10 skipped. tests/core with the contract and hygiene tests:
    1346 passed. pre-commit passed (pyright included).

Merge Danger

Door: two-way

Blast Radius: worker plans

Worker plans get smaller or are refused sooner where /dev/shm or the RAM budget is tight. With
two leaves a unit, the term is 25 units, against 2o + 2 = 6 a worker in the queues; on Docker's
default 64 MiB /dev/shm it will often decide the plan. This is from the formula, not measured
on a real pipeline. Thread plans are unchanged. No sibling uses the read plan (0 lines in nine
repositories). A process that enables Grain's async deletion first with a larger bound is still
undercounted; a Grain issue and PR for a public read of the bound in effect is drafted for
review.

The training process unlinks each segment a worker sends once the unit
holding it is dropped, and Grain hands that close and unlink to one
thread, letting max_outstanding_requests (50) wait. Those segments stay
in /dev/shm and RAM, so a worker plan now charges ceil(M / L) units once
per process to both bounds: M read from enable_async_del's signature,
L the leaves a unit puts in shared memory (HostTerms.unit_segments).

Sampled continuously over 400 batches of 2 workers and 5 leaves,
/dev/shm held up to 113 segments against the old bound of 65 with 47 to
50 deletions pending, and at most 65 with deletion synchronous; the
lifetime test failed with (67, 65) and (69, 65) in CI. Its bound and the
materialize segment test's include the term now.
@mahdi-shafiei
mahdi-shafiei marked this pull request as ready for review November 18, 2026 22:59
@mahdi-shafiei
mahdi-shafiei merged commit 1bd7e18 into main Nov 18, 2026
30 checks passed
@mahdi-shafiei
mahdi-shafiei deleted the fix/shm-backlog-in-the-read-plan branch November 18, 2026 22:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant