Repository navigation
fix(pipeline): charge Grain's pending deletions to the worker plan - #123
Merged
Merged
Conversation
The training process unlinks each segment a worker sends once the unit holding it is dropped, and Grain hands that close and unlink to one thread, letting max_outstanding_requests (50) wait. Those segments stay in /dev/shm and RAM, so a worker plan now charges ceil(M / L) units once per process to both bounds: M read from enable_async_del's signature, L the leaves a unit puts in shared memory (HostTerms.unit_segments). Sampled continuously over 400 batches of 2 workers and 5 leaves, /dev/shm held up to 113 segments against the old bound of 65 with 47 to 50 deletions pending, and at most 65 with deletion synchronous; the lifetime test failed with (67, 65) and (69, 65) in CI. Its bound and the materialize segment test's include the term now.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
M(DELETION_BACKLOG_SEGMENTS): the segments of dropped units Grain lets wait for itsdeletion thread to close and unlink, read from
SharedMemoryArray.enable_async_del'ssignature (
max_outstanding_requests=50). Grain applies the first call in a process and makesit with the defaults; datarax never sets it.
L(HostTerms.unit_segments): the segments one unit takes, one per leaf Grain copies toshared memory. A pipeline's is the spec's non-empty leaves and the three identity arrays
(
host_unit_segments), and the build's is its first unit's (sent_segments): its columns, ornone for a decoded copy.
Evidence
Before:
test_steady_state_segments_are_the_units_the_queues_holdfailed on main3901626with(67, 65)and on the 2026-11-17 scheduled run with(69, 65). Sampling/dev/shmcontinuously over 400 batches (2 workers,o = 2,d = 0, 5 leaves):In 18 of the 20 over-bound samples, 47 to 50 deletions were pending, and the training process
alone mapped the extra names. Two had none pending, 1 and 48 over the bound; the 48 is not
explained (the names and Grain's pending count are separate reads).
After: the lifetime bound is the plan's:
(k (2o + 2) + d + 1) L + Msegments. The test nowchecks the plan's
Lagainst the leaves of a real unit, which is 5: 2 data leaves and 3 identityarrays.
tests/pipeline: 2272 passed, 10 skipped.tests/corewith the contract and hygiene tests:1346 passed. pre-commit passed (pyright included).
Merge Danger
Door: two-way
Blast Radius: worker plans
Worker plans get smaller or are refused sooner where
/dev/shmor the RAM budget is tight. Withtwo leaves a unit, the term is 25 units, against
2o + 2 = 6a worker in the queues; on Docker'sdefault 64 MiB
/dev/shmit will often decide the plan. This is from the formula, not measuredon a real pipeline. Thread plans are unchanged. No sibling uses the read plan (0 lines in nine
repositories). A process that enables Grain's async deletion first with a larger bound is still
undercounted; a Grain issue and PR for a public read of the bound in effect is drafted for
review.