Repository navigation
Respect the live parameter budget when prefetching ZeRO-3 weights - #9125
Open
YimingQiao wants to merge 2 commits into
Open
YimingQiao wants to merge 2 commits into
YimingQiao wants to merge 2 commits into
Conversation
Signed-off-by: Yiming Qiao <yimingqiao3163@gmail.com>
YimingQiao
requested review from
loadams,
tjruwase and
tohtana
as code owners
November 19, 2026 08:20
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Signed-off-by: Yiming Qiao <yimingqiao3163@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ZeRO-3 checks how many elements it has already selected before adding the next whole parameter to a prefetch. On master (
b5e0ba6), with a prefetch bucket of 32 elements andstage3_max_live_parameters=48, weights of 16 and 64 elements are both gathered ahead of use. The second gather takes the batch to 80 elements.Check the candidate's size against the remaining live-parameter budget before marking its trace entry as fetched. If it would exceed the budget, leave it at the front of the queue and end the batch. A layer can still gather its own weight on demand, including a weight larger than the budget.
A local two-rank reproduction observed both weights before the first layer runs. At a budget of 48, master prefetches 80 elements and the patch prefetches 16. At the exact-fit budget of 80, both prefetch 80. In both patched cases, losses over four SGD steps and the final weights match an unsharded PyTorch reference within test tolerances.
This reproduces with a fixed bucket. #8193 and #8431 cover adapting the bucket size at runtime. The affected parameter-prefetch loop belongs to eager ZeRO-3; ZeRO-1/2 use a separate optimizer path.
Validation
Two Gloo ranks on one dual-socket Intel Xeon Platinum 8474C host, Python 3.12.12, PyTorch 2.8.0+cpu. The local run exercised FP32 data parallelism with parameter and optimizer offload disabled.
Both local reproduction cases passed. The four existing CPU test files passed all 31 cases covering gather-handle ordering, padded parameter reconstruction, zero-sized parameters and ZeRO configuration.
export DS_ACCELERATOR=cpu LOCAL_SIZE=2 OMP_NUM_THREADS=1 pytest -q tests/unit/runtime/zero/{test_zero_stale_handle_wait,test_zero_allreduce_fetch_params,test_zero_numel_param_shape,test_zero_config}.py pre-commit run --files deepspeed/runtime/zero/partitioned_param_coordinator.pyGPU peak allocation, training throughput and long-run convergence are unmeasured. Mixed precision, offload, hybrid parallelism and compiled execution were outside the focused regression. Deferring a gather can reduce communication overlap when the live budget is tight.
AI assistance was used to develop the patch, run the tests and draft this description.