Skip to content

Respect the live parameter budget when prefetching ZeRO-3 weights - #9125

Open
YimingQiao wants to merge 2 commits into
deepspeedai:masterfrom
YimingQiao:zero3-prefetch-live-budget
Open

YimingQiao wants to merge 2 commits into
deepspeedai:masterfrom
YimingQiao:zero3-prefetch-live-budget

Conversation

@YimingQiao

@YimingQiao YimingQiao commented Nov 19, 2026 •

Copy link
Copy Markdown

ZeRO-3 checks how many elements it has already selected before adding the next whole parameter to a prefetch. On master (b5e0ba6), with a prefetch bucket of 32 elements and stage3_max_live_parameters=48, weights of 16 and 64 elements are both gathered ahead of use. The second gather takes the batch to 80 elements.

Check the candidate's size against the remaining live-parameter budget before marking its trace entry as fetched. If it would exceed the budget, leave it at the front of the queue and end the batch. A layer can still gather its own weight on demand, including a weight larger than the budget.

A local two-rank reproduction observed both weights before the first layer runs. At a budget of 48, master prefetches 80 elements and the patch prefetches 16. At the exact-fit budget of 80, both prefetch 80. In both patched cases, losses over four SGD steps and the final weights match an unsharded PyTorch reference within test tolerances.

This reproduces with a fixed bucket. #8193 and #8431 cover adapting the bucket size at runtime. The affected parameter-prefetch loop belongs to eager ZeRO-3; ZeRO-1/2 use a separate optimizer path.

Validation

Two Gloo ranks on one dual-socket Intel Xeon Platinum 8474C host, Python 3.12.12, PyTorch 2.8.0+cpu. The local run exercised FP32 data parallelism with parameter and optimizer offload disabled.

Both local reproduction cases passed. The four existing CPU test files passed all 31 cases covering gather-handle ordering, padded parameter reconstruction, zero-sized parameters and ZeRO configuration.

export DS_ACCELERATOR=cpu LOCAL_SIZE=2 OMP_NUM_THREADS=1
pytest -q tests/unit/runtime/zero/{test_zero_stale_handle_wait,test_zero_allreduce_fetch_params,test_zero_numel_param_shape,test_zero_config}.py
pre-commit run --files deepspeed/runtime/zero/partitioned_param_coordinator.py

GPU peak allocation, training throughput and long-run convergence are unmeasured. Mixed precision, offload, hybrid parallelism and compiled execution were outside the focused regression. Deferring a gather can reduce communication overlap when the live budget is tight.

AI assistance was used to develop the patch, run the tests and draft this description.

Signed-off-by: Yiming Qiao <yimingqiao3163@gmail.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T08:27:38.214670Z 815bf53 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 19, 2026
Signed-off-by: Yiming Qiao <yimingqiao3163@gmail.com>
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant