Repository navigation
ZeRO-3: take the AutoTP shard from universal checkpoints converted without TP - #9069
Conversation
…thout TP ds_to_universal writes each state of a ZeRO-3 checkpoint saved without TP as a bare tensor, while every other path writes a dict. The ZeRO-3 loader resolved the AutoTP shard only for a dict, so loading such a checkpoint into an AutoTP run partitioned the whole weight and failed. _resolve_autotp_partition reads only the target parameter's metadata and the loaded tensor, so let the target decide. Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com>
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
A real-model check of the same change: Qwen3-0.6B, bf16, 4× A100 (driver 595.71.05, torch 2.12.1+cu130, NCCL 2.29.7), DP4 → TP2×DP2 → DP4 under ZeRO-3, 20 steps per layout. Each change saves, runs
|
…time (deepspeedai#8844) # Universal Checkpointing: read AutoEP ZeRO-3 optimizer files one at a time ## Problem `ds_to_universal` converts a ZeRO-3 AutoEP checkpoint in two parts. The dense parameters go through the usual extract-and-merge path, which reads one optimizer file per worker and writes fragments to `tmp/`. The routed experts go through `_consolidate_zero3_autoep_expert_states`, which first loads **every** rank's optimizer file into one dict (`optim_states_by_rank`) and then keeps a copy of every expert fragment in memory until all of them are merged. Together those files hold every optimizer state of the model: fp32 (or bf16) master weights plus both Adam moments. For a large MoE model that is more than one host has: | Model | Parameters | Optimizer state, fp32 master + fp32 Adam (12 B/param) | bf16 master + bf16 Adam (6 B/param) | |---|---|---|---| | MiniMax-M3 | 426.2B | 4.65 TiB | 2.33 TiB | | GLM-5.2 | 743.4B | 8.11 TiB | 4.06 TiB | The H200 nodes this was found on have 2 TiB of host memory, so the conversion cannot finish for either model. ## Change `_consolidate_zero3_autoep_expert_states` reads one rank's optimizer file at a time: - Each expert fragment it finds is written to `<output>/tmp/autoep_expert_fragments/` (one file per parameter, optimizer state, EP rank and partition rank). - Every reference into the rank's optimizer state, including the loop variables, is dropped before the next file is read. - The merge then loads, for one parameter and one state at a time, only that parameter's fragments, and deletes each fragment file once it is read, so the disk does not hold every fragment and every universal file at the same time. Peak host memory becomes one optimizer file plus one full expert tensor, instead of all optimizer files. The checks are unchanged: missing ranks, duplicate or missing expert-data-parallel partitions, shape mismatches and placement disagreements raise the same errors. The temporary fragments live under `<output>/tmp`, which the converter already uses for the dense fragments and removes at the end. ## Peak host memory Measured with a probe on one checkpoint: a 4-layer mock MoE model (64 experts, hidden 1024, expert intermediate 4096: 3.22B expert parameters) trained one step and saved with ZeRO-3 and AutoEP EP 4 on 8 GPUs (bf16, Adam), giving 8 optimizer files of 36.19 GiB in total. Each version ran only `_consolidate_zero3_autoep_expert_states`, in its own process; peak resident memory is `ru_maxrss` (0.4 GiB of it is the imports). | | Peak host memory | Time | Output | |---|---|---|---| | master 3fe2d95 | 109.20 GiB | 93.6 s | | | this change | 13.13 GiB | 203.6 s | all 36 universal files (fp32, exp_avg, exp_avg_sq of each expert weight) bitwise equal to master's | Master holds about 3 times the optimizer files at once; with this change the peak is about one optimizer file plus one parameter's tensors, and it does not grow with the number of ranks. The cost is time: the fragments are written to and read back from `<output>/tmp`, 2.2x slower here on a shared Lustre file system. For MiniMax-M3 and GLM-5.2 the old path cannot finish at all on a 2 TiB host. Hardware: 1 node of 8 NVIDIA H100 80GB (driver 595.112.07), torch 2.8.0+cu126. Probe script: `ucp_expert_memory_probe.py` (modes `save`, `convert`, `compare`). ## Tests On master 3fe2d95 with this change, 1 node of 8 NVIDIA H100 80GB (driver 595.112.07), torch 2.8.0+cu126, pytest 8.3.5: | Test | Result | |---|---| | `tests/unit/checkpoint/test_autoep_affine_universal.py` (calls the function directly) and `tests/unit/v1/moe/test_autoep_checkpoint.py` (AutoEP ZeRO-3 save, convert, load at other layouts) | 69 cases: 67 passed, 2 skipped (ZeRO-1/2 universal-checkpoint cases that run only with `DEEPSPEED_RUN_H100_TESTS=1`; this change is on the ZeRO-3 path) | `pre-commit run --files` passes on the changed file. ## Duplicate check Searched open pull requests on 2026-11-13 ("ds_to_universal", "universal checkpoint AutoEP"). deepspeedai#9069 and deepspeedai#8973 change how ZeRO-3 loads universal checkpoints and partial-offload checkpoints, not `ds_to_universal.py`. deepspeedai#8848 (mine) also edits `_consolidate_zero3_autoep_expert_states` (keys by tensor-parallel and data-parallel rank); whichever of the two merges second needs a small rebase. ## Compatibility | Cell | Validated | |---|---| | ZeRO-3 + AutoEP checkpoint conversion, save / convert / load at other layouts | yes (tests) | | Dense (non-AutoEP) ZeRO-3 conversion | path unchanged | | ZeRO-1 / ZeRO-2 conversion | path unchanged | ## AI assistance This change, its tests and this description were prepared with AI assistance (Claude). --------- Signed-off-by: pengdurice <pengduhit@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A ZeRO-3 checkpoint saved without TP can't be resumed in an AutoTP + ZeRO-3 run through the universal checkpoint.
Why
ds_to_universalwrites each state of a ZeRO-3 checkpoint saved without TP as a bare tensor (merge_zero3_slices); every other path writes a dict with the tensor underparam.DeepSpeedZeroOptimizer_Stage3.load_hp_checkpoint_statetakes the AutoTP shard only when the file holds a dict. With a bare tensor it partitions the whole weight as if it were this rank's TP shard, and the copy into the flat buffer fails:_resolve_autotp_partitioncuts the shard using the target parameter's AutoTP metadata and the loaded tensor, and never reads the dict, so the check isn't needed.What
The target parameter alone decides whether to slice (
if param is not None:). Checkpoints converted before this keep loading, since the loader accepts both formats as before.Whether every conversion path should write the same format is left for later, as discussed with @delock: the loader would still have to accept bare tensors from checkpoints that were already converted.
No open PR touches this check (searched for
load_hp_checkpoint_stateand ZeRO-3 universal AutoTP; #8848 changes the converter, notstage3.py).Test
TestZero3UniversalCheckpointWithoutTpintest_autotp_uc_checkpoint.pytrains 4 steps under ZeRO-3 without TP, converts, and loads into ZeRO-3 at TP1 and at AutoTP TP2. It compares the fp32 weights, both Adam moments and the step counts with the saved state bit for bit, since conversion and restore only copy. On master the TP2 case fails with the error above; the TP1 case passes before and after._affine_resume_enginegains azero_stageargument (default 1) so the test reuses the existing resume model and helpers.Validation
pytest --forked tests/unit/checkpoint/test_autotp_uc_checkpoint.py tests/unit/checkpoint/test_universal_checkpoint.py tests/unit/checkpoint/test_autoep_affine_universal.py: 217 passed with the fix. With only thestage3.pychange reverted, the new TP2 case fails with the error above and the TP1 case passes.DS_ACCELERATOR=cpu LOCAL_SIZE=4): the same result for the new test, before and after. The FusedAdam tests in these files can't build the op on macOS, so they ran on GPU only.ds_to_universaland loads in the new layout, then compares every fp32 weight and both Adam moments with the saved state bit for bit. On master the TP2×DP2 load fails withstart (187907840) + length (1048576) exceeds dimension size (187924480); with the fix both changes restore all 930 tensors bit for bit.Checked and not changed
universal_checkpoint.load_hp_checkpoint_state) readsckpt_dict[PARAM]unconditionally, but stage 1/2 conversion always writes the dict, and a ZeRO-3 universal checkpoint doesn't reach that loader today: a ZeRO-1 engine looks formp_rank_00_model_states.pt, while the converted ZeRO-3 folder haszero_pp_rank_0_mp_rank_00_model_states.pt, so the load stops instate_dict_factory.check_ckpt_list.Validated: ZeRO-3 without offload; fp32 (the unit test) and bf16 (Qwen3-0.6B); TP1 → TP1, TP1 → TP2, and DP4 → TP2×DP2 → DP4. Not validated: CPU or NVMe offload, fp16, pipeline parallelism.