Repository navigation
Conversation
A universal checkpoint converted from ZeRO-3 could not be loaded into ZeRO-1/2, and one converted from ZeRO-1/2 could not be loaded into ZeRO-3: - the engine looked only for the model-states file named by its own stage; - the ZeRO-3 converter dropped ds_version, which the ZeRO-1/2 loader asserts on; - the ZeRO-1/2 loader expected dict-wrapped states, but the ZeRO-3 converter writes bare flat tensors; - the step was read from where the target's own stage keeps it. Fall back to the other stage's file name on universal loads, copy ds_version in the converter, accept flat tensors when they fill the whole parameter, and read the step and param_groups from either layout. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2bec47662e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| """ | ||
| state = optim_sd.get(OPTIMIZER_STATE_DICT, {}).get('state') | ||
| if state: | ||
| return next(iter(state.values())).get('step') |
There was a problem hiding this comment.
Give each restored state an independent step tensor
When a cross-stage checkpoint was written by torch Adam (so step is a tensor) and the target optimizer has more than one group/subgroup, this returns one object that base_optimizer.py:405-408 caches across all groups and stage3.py:3752,3770 assigns across all subgroups. Adam increments each state's step in place, so a single resumed optimizer step advances this aliased scalar once per group/subgroup (for example, 2 to 4 for two groups) and uses incorrect bias correction; clone the value for each state before storing it.
AGENTS.md reference: AGENTS.md:L19-L19
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in ce758ab. Both universal loaders now give every group and sub-group its own copy of the step, so one optimizer step advances each state once. The cross-stage test splits ZeRO-3 into sub-groups and checks that every step goes up by exactly one after a step following the load.
| for key, value in optim_sd[OPTIMIZER_STATE_DICT]["param_groups"][0].items(): | ||
| # A ZeRO-1/2 source has no OPTIMIZER_STATE_DICT, only the top-level PARAM_GROUPS. | ||
| param_groups = optim_sd.get(PARAM_GROUPS) or optim_sd[OPTIMIZER_STATE_DICT][PARAM_GROUPS] | ||
| for key, value in param_groups[0].items(): |
There was a problem hiding this comment.
Restore the corresponding ZeRO-3 optimizer group
When a ZeRO-1/2 source uses multiple optimizer parameter groups (for example, separate decay/no-decay groups or learning rates), PARAM_GROUPS contains one saved mapping per group, but this loop applies only param_groups[0] to every ZeRO-3 target group. The moments are loaded by parameter name while later groups silently inherit group 0's lr, weight_decay, and other settings, changing the trajectory immediately after resume; iterate target and saved groups together as the ZeRO-1/2 loader does.
AGENTS.md reference: AGENTS.md:L19-L19
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in ce758ab. The ZeRO-3 loader now restores each optimizer group from the saved group with the same index, and raises if the group counts differ. The test uses two parameter groups with different settings and checks that both are restored.
The ZeRO-3 universal loader applied the first saved parameter group to every optimizer group; it now restores each group from the saved group with the same index and raises if the counts differ. Both universal loaders gave every group or sub-group the same step tensor, which the optimizer advances in place once per group; each state now gets its own copy. The cross-stage test uses two parameter groups with different settings and a sub_group_size that splits ZeRO-3's groups, checks the restored settings, and checks that one step after the load advances every state's step once. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
|
@jinyouzhi, as offered: a data point for the TP/PP cell of "Not validated cross-stage". A tensor- and pipeline-parallel ZeRO-1 source, loaded into ZeRO-3 through a universal checkpoint. Setup
Results
The pipeline engine writes one model-states file per stage and TP rank ( To check that the comparison can fail:
Script (run from the repository root:
|
The pipeline engine runs at most ZeRO-1, so a pipeline-parallel run reaches ZeRO-3 only through a universal checkpoint loaded across stages. The new test trains Megatron-style column- and row-parallel layers with TP=2 and PP=2 under ZeRO-1, converts the checkpoint, and loads it as ZeRO-3 over 4 data-parallel ranks. It checks that every weight and Adam moment matches the source's reassembled tensors bit for bit, that the settings and step are restored, and that the target keeps training. On master the load fails to find the model states file. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
After the save, the TP2 x PP2 source keeps training for 3 steps. The ZeRO-3 target follows on the same micro-batches, each data-parallel rank taking one of the source's pair, and must match the source's loss, weights and both Adam moments after every step within fp32 tolerance. Rank 0 now shares the outcome of the conversion, so a converter failure fails every rank instead of leaving the others waiting at a barrier. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
A universal checkpoint converted from a ZeRO-3 run cannot be loaded into ZeRO-1/2, and one converted from a ZeRO-1/2 run cannot be loaded into ZeRO-3. This PR makes both directions load, with the fp32 master weights, Adam states and step restored exactly.
Reproduction
Train under one stage,
save_checkpoint, runds_to_universal, thenload_checkpoint(..., load_optimizer_states=True)withcheckpoint.load_universal: trueunder the other stage. On master a75ab89 each gap below surfaces once the previous one is worked around:assert len(self.ckpt_list) > 0_get_all_ckpt_namesglobs only the current stage's model-states name:zero_pp_rank_0_mp_rank_*_model_states.ptfor ZeRO-3,mp_rank_*_model_states.ptfor ZeRO-1/2. A universal load reads only metadata from this file, so either name works.AssertionError: Empty ds_version_save_optimizer_state_stage3writes the inner optimizer dict, but ZeRO-3 keepsds_versionbeside it.DeepSpeedZeroOptimizer._load_global_stateasserts on it.IndexError: too many indices for tensor of dimension 1merge_zero3_slices) writes each state as a bare flat tensor;load_hp_checkpoint_stateindexesckpt_dict[PARAM]. The stage-3 loader already accepts both forms.step.pt; its step is inoptimizer_state_dict.state.KeyError: 'optimizer_state_dict'param_groupsfromoptimizer_state_dict, which a ZeRO-1/2 source does not have (it has top-levelparam_groupsand per-parameterstep.pt).Changes
engine._get_all_ckpt_names: on a universal load with no match, glob the other stage's name, built by_get_ckpt_name(new optionalpartition_weightsargument).ds_to_universal._save_optimizer_state_stage3: copyds_versionintozero/optimizer_state.pt.universal_checkpoint.py:load_hp_checkpoint_stateviews a bare flat tensor as the parameter's shape. A flat tensor carries no TP layout, so if it does not fill the whole parameter (ZeRO-3 source → ZeRO-1/2 with TP>1) it raises aValueErrorsaying so, instead of anIndexError.load_universal_optimizer_stepreads the step from either layout. It is used by the ZeRO-1/2 path (ZeROOptimizer.load_hp_checkpoint_state_from_checkpoint_dir, when nostep.ptexists) and by the ZeRO-3 path.stage3.load_hp_checkpoint_state_from_checkpoint_dir_stage3: takeparam_groupsfrom the top level when present (the ZeRO-3 converter has always written it there too).Same-stage loads take the same paths as before: every new branch is reached only when the old lookup finds nothing.
Sibling paths checked:
BF16_Optimizersharesload_hp_checkpoint_state_from_checkpoint_dirand its_load_global_stateis a no-op, so it gets fixes 3 and 4 and is unaffected by 2.stage_1_and_2andstage3are both patched.Not a duplicate
No existing issue or PR covers cross-stage universal loading. #9069 also touches bare flat tensors but fixes the ZeRO-3 + AutoTP load path, not ZeRO-1/2 targets. #7546, #8848 and #8898 are different bugs.
Tests
New
TestUniversalCheckpointAcrossZeROStagesintests/unit/checkpoint/test_universal_checkpoint.py(world size 2, fp32,torch.optim.Adam), parametrized over (1→3), (2→3), (3→1), (3→2). It trains 2 steps, saves, converts, loads into the other stage, then checks:exp_avgandexp_avg_sqof every parameter aretorch.equalto the source (the conversion only moves fp32 data, so the comparison is exact);New
TestUniversalCheckpointPipelineToZeRO3(world size 4): the pipeline engine runs at most ZeRO-1, so a pipeline-parallel run reaches ZeRO-3 only this way. It trains Megatron-style column- and row-parallel layers with TP=2 × PP=2 under ZeRO-1, converts, and loads as ZeRO-3 over 4 data-parallel ranks. It checks that every weight and Adam moment istorch.equalto the source's TP blocks put back together and that the learning rate and step are restored. It then trains the target 3 more steps on the micro-batches the source used after its save, and after every step compares the loss, weights and both moments with the source run that never stopped (assert_close, fp32 tolerance: the TP all-reduces sum in a different order).Hardware: Intel Arc Pro B70 (XPU), Linux 6.19.10 in-tree
xedriver, compute runtime 26.27.39122.11, torch 2.14.0.dev20260804+xpu. Run from the repo root.assert len(self.ckpt_list) > 0)TestUniversalCheckpointPipelineToZeRO3on 4× L20: fails on master a75ab89 at the same assert, passes with this PR.Regression on B70, master and this PR in the same environment:
TestUniversalCheckpointAcrossZeROStagescases.TestZeROUniversalCheckpointDPwith ZeRO-1 andload_optim=True: the bit-exact optimizer-state check fails intermittently on this XPU setup. Which cases fail changes from run to run; re-running 7 of them gave 5/7 failures with this PR and 4/7 and 5/7 on master.TestRealCheckpointUniversalConversionTPxPP:No backend type associated with device type xpu.End to end on B70 (2 ranks): Qwen3-0.6B, bf16,
torch.optim.AdamW. Train 2 steps under one stage, save,ds_to_universal, load under the other stage, and comparesafe_get_full_fp32_param/safe_get_full_optimizer_statebefore and after:Validated matrix cells
Limitations
ds_version, so loading it into ZeRO-1/2 still fails the version assert; re-runds_to_universalon the original checkpoint.AI assistance
This PR was prepared with AI assistance (GitHub Copilot CLI), covering the root-cause analysis, the code, the tests and this description. I reviewed the changes and ran the tests reported above on the hardware listed.