Skip to content

Load universal checkpoints across ZeRO stages - #9080

Draft
jinyouzhi wants to merge 4 commits into
deepspeedai:masterfrom
jinyouzhi:uc-cross-stage
Draft

jinyouzhi wants to merge 4 commits into
deepspeedai:masterfrom
jinyouzhi:uc-cross-stage

Conversation

@jinyouzhi

@jinyouzhi jinyouzhi commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

A universal checkpoint converted from a ZeRO-3 run cannot be loaded into ZeRO-1/2, and one converted from a ZeRO-1/2 run cannot be loaded into ZeRO-3. This PR makes both directions load, with the fp32 master weights, Adam states and step restored exactly.

Reproduction

Train under one stage, save_checkpoint, run ds_to_universal, then load_checkpoint(..., load_optimizer_states=True) with checkpoint.load_universal: true under the other stage. On master a75ab89 each gap below surfaces once the previous one is worked around:

# Direction Error Root cause
1 both assert len(self.ckpt_list) > 0 _get_all_ckpt_names globs only the current stage's model-states name: zero_pp_rank_0_mp_rank_*_model_states.pt for ZeRO-3, mp_rank_*_model_states.pt for ZeRO-1/2. A universal load reads only metadata from this file, so either name works.
2 3 → 1/2 AssertionError: Empty ds_version _save_optimizer_state_stage3 writes the inner optimizer dict, but ZeRO-3 keeps ds_version beside it. DeepSpeedZeroOptimizer._load_global_state asserts on it.
3 3 → 1/2 IndexError: too many indices for tensor of dimension 1 The ZeRO-3 converter (merge_zero3_slices) writes each state as a bare flat tensor; load_hp_checkpoint_state indexes ckpt_dict[PARAM]. The stage-3 loader already accepts both forms.
4 3 → 1/2 step silently not restored A ZeRO-3 source writes no per-parameter step.pt; its step is in optimizer_state_dict.state.
5 1/2 → 3 KeyError: 'optimizer_state_dict' The stage-3 loader reads the step and param_groups from optimizer_state_dict, which a ZeRO-1/2 source does not have (it has top-level param_groups and per-parameter step.pt).

Changes

  • engine._get_all_ckpt_names: on a universal load with no match, glob the other stage's name, built by _get_ckpt_name (new optional partition_weights argument).
  • ds_to_universal._save_optimizer_state_stage3: copy ds_version into zero/optimizer_state.pt.
  • universal_checkpoint.py:
    • load_hp_checkpoint_state views a bare flat tensor as the parameter's shape. A flat tensor carries no TP layout, so if it does not fill the whole parameter (ZeRO-3 source → ZeRO-1/2 with TP>1) it raises a ValueError saying so, instead of an IndexError.
    • New load_universal_optimizer_step reads the step from either layout. It is used by the ZeRO-1/2 path (ZeROOptimizer.load_hp_checkpoint_state_from_checkpoint_dir, when no step.pt exists) and by the ZeRO-3 path.
  • stage3.load_hp_checkpoint_state_from_checkpoint_dir_stage3: take param_groups from the top level when present (the ZeRO-3 converter has always written it there too).

Same-stage loads take the same paths as before: every new branch is reached only when the old lookup finds nothing.

Sibling paths checked: BF16_Optimizer shares load_hp_checkpoint_state_from_checkpoint_dir and its _load_global_state is a no-op, so it gets fixes 3 and 4 and is unaffected by 2. stage_1_and_2 and stage3 are both patched.

Not a duplicate

No existing issue or PR covers cross-stage universal loading. #9069 also touches bare flat tensors but fixes the ZeRO-3 + AutoTP load path, not ZeRO-1/2 targets. #7546, #8848 and #8898 are different bugs.

Tests

New TestUniversalCheckpointAcrossZeROStages in tests/unit/checkpoint/test_universal_checkpoint.py (world size 2, fp32, torch.optim.Adam), parametrized over (1→3), (2→3), (3→1), (3→2). It trains 2 steps, saves, converts, loads into the other stage, then checks:

  • fp32 master, exp_avg and exp_avg_sq of every parameter are torch.equal to the source (the conversion only moves fp32 data, so the comparison is exact);
  • the step is 2;
  • one more training step runs.

New TestUniversalCheckpointPipelineToZeRO3 (world size 4): the pipeline engine runs at most ZeRO-1, so a pipeline-parallel run reaches ZeRO-3 only this way. It trains Megatron-style column- and row-parallel layers with TP=2 × PP=2 under ZeRO-1, converts, and loads as ZeRO-3 over 4 data-parallel ranks. It checks that every weight and Adam moment is torch.equal to the source's TP blocks put back together and that the learning rate and step are restored. It then trains the target 3 more steps on the micro-batches the source used after its save, and after every step compares the loss, weights and both moments with the source run that never stopped (assert_close, fp32 tolerance: the TP all-reduces sum in a different order).

Hardware: Intel Arc Pro B70 (XPU), Linux 6.19.10 in-tree xe driver, compute runtime 26.27.39122.11, torch 2.14.0.dev20260804+xpu. Run from the repo root.

pytest -q tests/unit/checkpoint/test_universal_checkpoint.py::TestUniversalCheckpointAcrossZeROStages
Tree B70 NVIDIA L20 (driver 580.173.02, torch 2.14.1+cu130)
master a75ab89 4 failed (gap 1: assert len(self.ckpt_list) > 0) 4 failed, same assert
this PR 4 passed 4 passed

TestUniversalCheckpointPipelineToZeRO3 on 4× L20: fails on master a75ab89 at the same assert, passes with this PR.

Regression on B70, master and this PR in the same environment:

pytest -q tests/unit/checkpoint/test_universal_checkpoint.py tests/unit/checkpoint/test_autotp_uc_checkpoint.py tests/unit/checkpoint/test_autoep_affine_universal.py
Tree Result (219 tests)
master a75ab89 30 failed, 189 passed
this PR 25 failed, 194 passed
  • The 4 extra failures on master are the new TestUniversalCheckpointAcrossZeROStages cases.
  • All other failures occur on both trees and are unrelated to this change:
    • TestZeROUniversalCheckpointDP with ZeRO-1 and load_optim=True: the bit-exact optimizer-state check fails intermittently on this XPU setup. Which cases fail changes from run to run; re-running 7 of them gave 5/7 failures with this PR and 4/7 and 5/7 on master.
    • TestRealCheckpointUniversalConversionTPxPP: No backend type associated with device type xpu.

End to end on B70 (2 ranks): Qwen3-0.6B, bf16, torch.optim.AdamW. Train 2 steps under one stage, save, ds_to_universal, load under the other stage, and compare safe_get_full_fp32_param / safe_get_full_optimizer_state before and after:

Direction Parameters Mismatched (fp32, exp_avg, exp_avg_sq; exact) Step after one more step
ZeRO-3 → ZeRO-1 310 0 3
ZeRO-1 → ZeRO-3 310 0 3

Validated matrix cells

  • Validated cross-stage: ZeRO 1/2 ↔ 3 with fp32 (unit test, XPU and CUDA) and ZeRO 1 ↔ 3 with bf16 (Qwen3-0.6B on XPU); DP=2; no offload. TP2 × PP2 ZeRO-1 → ZeRO-3 DP4 (unit test, CUDA).
  • Same-stage universal loads: covered by the regression suite above (DP resizing, AutoTP, AutoEP).
  • Not validated cross-stage: fp16, CPU/NVMe offload, MoE, ZeRO-3 → a TP>1 or pipeline-parallel target. The change has no device-specific code.

Limitations

  • A checkpoint converted from ZeRO-3 before this PR lacks ds_version, so loading it into ZeRO-1/2 still fails the version assert; re-run ds_to_universal on the original checkpoint.
  • ZeRO-3 source → ZeRO-1/2 target with TP>1 is not supported (the ZeRO-3 converter records no TP layout); it now fails with a clear error.

AI assistance

This PR was prepared with AI assistance (GitHub Copilot CLI), covering the root-cause analysis, the code, the tests and this description. I reviewed the changes and ran the tests reported above on the hardware listed.

A universal checkpoint converted from ZeRO-3 could not be loaded into
ZeRO-1/2, and one converted from ZeRO-1/2 could not be loaded into
ZeRO-3:

- the engine looked only for the model-states file named by its own
  stage;
- the ZeRO-3 converter dropped ds_version, which the ZeRO-1/2 loader
  asserts on;
- the ZeRO-1/2 loader expected dict-wrapped states, but the ZeRO-3
  converter writes bare flat tensors;
- the step was read from where the target's own stage keeps it.

Fall back to the other stage's file name on universal loads, copy
ds_version in the converter, accept flat tensors when they fill the
whole parameter, and read the step and param_groups from either layout.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 14, 2026
@jinyouzhi
jinyouzhi marked this pull request as ready for review November 15, 2026 04:42
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T04:47:38.771570Z 2bec476 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2bec47662e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

"""
state = optim_sd.get(OPTIMIZER_STATE_DICT, {}).get('state')
if state:
return next(iter(state.values())).get('step')

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Give each restored state an independent step tensor

When a cross-stage checkpoint was written by torch Adam (so step is a tensor) and the target optimizer has more than one group/subgroup, this returns one object that base_optimizer.py:405-408 caches across all groups and stage3.py:3752,3770 assigns across all subgroups. Adam increments each state's step in place, so a single resumed optimizer step advances this aliased scalar once per group/subgroup (for example, 2 to 4 for two groups) and uses incorrect bias correction; clone the value for each state before storing it.

AGENTS.md reference: AGENTS.md:L19-L19

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ce758ab. Both universal loaders now give every group and sub-group its own copy of the step, so one optimizer step advances each state once. The cross-stage test splits ZeRO-3 into sub-groups and checks that every step goes up by exactly one after a step following the load.

Comment thread deepspeed/runtime/zero/stage3.py Outdated
for key, value in optim_sd[OPTIMIZER_STATE_DICT]["param_groups"][0].items():
# A ZeRO-1/2 source has no OPTIMIZER_STATE_DICT, only the top-level PARAM_GROUPS.
param_groups = optim_sd.get(PARAM_GROUPS) or optim_sd[OPTIMIZER_STATE_DICT][PARAM_GROUPS]
for key, value in param_groups[0].items():

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restore the corresponding ZeRO-3 optimizer group

When a ZeRO-1/2 source uses multiple optimizer parameter groups (for example, separate decay/no-decay groups or learning rates), PARAM_GROUPS contains one saved mapping per group, but this loop applies only param_groups[0] to every ZeRO-3 target group. The moments are loaded by parameter name while later groups silently inherit group 0's lr, weight_decay, and other settings, changing the trajectory immediately after resume; iterate target and saved groups together as the ZeRO-1/2 loader does.

AGENTS.md reference: AGENTS.md:L19-L19

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ce758ab. The ZeRO-3 loader now restores each optimizer group from the saved group with the same index, and raises if the group counts differ. The test uses two parameter groups with different settings and checks that both are restored.

@jinyouzhi
jinyouzhi marked this pull request as draft November 15, 2026 06:57
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 18, 2026
The ZeRO-3 universal loader applied the first saved parameter group to every
optimizer group; it now restores each group from the saved group with the same
index and raises if the counts differ. Both universal loaders gave every
group or sub-group the same step tensor, which the optimizer advances in
place once per group; each state now gets its own copy.

The cross-stage test uses two parameter groups with different settings and a
sub_group_size that splits ZeRO-3's groups, checks the restored settings, and
checks that one step after the load advances every state's step once.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 19, 2026
@Achyuthan-S

Copy link
Copy Markdown
Contributor

@jinyouzhi, as offered: a data point for the TP/PP cell of "Not validated cross-stage". A tensor- and pipeline-parallel ZeRO-1 source, loaded into ZeRO-3 through a universal checkpoint.

Setup

  • Source: TP2 × PP2, trained the way DeepSpeed runs both. That's a PipelineModule on PipeModelDataParallelTopology(num_pp=2, num_mp=2, num_dp=1) with Megatron-style tensor-parallel blocks: column-parallel up, row-parallel down, and down's bias replicated. It runs ZeRO-1, the most the pipeline engine allows: 4 blocks, 2 per stage.
    • The blocks publish their shards as affine maps in UNIVERSAL_CHECKPOINT_INFO, as AutoTP's layers do.
    • AutoTP itself builds only a data × tensor mesh, so it doesn't run inside the pipeline engine. That's why the TP layers are written by hand.
  • The move: after the first steps the source saves, ds_to_universal converts, and the target loads.
  • Target: the same blocks in one plain module, under the names the pipeline gives them (0 … 3), loaded with load_universal under ZeRO-3 over 4 data-parallel ranks.
  • Reference: the source keeps training, as the run that never stopped.
  • Gradient clipping is off in both, because the pipeline engine and ZeRO-3 count a TP-replicated gradient's norm differently.

Results

Setting master #9080 ce758ab
CPU (macOS arm64, gloo, torch 2.8.0), 5 + 5 steps 97aa3c1: fails at the load, assert len(self.ckpt_list) > 0 48 of 48 tensors bit for bit (fp32, exp_avg and exp_avg_sq of 16 parameters), both as converted and after the load; losses within 2.4e-7 of the source
4× A100-SXM4-40GB (driver 595.71.05, torch 2.12.1+cu130, NCCL 2.29.7), deterministic_mode, 20 + 20 steps b9896d0: the same assert 48 of 48 bit for bit; losses within 2.4e-7

The pipeline engine writes one model-states file per stage and TP rank (mp_rank_00 … 03), and the converter merged them correctly. On master the run stops at gap 1; with #9080, nothing else breaks.

To check that the comparison can fail:

  • publishing a wrong map for down.weight (cut by rows instead of columns) flags its 12 tensors;
  • nudging one value by 1e-6 after the load flags exactly that tensor.
Script (run from the repository root: DS_ACCELERATOR=cpu python tp2pp2_to_zero3_dp4.py; TP2PP2_DEVICE=cuda TP2PP2_DETERMINISTIC=1 on 4 GPUs)
# SPDX-License-Identifier: Apache-2.0
# DeepSpeed Team
"""TP2 x PP2 -> ZeRO-3 DP4 through a universal checkpoint, checked bit for bit and by the loss.

The source is trained the way DeepSpeed trains tensor and pipeline parallelism together: a
PipelineModule on a pipe x data x model topology, with tensor-parallel layers written for that
topology (AutoTP builds a data x tensor mesh only, so it does not run inside the pipeline engine).
The pipeline engine allows ZeRO stage 1 at most, so the source runs ZeRO-1. Its layers describe
their shards with affine maps in the checkpoint's universal-checkpoint info, as AutoTP's do.

The target is the same layers in one module, registered under the names the pipeline gives them
("0", "1", ...), under ZeRO-3 over 4 data-parallel ranks.

Evidence:
  bitwise -- conversion wrote exactly the source's full tensors (parameters and Adam moments), and
             the target restored exactly what conversion wrote;
  curves  -- the target's losses track the source run that never stopped.
Usage: python tp2pp2_to_zero3_dp4.py [--steps-before N] [--steps-after N] [--workdir DIR]
       TP2PP2_DEVICE=cuda for 4 GPUs (gloo on CPU otherwise); TP2PP2_DETERMINISTIC=1 for deterministic_mode.
Run it from the repository root: DeepSpeed reads version.txt from the current directory.
"""
import argparse
import json
import os
import re
import socket
import sys
import tempfile
import time
from types import SimpleNamespace

import torch
import torch.multiprocessing as mp

BATCH, HIDDEN, LR = 4, 16, 1e-2
FFN = 42  # 21 rows of up's output per TP rank
DEVICE = os.environ.get("TP2PP2_DEVICE", "cpu")
DETERMINISTIC = os.environ.get("TP2PP2_DETERMINISTIC") == "1"


def _free_port():
    with socket.socket() as s:
        s.bind(("127.0.0.1", 0))
        return s.getsockname()[1]


def batch(step):
    g = torch.Generator().manual_seed(1000 + step)
    return torch.randn(BATCH, HIDDEN, generator=g), torch.randn(BATCH, HIDDEN, generator=g)


def convert(workdir, tag):
    from deepspeed.checkpoint.ds_to_universal import main as convert_to_universal
    cp = os.path.join(workdir, "ckpt", tag)
    convert_to_universal(
        SimpleNamespace(input_folder=cp,
                        output_folder=f"{cp}_universal",
                        num_extract_workers=1,
                        num_merge_workers=1,
                        keep_temp_folder=False,
                        strict=True,
                        inject_missing_state=False))
    with open(os.path.join(workdir, "ckpt", "latest_universal"), "w") as f:
        f.write(f"{tag}_universal")


TP, PP, DP_TARGET = 2, 2, 4
LAYERS = 4  # two per pipeline stage


def full_weights(layer):
    """The full parameters of one block, from a seed of its own, so every layout starts from the same values."""
    generator = torch.Generator().manual_seed(100 + layer)
    up = torch.nn.Linear(HIDDEN, FFN)
    down = torch.nn.Linear(FFN, HIDDEN)
    with torch.no_grad():
        for p in list(up.parameters()) + list(down.parameters()):
            p.copy_(torch.randn(p.shape, generator=generator) * 0.2)
    return {"up.weight": up.weight, "up.bias": up.bias, "down.weight": down.weight, "down.bias": down.bias}


def shard_maps():
    """How a TP2 block holds each parameter: up by rows (column parallel), down by columns (row parallel)."""
    from deepspeed.checkpoint.affine import contiguous_split_map, replicated_map
    halves = [FFN // TP + (1 if i < FFN % TP else 0) for i in range(TP)]
    return {
        "up.weight": contiguous_split_map((FFN, HIDDEN), halves, 0),
        "up.bias": contiguous_split_map((FFN, ), halves, 0),
        "down.weight": contiguous_split_map((HIDDEN, FFN), halves, 1),
        "down.bias": replicated_map((HIDDEN, ), TP),
    }


class Block(torch.nn.Module):
    """x + down(gelu(up(x))), whole: the target's layer."""

    def __init__(self, layer):
        super().__init__()
        weights = full_weights(layer)
        self.up = torch.nn.Linear(HIDDEN, FFN)
        self.down = torch.nn.Linear(FFN, HIDDEN)
        with torch.no_grad():
            for name, p in self.named_parameters():
                p.copy_(weights[name])

    def forward(self, x):
        return x + self.down(torch.nn.functional.gelu(self.up(x)))


class _CopyToTensorParallel(torch.autograd.Function):
    """Identity forward; the backward sums the input gradient over the TP group (Megatron's f)."""

    @staticmethod
    def forward(ctx, x):
        return x

    @staticmethod
    def backward(ctx, grad):
        import deepspeed.comm as dist
        grad = grad.clone()
        dist.all_reduce(grad, group=TP_GROUP[0])
        return grad


class _ReduceFromTensorParallel(torch.autograd.Function):
    """Sum over the TP group forward; identity backward (Megatron's g)."""

    @staticmethod
    def forward(ctx, x):
        import deepspeed.comm as dist
        x = x.clone()
        dist.all_reduce(x, group=TP_GROUP[0])
        return x

    @staticmethod
    def backward(ctx, grad):
        return grad


TP_GROUP = [None]  # set once the pipeline grid exists; the layers read it at run time


class TPBlock(torch.nn.Module):
    """The same block held by one TP rank: up's rows and down's columns for that rank, down's bias whole."""

    def __init__(self, layer, tp_rank):
        super().__init__()
        weights = full_weights(layer)
        maps = shard_maps()
        shards = {name: maps[name].extract(weights[name].detach(), tp_rank) for name in weights}
        self.up = torch.nn.Linear(HIDDEN, shards["up.weight"].shape[0])
        self.down = torch.nn.Linear(shards["down.weight"].shape[1], HIDDEN)
        with torch.no_grad():
            for name, p in self.named_parameters():
                p.copy_(shards[name].view_as(p))

    def forward(self, x):
        hidden = torch.nn.functional.gelu(self.up(_CopyToTensorParallel.apply(x)))
        partial = torch.nn.functional.linear(hidden, self.down.weight)
        return x + _ReduceFromTensorParallel.apply(partial) + self.down.bias


class Plain(torch.nn.Module):
    """The target: every block in one module, under the pipeline's layer names."""

    def __init__(self):
        super().__init__()
        for layer in range(LAYERS):
            self.add_module(str(layer), Block(layer))

    def forward(self, x, y):
        for layer in range(LAYERS):
            x = getattr(self, str(layer))(x)
        return torch.nn.functional.mse_loss(x, y)


def universal_info():
    """The universal-checkpoint info the source publishes: one affine map per parameter of every stage."""
    from deepspeed.checkpoint.affine import AFFINE_MAP_FORMAT_VERSION
    from deepspeed.checkpoint.constants import (AFFINE_MAP, AFFINE_MAP_PARAMS, AFFINE_MAP_VERSION,
                                                UNIVERSAL_CHECKPOINT_VERSION_KEY, UNIVERSAL_CHECKPOINT_VERSION_VALUE)
    maps = shard_maps()
    params = {rf"^{re.escape(f'{layer}.{name}')}$": m.to_dict() for layer in range(LAYERS) for name, m in maps.items()}
    return {
        UNIVERSAL_CHECKPOINT_VERSION_KEY: UNIVERSAL_CHECKPOINT_VERSION_VALUE,
        AFFINE_MAP: {
            AFFINE_MAP_VERSION: AFFINE_MAP_FORMAT_VERSION,
            AFFINE_MAP_PARAMS: params
        },
    }


def config(zero_stage, load_universal=False):
    cfg = {
        "train_batch_size": BATCH,
        "train_micro_batch_size_per_gpu": BATCH if zero_stage == 1 else BATCH // DP_TARGET,
        "gradient_accumulation_steps": 1,
        "optimizer": {
            "type": "Adam",
            "params": {
                "lr": LR,
                "torch_adam": True
            }
        },
        "zero_optimization": {
            "stage": zero_stage
        },
        "steps_per_print": 10**9,
        # Off in both layouts: a pipeline engine and a ZeRO-3 engine count a TP-replicated gradient's norm by
        # different rules, and clipping is not what this check is about.
        "gradient_clipping": 0.0,
    }
    if zero_stage == 3:
        cfg["zero_optimization"]["stage3_param_persistence_threshold"] = 0
    if DETERMINISTIC:
        cfg["deterministic_mode"] = True
    if load_universal:
        cfg["checkpoint"] = {"load_universal": True}
    return cfg


def _setup(rank, world, port):
    os.environ.update(MASTER_ADDR="127.0.0.1",
                      MASTER_PORT=str(port),
                      RANK=str(rank),
                      LOCAL_RANK=str(rank),
                      WORLD_SIZE=str(world),
                      LOCAL_SIZE=str(world))
    if DETERMINISTIC:
        os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
    import deepspeed
    if DEVICE == "cuda":
        torch.cuda.set_device(rank)
    deepspeed.init_distributed(dist_backend="nccl" if DEVICE == "cuda" else "gloo")
    return deepspeed


def _state(getter_names, engine):
    from deepspeed.utils.tensor_fragment import safe_get_full_fp32_param, safe_get_full_optimizer_state
    getters = {
        "fp32": safe_get_full_fp32_param,
        "exp_avg": lambda p: safe_get_full_optimizer_state(p, "exp_avg"),
        "exp_avg_sq": lambda p: safe_get_full_optimizer_state(p, "exp_avg_sq"),
    }
    return {
        key: {
            name: getters[key](p).detach().cpu().clone()
            for name, p in engine.module.named_parameters()
        }
        for key in getter_names
    }


def source_worker(rank, port, workdir, steps_before, steps_after):
    """TP2 x PP2 under ZeRO-1: train, save, gather the full state, then keep training as the reference."""
    deepspeed = _setup(rank, TP * PP, port)
    import deepspeed.comm as dist
    from deepspeed.checkpoint.constants import UNIVERSAL_CHECKPOINT_INFO
    from deepspeed.pipe import LayerSpec, PipelineModule
    from deepspeed.runtime.pipe.topology import PipeModelDataParallelTopology
    topology = PipeModelDataParallelTopology(num_pp=PP, num_mp=TP, num_dp=1)
    tp_rank = topology.get_coord(rank).model
    model = PipelineModule(layers=[LayerSpec(TPBlock, layer, tp_rank) for layer in range(LAYERS)],
                           topology=topology,
                           loss_fn=torch.nn.functional.mse_loss,
                           partition_method="uniform")
    TP_GROUP[0] = model._grid.get_slice_parallel_group()
    engine, _, _, _ = deepspeed.initialize(model=model, model_parameters=model.parameters(), config=config(1))

    def step(index):
        x, y = batch(index)
        return engine.train_batch(data_iter=iter([(x.to(engine.device), y.to(engine.device))])).item()

    losses = [step(index) for index in range(steps_before)]
    ckpt = os.path.join(workdir, "ckpt")
    engine.save_checkpoint(ckpt, tag="p0", client_state={UNIVERSAL_CHECKPOINT_INFO: universal_info()})

    # Each TP rank holds its shard; rebuild every parameter's full tensors through the published maps,
    # then collect the stages' tensors on every rank.
    maps = shard_maps()
    local = _state(("fp32", "exp_avg", "exp_avg_sq"), engine)
    full = {}
    for key, tensors in local.items():
        full[key] = {}
        for name, shard in tensors.items():
            shards = [None] * TP
            dist.all_gather_object(shards, shard, group=TP_GROUP[0])
            full[key][name] = maps[name.split(".", 1)[1]].rebuild(dict(enumerate(shards)))
    stages = [None] * (TP * PP)
    dist.all_gather_object(stages, full)
    losses += [step(index) for index in range(steps_before, steps_before + steps_after)]
    if rank == 0:
        saved = {key: {} for key in full}
        for stage_state in stages:
            for key, tensors in stage_state.items():
                saved[key].update(tensors)
        torch.save(saved, os.path.join(workdir, "p0_saved.pt"))
        json.dump({"losses": losses}, open(os.path.join(workdir, "p0_record.json"), "w"))


def target_worker(rank, port, workdir, steps_before, steps_after):
    """ZeRO-3 over 4 data-parallel ranks: load the universal checkpoint, record the state, and train on."""
    deepspeed = _setup(rank, DP_TARGET, port)
    torch.manual_seed(0)
    model = Plain()
    engine, _, _, _ = deepspeed.initialize(model=model,
                                           model_parameters=model.parameters(),
                                           config=config(3, load_universal=True))
    engine.load_checkpoint(os.path.join(workdir, "ckpt"), tag="p0_universal", load_optimizer_states=True)
    restored = _state(("fp32", "exp_avg", "exp_avg_sq"), engine)
    import deepspeed.comm as dist
    micro = BATCH // DP_TARGET
    rows = slice(rank * micro, rank * micro + micro)
    losses = []
    for index in range(steps_before, steps_before + steps_after):
        x, y = batch(index)
        loss = engine(x[rows].to(engine.device), y[rows].to(engine.device))
        engine.backward(loss)
        engine.step()
        total = loss.detach().to(torch.float32, copy=True)
        dist.all_reduce(total)
        losses.append(total.item() / DP_TARGET)
    if rank == 0:
        torch.save(restored, os.path.join(workdir, "p1_restored.pt"))
        json.dump({"losses": losses}, open(os.path.join(workdir, "p1_record.json"), "w"))


def main():
    p = argparse.ArgumentParser()
    p.add_argument("--steps-before", type=int, default=5)
    p.add_argument("--steps-after", type=int, default=5)
    p.add_argument("--workdir", default=None)
    args = p.parse_args()
    os.environ["DS_ACCELERATOR"] = "cuda" if DEVICE == "cuda" else "cpu"
    workdir = args.workdir or tempfile.mkdtemp(prefix="uc_tp2pp2_to_z3dp4_")
    seconds = {}
    begin = time.perf_counter()
    mp.spawn(source_worker,
             args=(_free_port(), workdir, args.steps_before, args.steps_after),
             nprocs=TP * PP,
             join=True)
    seconds["source"] = time.perf_counter() - begin
    begin = time.perf_counter()
    convert(workdir, "p0")
    seconds["convert"] = time.perf_counter() - begin
    begin = time.perf_counter()
    mp.spawn(target_worker,
             args=(_free_port(), workdir, args.steps_before, args.steps_after),
             nprocs=DP_TARGET,
             join=True)
    seconds["target"] = time.perf_counter() - begin

    saved = torch.load(os.path.join(workdir, "p0_saved.pt"))
    restored = torch.load(os.path.join(workdir, "p1_restored.pt"))
    univ = os.path.join(workdir, "ckpt", "p0_universal", "zero")
    bitwise = {}
    for key, tensors in saved.items():
        for name, want in tensors.items():
            raw = torch.load(os.path.join(univ, name, f"{key}.pt"), weights_only=False)
            written = raw["param"] if isinstance(raw, dict) else raw
            bitwise[f"{name}/{key}"] = {
                "convert": torch.equal(written.view_as(want), want),
                "restore": torch.equal(restored[key][name].view_as(want), want)
            }
    reference = json.load(open(os.path.join(workdir, "p0_record.json")))["losses"][args.steps_before:]
    got = json.load(open(os.path.join(workdir, "p1_record.json")))["losses"]
    report = {
        "from": f"TP{TP}xPP{PP} ZeRO-1",
        "to": f"DP{DP_TARGET} ZeRO-3",
        "device": DEVICE,
        "all_bitwise_exact": bool(bitwise) and all(v["convert"] and v["restore"] for v in bitwise.values()),
        "tensors_checked": len(bitwise),
        "not_exact": sorted(k for k, v in bitwise.items() if not (v["convert"] and v["restore"])),
        "max_abs_loss_diff": max(abs(a - b) for a, b in zip(reference, got)),
        "resumed_losses": got,
        "reference_losses": reference,
        "seconds": seconds,
        "workdir": workdir,
    }
    print(json.dumps(report, indent=1))
    return 0 if report["all_bitwise_exact"] else 1


if __name__ == "__main__":
    sys.exit(main())

I'm happy to turn it into a 4-rank test in tests/unit/checkpoint/test_universal_checkpoint.py next to TestUniversalCheckpointAcrossZeROStages, if that's useful.

The pipeline engine runs at most ZeRO-1, so a pipeline-parallel run reaches
ZeRO-3 only through a universal checkpoint loaded across stages. The new test
trains Megatron-style column- and row-parallel layers with TP=2 and PP=2 under
ZeRO-1, converts the checkpoint, and loads it as ZeRO-3 over 4 data-parallel
ranks. It checks that every weight and Adam moment matches the source's
reassembled tensors bit for bit, that the settings and step are restored, and
that the target keeps training. On master the load fails to find the model
states file.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
After the save, the TP2 x PP2 source keeps training for 3 steps. The ZeRO-3
target follows on the same micro-batches, each data-parallel rank taking one
of the source's pair, and must match the source's loss, weights and both Adam
moments after every step within fp32 tolerance. Rank 0 now shares the outcome
of the conversion, so a converter failure fails every rank instead of leaving
the others waiting at a barrier.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants