Skip to content

[RFC] Cleanup: move process-wide parallel-group globals to per-engine groups — follow-up to #8241 / #8851 #9081

Open

Description

@jinyouzhi

Summary

DeepSpeed keeps an engine's tensor-parallel and data-parallel process groups as module globals in deepspeed/utils/groups.py. The first engine that sets them up decides them for every later engine in the process. This RFC proposes that each engine own its groups again, as it did before the global registry existed.

This continues two earlier fixes:

The process groups are the remaining process-wide state of this kind. They are also what flexible TP/DP redistribution needs: a trainer and a rollout, or the same trainer before and after a resize, living in one process with different layouts.

I found no existing issue or PR for this. I'd like the maintainers' view on whether and how to change it before writing code.

Why it matters now

Several pieces of open work need two engines with different layouts in one process:

How it got here

  • Before August 2021, the engine took a user mpu (for example Megatron's), kept it in self.mpu, and kept its own self.data_parallel_group. Nothing about the layout was process-wide.
  • DeepSpeed MoE  #1310 (MoE, August 2021) added deepspeed/utils/groups.py as a registry of expert groups.
  • Refactor MoE and Groups API to simplify model creation and mangement #1798 (February 2022) began copying the engine's mpu into the module: groups.mpu = self.mpu, "For subsequent use, just query groups".
  • autotp training(fix dco) #7004 (AutoTP training, February 2025) made the module itself the engine's mpu: self.mpu = groups, followed by self.mpu._init_tp_mesh_device(...), which sets the module's TP/DP groups and mesh.

So the proposal is mostly a return to the earlier design, keeping what the registry added.

What happens today

_init_tp_mesh_device returns early if _TENSOR_MODEL_PARALLEL_GROUP is already set. A second engine with another autotp_size therefore silently uses the first engine's groups.

Reproducer (4 ranks)
import sys, torch, transformers, deepspeed
import deepspeed.comm as dist
from deepspeed.utils import groups


def engine(tp):
    torch.manual_seed(0)
    cfg = transformers.LlamaConfig(vocab_size=64, hidden_size=64, intermediate_size=128, num_hidden_layers=2,
                                   num_attention_heads=4, num_key_value_heads=4, use_cache=False,
                                   tie_word_embeddings=False)
    m = transformers.LlamaForCausalLM(cfg)
    ds = {"train_micro_batch_size_per_gpu": 1, "bf16": {"enabled": True}, "zero_optimization": {"stage": 1},
          "optimizer": {"type": "Adam", "params": {"lr": 1e-3, "torch_adam": True}},
          "tensor_parallel": {"autotp_size": tp}}
    return deepspeed.initialize(model=m, model_parameters=m.parameters(), config=ds)[0]


def show(tag, e):
    w = e.module.model.layers[0].self_attn.q_proj.weight
    if dist.get_rank() == 0:
        print(tag, "autotp_size", e.autotp_size(), "mpu tp", e.mpu.get_tensor_model_parallel_world_size(),
              "q_proj", tuple(w.shape))


deepspeed.init_distributed()
a = engine(2)
if sys.argv[1] == "release":  # what DeepSpeedExamples#1021 does before building the second engine
    groups._TENSOR_MODEL_PARALLEL_GROUP = groups._DATA_PARALLEL_GROUP = None
b = engine(4)
show("B", b)
show("A", a)

Run with torchrun --nproc_per_node 4:

Case Engine autotp_size engine.mpu TP size q_proj shard on a rank
B built alone B 4 4 (16, 64)
A (TP2), then B (TP4) B 4 2 (32, 64): B is a TP2 model; its 3-step losses equal A's
A, clear the globals, then B B 4 4 (16, 64)
A 2 4 (32, 64)
  • In the second case, B is not what its config asks for, and nothing warns.
  • In the third case, B is right, but A now reports TP4 for its TP2 shards.
    • A's training steps and checkpoint file names were still correct in this small test: AutoTP layers keep their own group, and ZeRO took its DP group at initialize.
    • Every path that queries groups or engine.mpu later reads B's groups. That is why ZeRO3: Gradient norm allreduce for DP #1021 saves and restores the globals per engine.

What reads and writes the globals

Writers (all in deepspeed/utils/groups.py unless noted):

  • _init_tp_mesh_device and _init_tp_groups_with_new_group: _TENSOR_MODEL_PARALLEL_GROUP, _MODEL_PARALLEL_GROUP, _DATA_PARALLEL_GROUP.
  • mesh_device; also assigned in engine.py (groups.mesh_device = self.mesh_device).
  • mpu, assigned in engine.py (groups.mpu = self.mpu).
  • _MPU_TENSOR_MODEL_PARALLEL_WORLD_SIZE / _RANK.
  • The expert and expert-data-parallel tables (_EXPERT_PARALLEL_GROUP, ...). These are keyed by name (ep_size_N), so two EP sizes can coexist. They are listed for completeness.

Readers outside groups.py (on master 84481c3; about 48 references in 13 files):

Area Where Reads
Engine engine.py: 13 places, including the dataloader check, tp_config.tp_group, rank getters, checkpoint tp_rank, AutoEP folding TP group, rank and size; DP rank; mpu
ZeRO / BF16 stage_1_and_2.py, stage3.py (2), bf16_optimizer.py TP group for AutoEP folding; in stage3.py also for the universal-checkpoint load of AutoTP shards
AutoTP auto_tp.py (set_tensor_parallel_config in training mode), tp_manager.py (2) TP group and its source rank
Checkpoint model_checkpointing/utils.py (3), checkpoint/zero_affine.py (default tp_group) mpu, TP group
MoE / AutoEP moe/sharded_moe.py, moe/mappings.py, moe/layer.py, auto_ep_layer.py mpu, TP group
Compile compile/custom_ops/tp_collectives.py TP group

Open PRs that add readers. #9037 (multi-turn tool-calling rollout, draft) reads the global TP group in runtime/rollout/hybrid_engine_rollout.py:

  • _host_tensor_parallel_group() builds Gloo groups over groups.get_tensor_model_parallel_group() and uses groups.get_tensor_model_parallel_src_rank() as the rank that forwards tool output;
  • _sync_rng_across_tp() broadcasts the RNG state over the same group.

The rollout takes its TP size from engine.autotp_size() but the group from the module. Next to a trainer with another TP size, both would run over the first engine's groups. Reading engine.mpu instead would not help on master, since under AutoTP engine.mpu is the groups module itself. With stage 2 (#9120), engine.mpu is the engine's own groups object, so these readers can move to it.

Options

  1. Keep the globals; document a supported way to switch.
  2. Each engine owns its groups (return to the pre-2022 design).
  3. A guard only: raise if a second initialize asks for a different autotp_size than the existing groups. This fixes the silent misconfiguration but does not enable two layouts.

My preference is 2, with 3 as an immediate first step, done in stages:

  1. The guard (option 3), plus a test that builds TP2 and TP4 engines in one process.
  2. Per-engine group object; engine.py, ZeRO and AutoTP readers move to it.
  3. Checkpoint and zero_affine readers.
  4. MoE / AutoEP and compile readers.
  5. Group lifetime: destroy an engine's groups with it, share groups between engines, replace them in place.

Each stage keeps the module getters working, so it can merge on its own.

Roadmap and status

Updated after the discussion below: @LiRunGuo and @Achyuthan-S both support option 2, with the guard first.

The ZeRO-3 rank-agreement checks (process-wide ids, trace state) are tracked in #9098 and fixed in a separate PR by Achyuthan-S; stage 2 (#9120) does not touch them.

Stage Scope Owner Status
1. Guard Raise when a later engine asks for another TP layout, including an engine without AutoTP and without its own mpu (@Achyuthan-S). Test with TP2/TP4 engines in one process. @jinyouzhi #9088 (in review)
2. Per-engine groups Per-engine group object; engine.py, ZeRO, BF16 and AutoTP readers move to it. This includes _reduce_non_expert_gradients and the DP case @LiRunGuo reported: an engine without an mpu takes the previous engine's groups.mpu. The rollout readers of #9037 belong here too. @jinyouzhi #9120 (draft, on top of #9088). Every AutoTP engine's engine.mpu is its own groups object; with the process-wide layout it holds the process-wide groups, so no communicator is added. An engine without an mpu no longer inherits an earlier engine's, and an engine can share another's groups with mpu=other.mpu. Tested with #9074's DP4 → DP2 → DP4 order. An engine on groups other than the process-wide ones still raises for mesh_param, AutoEP, the DeepCompile autotp pass and MoE (stage 4). The #9037 readers move once #9037 merges. @LiRunGuo offered to run it against #9074's tests
3. Checkpoint readers model_checkpointing/utils.py, the default tp_group of zero_affine.py, and the universal-checkpoint load of AutoTP shards in stage3.py stage3.py: @Achyuthan-S; rest open Partly done in #9120, because these readers break once an engine has groups of its own: the stage3.py universal-checkpoint load, the checkpoint.writer DP layout, and the default tp_group of zero_affine.py (taken from the model's AutoTP layers) use the engine's groups. Remaining: model_checkpointing/utils.py, and the zero_affine.py default replica group of parameters outside a ZeRO optimizer
4. MoE / AutoEP, compile sharded_moe.py, mappings.py, moe/layer.py, auto_ep_layer.py, tp_collectives.py open not started
5. Group lifetime An owner and a reference count per group: an engine destroys the groups it created when it goes away, unless another engine shares them; groups passed in as mpu stay with their creator; the process-wide groups are never destroyed. Then, optionally, reuse groups automatically by layout (DeepSpeed creates groups without options, so equal ranks give equivalent groups). An API to replace an engine's groups in place, for what #9074's Zero3Repartition does by hand. open not started. Until then, sharing is explicit (mpu=other.mpu) and no group is destroyed

Related, outside these stages:

  • Membership changes: with stage 2, groups belong to their engine; stage 5 lets them be destroyed when it goes away. Rebuilding the world after a rank leaves, and use_local_synchronization, belong to the membership side of the elastic design.

Questions for the maintainers

@delock @tohtana @tjruwase (cc @LiRunGuo @Achyuthan-S from #9026)

  1. Should DeepSpeed support several engines with different TP/DP layouts in one process, or keep the process-wide registry and leave switching to external code (as ZeRO3: Gradient norm allreduce for DP #1021 and [rollout] In-memory relayout between sharded layouts (two-stage relay, pipelined execution) and ZeRO-3 training-state move #9074 do now)?
  2. If supported: is returning to engine-owned groups (option 2) the right direction, or is an official snapshot/activate API (option 1) preferred?
  3. In stages as above, or as one change?
  4. Should the module getters (groups.get_tensor_model_parallel_group() and the rest) stay as they are, or become deprecated in favor of the engine's groups?

Tested on: 4× Intel Arc Pro B70 (XPU, oneCCL), torch 2.14.0.dev20260804+xpu, transformers 5.17.0, DeepSpeed master 84481c3. The #1021 results above were on 8× NVIDIA H20.

This RFC was drafted with AI assistance. I checked the code references and ran the reproducer myself.

Activity

  1. changed the title [-][RFC] Per-engine parallel groups: follow-up to #8241 / #8851 for flexible TP/DP redistribution[/-] [+][RFC] Cleanup: move process-wide parallel-group globals to per-engine groups — follow-up to #8241 / #8851[/+] on Nov 14, 2026
  2. LiRunGuo commented on Nov 14, 2026

    @LiRunGuo
    Contributor

    Thanks for writing this up, @jinyouzhi. Three data points for the RFC, from #9074 / #9076:

    The DP side. engine.py copies an engine's mpu into groups.mpu when one is given, and groups._get_data_parallel_group() returns that mpu's group. An engine initialized without an mpu after one with an mpu silently takes the earlier engine's DP group. In #9074's first DP4 ↔ DP2 test, the engine over all ranks came out over a subgroup on some ranks and over the world on others, and its first collective hung. #9074 works around it by giving every engine its own mpu, including the one over all ranks. That is enough for ZeRO-3, which takes its DP group at initialize. It would not be enough for an engine whose gradient all-reduce goes through _reduce_non_expert_gradients, which asks groups at every step.

    Process-wide ids. ZeRO-3 numbers modules and parameters with process-wide counters (_register_deepspeed_module(self, module, count=[0]) and zero.Init.param_id), and the trace checks compare the raw ids. A new engine whose ranks have built different numbers of ZeRO-3 modules or parameters before fails its first trace check ("disagreement on list contents"). #9074 hit this and builds the widest engine first; @Achyuthan-S described it on #9076. It is the same kind of process-wide state, so it may belong in this RFC's list or next to it. One direction, untested: compare each id's position among the engine's own ids instead of the process-wide number.

    Membership changes. Both get stricter when the set of ranks changes during a job. A joining process starts with no groups.mpu and its counters at 0, while the survivors keep theirs. After a rank leaves, any group still held in the globals contains it. Option 2 lets a new engine own groups over the new membership and lets the old ones be destroyed with the old engine; the ids would need the same per-engine treatment.

    +1 for option 2 with the guard first. I can run stage 2 against #9074's DP4 → DP2 → DP4 test when it is up.

  3. Achyuthan-S commented on Nov 14, 2026

    @Achyuthan-S
    Contributor

    Thanks @jinyouzhi, +1 for option 2 with the guard first. Two points from the affine-IR side:

    • Where the guard sits. engine.py builds the TP groups only when autotp_size() > 1, so an engine with autotp_size 1 that comes after a TP2 engine never reaches _init_tp_mesh_device. Its get_tensor_parallel_rank() and get_model_parallel_rank() still read the module's groups, so they return the TP2 engine's ranks; built alone, the same calls fail the "not initialized" assertion. That is from reading the code, not a run. A guard in initialize that compares the requested TP size, 1 included, with the groups already set would cover this case too.
    • A reader I touch. When the ZeRO-3 universal-checkpoint load slices an AutoTP shard, it takes the TP group from groups (stage3.py, from [AutoTP] Fix ZeRO-3 checkpoint consolidation to gather across TP and DP #8168), and ZeRO-3: take the AutoTP shard from universal checkpoints converted without TP #9069 sends more loads down that path. I can move that one onto the engine's group in the checkpoint stage.

    On the counters @LiRunGuo mentions: I'll open them as their own issue with a reproduction, as promised on #9076, and link it here. They need a per-engine fix inside ZeRO-3, next to stage 2.

  4. sfc-gh-truwase commented on Nov 17, 2026

    @sfc-gh-truwase
    Collaborator

    @jinyouzhi and team, thanks for the great discussion here. I am aligned with option 2, independent engines per process. This is a feature that we have previously supported, but somehow seems to have suffered regressions.

    - Independent engines per process

    1. DS Chat - A key feature was the ability to create independent actor, reference, critic, etc.
    2. HF multi-model training: https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed_multiple_model

    - Eliminating process-wide Globals
    This is a very worthy cause with a long history. Any progress here would be extremely useful.

    1. Diffusers: [deepspeed] partial ZeRO-3 support huggingface/diffusers#3076
    2. ZeRO3: Nested zero.Init() and dynamically defined model class #2989, Support model declaration in zero.Init context #3592
  5. jinyouzhi commented on Nov 17, 2026

    @jinyouzhi
    ContributorAuthor

    Thanks @sfc-gh-truwase, and thanks for the history and links. I fully agree: this is more than restoring a basic capability, it is, as you put it, a very worthy cause with a long history. DS-Chat's actor, reference and critic, and Accelerate's multi-model setup will be the regression cases. Beyond them, the goal is for each engine to own its parallel groups and its process-wide state, so that engines in one process can use different TP/DP layouts, and an engine can be torn down and rebuilt when the set of ranks changes. That is what RL weight sync, #9074's DP4 → DP2 → DP4 switch and elastic jobs need. I'll read #2989 / #3592 before touching ZeRO-3.

    On the guard, #9088:

    Next steps:

  6. LiRunGuo commented on Nov 17, 2026

    @LiRunGuo
    Contributor

    Thanks @sfc-gh-truwase for the direction and the history, and @jinyouzhi for the plan. For stage 2, these are the multi-engine checks on our side, all in #9074 now:

    • tests/unit/runtime/rollout/test_zero3_repartition.py (4 ranks, added to [rollout] In-memory relayout between sharded layouts (two-stage relay, pipelined execution) and ZeRO-3 training-state move #9074 today) is the DP4 → DP2 → DP4 test. On ranks 0 and 1 it holds four ZeRO-3 engines, each with its own mpu:
      • one engine re-partitioned in place (Zero3Repartition);
      • the two-engine path, engines over four ranks and over two with the state moved between them (Zero3StateMove);
      • a plain four-rank engine as the oracle.
    • tests/unit/runtime/rollout/test_native_rollout.py::TestTrainerSwitch (2 ranks) moves a trainer between rank 0 alone and both ranks, with two engines per process.

    Two things in that code that stage 2 should make unnecessary:

    1. Every engine gets its own TrainerMPU, made on every rank, with the engine over all ranks built first. The first part is the groups.mpu inheritance; the ordering is [BUG] ZeRO-3 ids come from process-wide counters, so ranks that built different models fail the rank-agreement checks #9098.
    2. Zero3Repartition re-points groups.mpu and every reference the engine holds to its old data-parallel group by hand. With per-engine groups, that becomes a re-point of the engine's own groups.

    I'll also run the end-to-end DP4 → DP2 → DP4 on Qwen3-0.6B (4 GPUs, both paths) against stage 2 once it's up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions