Repository navigation
Description
Summary
DeepSpeed keeps an engine's tensor-parallel and data-parallel process groups as module globals in deepspeed/utils/groups.py. The first engine that sets them up decides them for every later engine in the process. This RFC proposes that each engine own its groups again, as it did before the global registry existed.
This continues two earlier fixes:
- [AutoTP] Replace tp_shard process-wide globals with per-model AutoTPMeta #8241 (fixing Cleanup: remove tp_shard process-wide scalar globals (num_kv_heads / tp_grain_size / ...), thread explicitly #8231) moved the AutoTP shape state (
num_kv_heads,n_embd,tp_grain_size, ...) fromtp_shardglobals onto a per-modelAutoTPMeta. - [AutoTP] Support frozen AutoTP models and manage AutoTP mode per model #8851 made the AutoTP training/inference mode per model instead of the process-wide
DEEPSPEED_AUTOTP_MODE.
The process groups are the remaining process-wide state of this kind. They are also what flexible TP/DP redistribution needs: a trainer and a rollout, or the same trainer before and after a resize, living in one process with different layouts.
I found no existing issue or PR for this. I'd like the maintainers' view on whether and how to change it before writing code.
Why it matters now
Several pieces of open work need two engines with different layouts in one process:
- RFC [RFC] Two-stage relay for cross-node weight sync, priced by a calibrated cost model #9026, live TP/DP redistribution. Moving weights and training state between layouts without a checkpoint assumes both layouts exist side by side.
- Add AutoTP training to rollout weight sync example DeepSpeedExamples#1021. It moves training state across DP, ZeRO stage, AutoTP size and AutoEP size mid-training (results, 8× H20). To build a second AutoTP engine, it works around the globals:
- before each
deepspeed.initialize, it clears_TENSOR_MODEL_PARALLEL_GROUPand_DATA_PARALLEL_GROUP; - it saves
_TENSOR_MODEL_PARALLEL_GROUP,_MODEL_PARALLEL_GROUP,_DATA_PARALLEL_GROUP,_MPU_TENSOR_MODEL_PARALLEL_*,mesh_deviceandmpuper engine, and writes them back before that engine trains again.
- before each
- [rollout] In-memory relayout between sharded layouts (two-stage relay, pipelined execution) and ZeRO-3 training-state move #9074 / [zero] Run ZeRO-3's rank-agreement checks and barriers over the partition group #9076 (training-state move, ZeRO-3 DP4 → DP2 → DP4).
- [zero] Run ZeRO-3's rank-agreement checks and barriers over the partition group #9076 moves ZeRO-3's rank-agreement checks and barriers from the world to the partition group, so a ZeRO-3 engine can run on a subset of the ranks. That is the same class of problem, for ZeRO-3's checks.
- [rollout] In-memory relayout between sharded layouts (two-stage relay, pipelined execution) and ZeRO-3 training-state move #9074 changes only the DP size, so it does not reach the TP globals; its known limits list TP, ZeRO-1/2 and EP as not covered.
- [rollout] Affine weight-sync building block: ZeRO storage into a tensor-parallel inference copy, shard to shard #9045 (
AffineWeightSync) already takes an explicitgroup, andautotp_storage_affine_maps(model, tp_group=...)takes an explicit TP group. The explicit-group style exists; the engine is what does not follow it.
How it got here
- Before August 2021, the engine took a user
mpu(for example Megatron's), kept it inself.mpu, and kept its ownself.data_parallel_group. Nothing about the layout was process-wide. - DeepSpeed MoE #1310 (MoE, August 2021) added
deepspeed/utils/groups.pyas a registry of expert groups. - Refactor MoE and Groups API to simplify model creation and mangement #1798 (February 2022) began copying the engine's
mpuinto the module:groups.mpu = self.mpu, "For subsequent use, just query groups". - autotp training(fix dco) #7004 (AutoTP training, February 2025) made the module itself the engine's
mpu:self.mpu = groups, followed byself.mpu._init_tp_mesh_device(...), which sets the module's TP/DP groups and mesh.
So the proposal is mostly a return to the earlier design, keeping what the registry added.
What happens today
_init_tp_mesh_device returns early if _TENSOR_MODEL_PARALLEL_GROUP is already set. A second engine with another autotp_size therefore silently uses the first engine's groups.
Reproducer (4 ranks)
import sys, torch, transformers, deepspeed
import deepspeed.comm as dist
from deepspeed.utils import groups
def engine(tp):
torch.manual_seed(0)
cfg = transformers.LlamaConfig(vocab_size=64, hidden_size=64, intermediate_size=128, num_hidden_layers=2,
num_attention_heads=4, num_key_value_heads=4, use_cache=False,
tie_word_embeddings=False)
m = transformers.LlamaForCausalLM(cfg)
ds = {"train_micro_batch_size_per_gpu": 1, "bf16": {"enabled": True}, "zero_optimization": {"stage": 1},
"optimizer": {"type": "Adam", "params": {"lr": 1e-3, "torch_adam": True}},
"tensor_parallel": {"autotp_size": tp}}
return deepspeed.initialize(model=m, model_parameters=m.parameters(), config=ds)[0]
def show(tag, e):
w = e.module.model.layers[0].self_attn.q_proj.weight
if dist.get_rank() == 0:
print(tag, "autotp_size", e.autotp_size(), "mpu tp", e.mpu.get_tensor_model_parallel_world_size(),
"q_proj", tuple(w.shape))
deepspeed.init_distributed()
a = engine(2)
if sys.argv[1] == "release": # what DeepSpeedExamples#1021 does before building the second engine
groups._TENSOR_MODEL_PARALLEL_GROUP = groups._DATA_PARALLEL_GROUP = None
b = engine(4)
show("B", b)
show("A", a)Run with torchrun --nproc_per_node 4:
| Case | Engine | autotp_size |
engine.mpu TP size |
q_proj shard on a rank |
|---|---|---|---|---|
| B built alone | B | 4 | 4 | (16, 64) |
| A (TP2), then B (TP4) | B | 4 | 2 | (32, 64): B is a TP2 model; its 3-step losses equal A's |
| A, clear the globals, then B | B | 4 | 4 | (16, 64) |
| A | 2 | 4 | (32, 64) |
- In the second case, B is not what its config asks for, and nothing warns.
- In the third case, B is right, but A now reports TP4 for its TP2 shards.
- A's training steps and checkpoint file names were still correct in this small test: AutoTP layers keep their own group, and ZeRO took its DP group at
initialize. - Every path that queries
groupsorengine.mpulater reads B's groups. That is why ZeRO3: Gradient norm allreduce for DP #1021 saves and restores the globals per engine.
- A's training steps and checkpoint file names were still correct in this small test: AutoTP layers keep their own group, and ZeRO took its DP group at
What reads and writes the globals
Writers (all in deepspeed/utils/groups.py unless noted):
_init_tp_mesh_deviceand_init_tp_groups_with_new_group:_TENSOR_MODEL_PARALLEL_GROUP,_MODEL_PARALLEL_GROUP,_DATA_PARALLEL_GROUP.mesh_device; also assigned inengine.py(groups.mesh_device = self.mesh_device).mpu, assigned inengine.py(groups.mpu = self.mpu)._MPU_TENSOR_MODEL_PARALLEL_WORLD_SIZE/_RANK.- The expert and expert-data-parallel tables (
_EXPERT_PARALLEL_GROUP, ...). These are keyed by name (ep_size_N), so two EP sizes can coexist. They are listed for completeness.
Readers outside groups.py (on master 84481c3; about 48 references in 13 files):
| Area | Where | Reads |
|---|---|---|
| Engine | engine.py: 13 places, including the dataloader check, tp_config.tp_group, rank getters, checkpoint tp_rank, AutoEP folding |
TP group, rank and size; DP rank; mpu |
| ZeRO / BF16 | stage_1_and_2.py, stage3.py (2), bf16_optimizer.py |
TP group for AutoEP folding; in stage3.py also for the universal-checkpoint load of AutoTP shards |
| AutoTP | auto_tp.py (set_tensor_parallel_config in training mode), tp_manager.py (2) |
TP group and its source rank |
| Checkpoint | model_checkpointing/utils.py (3), checkpoint/zero_affine.py (default tp_group) |
mpu, TP group |
| MoE / AutoEP | moe/sharded_moe.py, moe/mappings.py, moe/layer.py, auto_ep_layer.py |
mpu, TP group |
| Compile | compile/custom_ops/tp_collectives.py |
TP group |
Open PRs that add readers. #9037 (multi-turn tool-calling rollout, draft) reads the global TP group in runtime/rollout/hybrid_engine_rollout.py:
_host_tensor_parallel_group()builds Gloo groups overgroups.get_tensor_model_parallel_group()and usesgroups.get_tensor_model_parallel_src_rank()as the rank that forwards tool output;_sync_rng_across_tp()broadcasts the RNG state over the same group.
The rollout takes its TP size from engine.autotp_size() but the group from the module. Next to a trainer with another TP size, both would run over the first engine's groups. Reading engine.mpu instead would not help on master, since under AutoTP engine.mpu is the groups module itself. With stage 2 (#9120), engine.mpu is the engine's own groups object, so these readers can move to it.
Options
- Keep the globals; document a supported way to switch.
- Make what ZeRO3: Gradient norm allreduce for DP #1021 and [rollout] In-memory relayout between sharded layouts (two-stage relay, pipelined execution) and ZeRO-3 training-state move #9074 do external, for example a public
groupssnapshot/activate API. - Smallest change, but every engine switch must remember to activate, and any reader that runs at the wrong time reads the other engine's groups without an error.
- Make what ZeRO3: Gradient norm allreduce for DP #1021 and [rollout] In-memory relayout between sharded layouts (two-stage relay, pipelined execution) and ZeRO-3 training-state move #9074 do external, for example a public
- Each engine owns its groups (return to the pre-2022 design).
_init_tp_mesh_devicebuilds a small per-engine object with the same interface as today'smpu.engine.mpupoints to it instead of to the module.- The readers above take the group from their engine or layer, as [AutoTP] Replace tp_shard process-wide globals with per-model AutoTPMeta #8241 and [AutoTP] Support frozen AutoTP models and manage AutoTP mode per model #8851 did for shape and mode.
- The
groups.get_*functions remain for compatibility and keep returning the first engine's groups, as today. Single-engine users see no change.
- A guard only: raise if a second
initializeasks for a differentautotp_sizethan the existing groups. This fixes the silent misconfiguration but does not enable two layouts.
My preference is 2, with 3 as an immediate first step, done in stages:
- The guard (option 3), plus a test that builds TP2 and TP4 engines in one process.
- Per-engine group object;
engine.py, ZeRO and AutoTP readers move to it. - Checkpoint and
zero_affinereaders. - MoE / AutoEP and compile readers.
- Group lifetime: destroy an engine's groups with it, share groups between engines, replace them in place.
Each stage keeps the module getters working, so it can merge on its own.
Roadmap and status
Updated after the discussion below: @LiRunGuo and @Achyuthan-S both support option 2, with the guard first.
The ZeRO-3 rank-agreement checks (process-wide ids, trace state) are tracked in #9098 and fixed in a separate PR by Achyuthan-S; stage 2 (#9120) does not touch them.
| Stage | Scope | Owner | Status |
|---|---|---|---|
| 1. Guard | Raise when a later engine asks for another TP layout, including an engine without AutoTP and without its own mpu (@Achyuthan-S). Test with TP2/TP4 engines in one process. |
@jinyouzhi | #9088 (in review) |
| 2. Per-engine groups | Per-engine group object; engine.py, ZeRO, BF16 and AutoTP readers move to it. This includes _reduce_non_expert_gradients and the DP case @LiRunGuo reported: an engine without an mpu takes the previous engine's groups.mpu. The rollout readers of #9037 belong here too. |
@jinyouzhi | #9120 (draft, on top of #9088). Every AutoTP engine's engine.mpu is its own groups object; with the process-wide layout it holds the process-wide groups, so no communicator is added. An engine without an mpu no longer inherits an earlier engine's, and an engine can share another's groups with mpu=other.mpu. Tested with #9074's DP4 → DP2 → DP4 order. An engine on groups other than the process-wide ones still raises for mesh_param, AutoEP, the DeepCompile autotp pass and MoE (stage 4). The #9037 readers move once #9037 merges. @LiRunGuo offered to run it against #9074's tests |
| 3. Checkpoint readers | model_checkpointing/utils.py, the default tp_group of zero_affine.py, and the universal-checkpoint load of AutoTP shards in stage3.py |
stage3.py: @Achyuthan-S; rest open |
Partly done in #9120, because these readers break once an engine has groups of its own: the stage3.py universal-checkpoint load, the checkpoint.writer DP layout, and the default tp_group of zero_affine.py (taken from the model's AutoTP layers) use the engine's groups. Remaining: model_checkpointing/utils.py, and the zero_affine.py default replica group of parameters outside a ZeRO optimizer |
| 4. MoE / AutoEP, compile | sharded_moe.py, mappings.py, moe/layer.py, auto_ep_layer.py, tp_collectives.py |
open | not started |
| 5. Group lifetime | An owner and a reference count per group: an engine destroys the groups it created when it goes away, unless another engine shares them; groups passed in as mpu stay with their creator; the process-wide groups are never destroyed. Then, optionally, reuse groups automatically by layout (DeepSpeed creates groups without options, so equal ranks give equivalent groups). An API to replace an engine's groups in place, for what #9074's Zero3Repartition does by hand. |
open | not started. Until then, sharing is explicit (mpu=other.mpu) and no group is destroyed |
Related, outside these stages:
- Membership changes: with stage 2, groups belong to their engine; stage 5 lets them be destroyed when it goes away. Rebuilding the world after a rank leaves, and
use_local_synchronization, belong to the membership side of the elastic design.
Questions for the maintainers
@delock @tohtana @tjruwase (cc @LiRunGuo @Achyuthan-S from #9026)
- Should DeepSpeed support several engines with different TP/DP layouts in one process, or keep the process-wide registry and leave switching to external code (as ZeRO3: Gradient norm allreduce for DP #1021 and [rollout] In-memory relayout between sharded layouts (two-stage relay, pipelined execution) and ZeRO-3 training-state move #9074 do now)?
- If supported: is returning to engine-owned groups (option 2) the right direction, or is an official snapshot/activate API (option 1) preferred?
- In stages as above, or as one change?
- Should the module getters (
groups.get_tensor_model_parallel_group()and the rest) stay as they are, or become deprecated in favor of the engine's groups?
Tested on: 4× Intel Arc Pro B70 (XPU, oneCCL), torch 2.14.0.dev20260804+xpu, transformers 5.17.0, DeepSpeed master 84481c3. The #1021 results above were on 8× NVIDIA H20.
This RFC was drafted with AI assistance. I checked the code references and ran the reproducer myself.
Activity
[-][RFC] Per-engine parallel groups: follow-up to #8241 / #8851 for flexible TP/DP redistribution[/-][+][RFC] Cleanup: move process-wide parallel-group globals to per-engine groups — follow-up to #8241 / #8851[/+]LiRunGuo commentedon Nov 14, 2026
Thanks for writing this up, @jinyouzhi. Three data points for the RFC, from #9074 / #9076:
The DP side.
engine.pycopies an engine'smpuintogroups.mpuwhen one is given, andgroups._get_data_parallel_group()returns thatmpu's group. An engine initialized without anmpuafter one with anmpusilently takes the earlier engine's DP group. In #9074's first DP4 ↔ DP2 test, the engine over all ranks came out over a subgroup on some ranks and over the world on others, and its first collective hung. #9074 works around it by giving every engine its ownmpu, including the one over all ranks. That is enough for ZeRO-3, which takes its DP group atinitialize. It would not be enough for an engine whose gradient all-reduce goes through_reduce_non_expert_gradients, which asksgroupsat every step.Process-wide ids. ZeRO-3 numbers modules and parameters with process-wide counters (
_register_deepspeed_module(self, module, count=[0])andzero.Init.param_id), and the trace checks compare the raw ids. A new engine whose ranks have built different numbers of ZeRO-3 modules or parameters before fails its first trace check ("disagreement on list contents"). #9074 hit this and builds the widest engine first; @Achyuthan-S described it on #9076. It is the same kind of process-wide state, so it may belong in this RFC's list or next to it. One direction, untested: compare each id's position among the engine's own ids instead of the process-wide number.Membership changes. Both get stricter when the set of ranks changes during a job. A joining process starts with no
groups.mpuand its counters at 0, while the survivors keep theirs. After a rank leaves, any group still held in the globals contains it. Option 2 lets a new engine own groups over the new membership and lets the old ones be destroyed with the old engine; the ids would need the same per-engine treatment.+1 for option 2 with the guard first. I can run stage 2 against #9074's DP4 → DP2 → DP4 test when it is up.
Achyuthan-S commentedon Nov 14, 2026
Thanks @jinyouzhi, +1 for option 2 with the guard first. Two points from the affine-IR side:
engine.pybuilds the TP groups only whenautotp_size() > 1, so an engine withautotp_size1 that comes after a TP2 engine never reaches_init_tp_mesh_device. Itsget_tensor_parallel_rank()andget_model_parallel_rank()still read the module's groups, so they return the TP2 engine's ranks; built alone, the same calls fail the "not initialized" assertion. That is from reading the code, not a run. A guard ininitializethat compares the requested TP size, 1 included, with the groups already set would cover this case too.groups(stage3.py, from [AutoTP] Fix ZeRO-3 checkpoint consolidation to gather across TP and DP #8168), and ZeRO-3: take the AutoTP shard from universal checkpoints converted without TP #9069 sends more loads down that path. I can move that one onto the engine's group in the checkpoint stage.On the counters @LiRunGuo mentions: I'll open them as their own issue with a reproduction, as promised on #9076, and link it here. They need a per-engine fix inside ZeRO-3, next to stage 2.
sfc-gh-truwase commentedon Nov 17, 2026
@jinyouzhi and team, thanks for the great discussion here. I am aligned with option 2, independent engines per process. This is a feature that we have previously supported, but somehow seems to have suffered regressions.
- Independent engines per process
- Eliminating process-wide Globals
This is a very worthy cause with a long history. Any progress here would be extremely useful.
jinyouzhi commentedon Nov 17, 2026
Thanks @sfc-gh-truwase, and thanks for the history and links. I fully agree: this is more than restoring a basic capability, it is, as you put it, a very worthy cause with a long history. DS-Chat's actor, reference and critic, and Accelerate's multi-model setup will be the regression cases. Beyond them, the goal is for each engine to own its parallel groups and its process-wide state, so that engines in one process can use different TP/DP layouts, and an engine can be torn down and rebuilt when the set of ranks changes. That is what RL weight sync, #9074's DP4 → DP2 → DP4 switch and elastic jobs need. I'll read #2989 / #3592 before touching ZeRO-3.
On the guard, #9088:
Next steps:
LiRunGuo commentedon Nov 17, 2026
Thanks @sfc-gh-truwase for the direction and the history, and @jinyouzhi for the plan. For stage 2, these are the multi-engine checks on our side, all in #9074 now:
tests/unit/runtime/rollout/test_zero3_repartition.py(4 ranks, added to [rollout] In-memory relayout between sharded layouts (two-stage relay, pipelined execution) and ZeRO-3 training-state move #9074 today) is the DP4 → DP2 → DP4 test. On ranks 0 and 1 it holds four ZeRO-3 engines, each with its ownmpu:Zero3Repartition);Zero3StateMove);tests/unit/runtime/rollout/test_native_rollout.py::TestTrainerSwitch(2 ranks) moves a trainer between rank 0 alone and both ranks, with two engines per process.Two things in that code that stage 2 should make unnecessary:
TrainerMPU, made on every rank, with the engine over all ranks built first. The first part is thegroups.mpuinheritance; the ordering is [BUG] ZeRO-3 ids come from process-wide counters, so ranks that built different models fail the rank-agreement checks #9098.Zero3Repartitionre-pointsgroups.mpuand every reference the engine holds to its old data-parallel group by hand. With per-engine groups, that becomes a re-point of the engine's own groups.I'll also run the end-to-end DP4 → DP2 → DP4 on Qwen3-0.6B (4 GPUs, both paths) against stage 2 once it's up.
Give every AutoTP engine its own groups object
Reject a mesh engine after an AutoTP or explicit-mpu engine, and boun…