Skip to content

[AutoEP] overlap on micro batch level of compute and communication - #9107

Draft
pengdurice wants to merge 2 commits into
deepspeedai:masterfrom
pengdurice:peng-add-autoep-microbatch-overlap-v1
Draft

pengdurice wants to merge 2 commits into
deepspeedai:masterfrom
pengdurice:peng-add-autoep-microbatch-overlap-v1

Conversation

@pengdurice

Copy link
Copy Markdown
Contributor

Part of: #8672 (Q4 2026 roadmap), "Enable training of large scale models".

Depends on: PR 1 of this series (zero empty-expert weight gradients without a host sync). The diff below includes
it until it merges.

Summary

With a trained router, AutoEP's experts receive uneven numbers of rows, and every DeepEP dispatch and combine waits
for the rank that received the most. On Qwen3-30B-A3B with real weights and real text on 8 H100s, one rank receives up
to 2.36x the average rows, and about 8.6 s of DeepSpeed's 14.6 s of transfer time per step is this waiting (with
uniform random routing the same transfers take 6.0 s). Within one micro-batch there is little independent work to run
beside those transfers: master's overlap_weight_grad hides the expert weight gradients, about 1.3 s of a step.

Megatron-LM solves this across micro-batches (--overlap-moe-expert-parallel-comm): the backward of micro-batch i runs
together with the forward of micro-batch i+1, so every transfer of one has a computation of the other beside it.

This PR adds the same schedule to AutoEP, as MicroBatchOverlap:

from deepspeed.module_inject.auto_ep_microbatch_overlap import MicroBatchOverlap

overlap = MicroBatchOverlap(engine)
for step_batches in loader:  # the gradient_accumulation_steps micro-batches of one optimizer step
    losses = overlap.forward_backward([{"input_ids": ids, "labels": ids} for ids in step_batches])
    engine.step()

It is opt-in and changes nothing for code that does not construct it. Run with EP_AVOID_RECORD_STREAM=1 (DeepEP's own
environment variable; see "DeepEP's record_stream" below).

Results

Qwen3-30B-A3B with its released weights, its trained router (no forced routing, no dropped tokens) and real text
(PG-19, 4,096-token sequences); 1 node of 8 x H100 80GB HBM3, driver 595.112.07; NVIDIA's NeMo AutoModel 26.10
container (torch 2.14.0a0, CUDA 13.4, NCCL 2.30.7), DeepEP v2 2.1.0, transformers 5.18.0. Expert parallel 8,
micro-batch 4, 16 accumulation steps (512 sequences per step), ZeRO-2, full activation recompute, fused gate+up,
master's fused row weighting, fused RMSNorm and RoPE, Python GC off. All rows ran back to back in one job on one node;
14 steps each, step time is the mean of steps 5-14.

Precision is bf16 master weights and bf16 Adam states, because master cannot fit fp32 master weights with fp32 Adam
states at this micro-batch. In our study, precision changed step time by 1% or less; at learning rate 1e-5, bf16 master
weights learn more slowly (global loss 2.263 here against 2.234 with fp32 master weights), which is the same in every
row.

Tree Schedule DeepEP SMs Step time vs master Peak allocated in step 14 (highest GPU) Peak reserved in step 14 (highest GPU) Allocator cache flushes (14 steps, 8 GPUs) Global loss at step 14
master 44661723e plain accumulation loop 64 32.902 s 62.05 GiB 65.42 GiB 0 2.263059
this PR plain accumulation loop 64 32.739 s -0.5% (PR 1's change) 62.04 GiB 64.68 GiB 0 2.263051
this PR MicroBatchOverlap, EP_AVOID_RECORD_STREAM=1 16 27.763 s -15.6% 60.18 GiB 67.23 GiB 0 2.263046
this PR MicroBatchOverlap, variable unset 16 27.795 s -15.5% 60.18 GiB 73.61 GiB 1 2.263085

The losses agree within 4e-5 (the mean over all 8 GPUs). The overlap uses 16 DeepEP SMs because the computation beside
each transfer gives up those SMs; in our prototype, 12 to 24 SMs changed step time by less than 0.25 s, and the plain
loop is fastest with more SMs (64).

Context from the study this PR comes from: the same schedule in our full prototype (with fp32 master weights and bf16
Adam states, and the routing weight inside the expert activation as in #8748, which lets the recompute skip the down
projection) took a step from 29.40 s to 24.63 s (-16.2%). In 30-step runs on one node with each framework's best
expert-parallel settings, that prototype took 24.90 s, Megatron-LM 26.33 s (with its own overlap and HybridEP),
NeMo AutoModel 28.60 s and TorchTitan 37.90 s.

Design

The schedule. Forward layer k of micro-batch i+1 is paired with backward layer L-1-k of micro-batch i. In each
paired layer step:

Transfer on DeepEP's stream Computation beside it
replayed dispatch and combine backward of micro-batch i attention and router of micro-batch i+1
dispatch of micro-batch i+1 expert recompute and expert backward of micro-batch i
dispatch backward of micro-batch i experts of micro-batch i+1
combine of micro-batch i+1 attention backward of micro-batch i

The first micro-batch's forward and the last one's backward run alone. Both micro-batches keep their full size, so no
matrix multiply shrinks.

How it plugs into the engine. Each micro-batch's loss is the output of an autograd Function whose backward runs that
micro-batch's backward together with the next micro-batch's forward. forward_backward calls engine.backward(loss)
for every micro-batch and marks the last one as the accumulation boundary, so loss scaling, ZeRO's gradient hooks and
the boundary bookkeeping are DeepSpeed's own. The engine's backward prologue hooks (which engine.forward attaches to a
model output) are attached to each loss. Each parameter is used by exactly one piece of each micro-batch, so its
gradient is accumulated once per micro-batch, as without the overlap. The caller makes the one engine.step(), as with
engine.set_gradient_accumulation_boundary in a plain loop (and as accelerate drives DeepSpeed).

Recompute. Every decoder layer is recomputed in its backward, keeping only its input and its DeepEP handle, as with
full activation checkpointing. The recompute replays the forward's dispatch layout (master's
replay_dispatch contract) and does not repeat the forward's combine.

Memory: saved-input slot buffer. Each micro-batch keeps every layer's input until its backward. Allocated one at a
time, the inputs of the micro-batch going forward land between the other micro-batch's short-lived recompute blocks,
and the allocator cache fragments step after step: in our study the first version took 34.7 s per step instead of
about 24.9 s, with about 0.25 s per allocator cache flush. The inputs now live in one allocation of L+1 slots: an even
micro-batch keeps layer l's input in slot l, an odd one in slot L-l, so each forward layer writes into exactly the slot
its paired backward layer has just finished with.

DeepEP's record_stream. DeepEP v2 reads EP_AVOID_RECORD_STREAM with std::getenv on every asynchronous call
(stream_control_epilogue). Unset, it calls record_stream on every tensor of an in-flight transfer for both streams,
so the allocator cannot reuse those blocks until each stream's later work has run. Set, it keeps the tensors alive in
the returned event instead. MicroBatchOverlap waits on every event before it drops it, so either setting is correct;
we measured both (table above). Step time is the same (27.76 s with it, 27.80 s without), but without it the highest
GPU reserves 6.4 GiB more memory (73.6 against 67.2 GiB), and with less free memory that extra reserved memory turns
into cache flushes: in our fp32-master prototype, before the slot buffer existed, setting it cut flushes from 560 to
238-312 over 14 steps. The schedule logs a warning on rank 0 when the variable is unset. It does not set it itself,
because the variable is process-wide (open question 2).

Other details.

  • The forward dispatch starts from an event captured after the forward attention (previous_event), so the host's
    wait for the received row counts overlaps the backward's expert work.
  • While a transfer is in flight, the computation beside it runs with comm_num_sm SMs withheld through PyTorch's
    experimental SM carveout (without it our prototype was 1.08 s per step slower).
  • A backward that is not paired with a forward waits for the layer two steps earlier, so the host cannot queue many
    layers ahead of the GPU while DeepEP's asynchronous calls hold their blocks.
  • The model's preamble (embedding, attention mask, rotary tables) comes from the model's own forward, stopped by a
    pre-hook on the first decoder layer.

Code changes outside the new module.

  • DeepEPExchange gains capture_event, dispatch_async, replay_dispatch_async, dispatch_with_handle_async and
    combine_async: the existing calls with async_with_compute_stream=True, returning DeepEP's event. Each pair shares
    one private implementation (_expanded_dispatch, _replayed_dispatch, _dispatch_with_handle), so the dispatch
    arguments and the recompute's routing check exist once; the synchronous methods behave as before.
  • auto_ep_comm.sm_carveout: a context-manager form of the existing run_with_sm_carveout, which now uses it.
  • AutoEPMoELayer._ensure_deepep_exchange: the buffer construction and capacity check, moved out of _deepep_route
    unchanged so the schedule can call it.

Tests

Hardware and software as in "Results" (the distributed tests use 4 ranks), with DEEPSPEED_RUN_H100_TESTS=1 and
DeepEP v2's EP_DISABLE_GIN=1:

cd tests
python -m pytest -v unit/v1/moe/test_autoep_deepep_parity.py unit/module_inject/test_auto_ep_comm.py \
    unit/v1/moe/test_autoep_expert_activation.py unit/v1/moe/test_autoep_unit.py
EP_AVOID_RECORD_STREAM=1 python -m pytest -v unit/v1/moe/test_autoep_deepep_parity.py -k TestMicroBatchOverlap
Test file Result
test_autoep_deepep_parity.py 37 passed, including the 3 new tests
test_auto_ep_comm.py 74 passed (64 subtests)
test_autoep_expert_activation.py 97 passed, 12 skipped (skips need hardware or options not present)
test_autoep_unit.py 214 passed
TestMicroBatchOverlap with EP_AVOID_RECORD_STREAM=1 3 passed

New tests (TestMicroBatchOverlap in test_autoep_deepep_parity.py):

  • test_overlapped_micro_batches_match_the_plain_accumulation_loop[False/True]: a two-layer Qwen3-MoE on 4 GPUs with
    DeepEP, three micro-batches, even routing and skewed routing (one expert receives no rows). Each micro-batch's loss
    is within 1% and every parameter's accumulated gradient within relative L2 2e-2 of the plain loop
    (engine(...) + engine.backward), and the schedule runs 4 paired layer steps. It catches a forward that reads
    another micro-batch's rows or DeepEP handle, a backward that pairs the wrong layer's saved input, a lost gradient
    piece (embedding, attention, router score, expert weight, output layer), and a gradient accumulated twice or not at
    all in one micro-batch.
  • test_rejects_options_it_would_bypass: overlap_weight_grad=true is rejected at construction.

Changed test: test_auto_ep_comm.py::TestBufferLifecycle::test_outgrowing_the_buffer_names_the_remedy still goes
through _deepep_route. Its stand-in layer is a mock.Mock, so it now gets the real _ensure_deepep_exchange (where
the capacity check moved unchanged); otherwise the route would call the mock's method and the check would not run.

pre-commit run --files passes on every changed file (the new module was marked with git add -N so that
check-license, which uses git grep, can see it).

Compatibility

Area Validated Not validated
ZeRO stage 2 (tests and timing) stages 0 and 1 (accepted, not run); stage 3 is rejected
Offload none run; not rejected
Precision bf16 (tests: bf16 master weights and Adam states; timing: bf16 master weights and Adam states) fp16 with loss scaling (not run); fp32
Parallelism expert parallel 4 (tests) and 8 (timing) on one node, DeepEP v2 multiple nodes; tensor, pipeline and sequence parallelism; AutoEP+AutoTP folding (rejected)
Routing trained router, even and skewed test routing (including an expert with no rows)
Models Hugging Face Qwen3-MoE (two-layer test model, Qwen3-30B-A3B) other AutoEP presets
Layer options row_weighting_impl eager (test default) and fused (timing), gate_up_impl fused score_apply="pre"; overlap_weight_grad and skip_recompute_combine are rejected

Limits

Rejected at construction, rather than silently ignored: models that are not Hugging Face causal LMs with
model.model.layers; layers whose mlp is not an AutoEP layer on comm_backend="deepep" with autoep_size > 1;
ZeRO stage 3 (the schedule runs each layer's pieces itself, outside the module hooks through which ZeRO-3 gathers
parameters); shared experts, router logits (the router auxiliary loss) and AutoEP+AutoTP folding; tied embeddings;
attention dropout (the recompute does not restore the random state); and overlap_weight_grad and
skip_recompute_combine, which act inside the layer's own forward that this schedule does not call.

The training loop must hand all micro-batches of one optimizer step to forward_backward. A standard accelerate
loop, which calls backward once per micro-batch, does not use it without a small change.

Open questions for reviewers

  1. User-facing API. This PR keeps a standalone class. Alternatives: an engine method such as
    engine.forward_backward(micro_batches) behind an expert_parallel config flag, or an accelerate integration that
    collects the micro-batches of one step. Which would you prefer?
  2. EP_AVOID_RECORD_STREAM. Should DeepSpeed set it when the schedule is constructed? DeepEP reads it on every call,
    so setting it for the process also changes master's overlap_weight_grad path, which also waits on its event
    before dropping it. PR 2 of this series sets it only around that path's one asynchronous call; the same scoping
    would work here around each asynchronous call, at the cost of touching the process environment more often.
  3. Combination with AutoEP: opt-in routing weights inside the expert activation #8748 (routing weight inside the expert activation). With the weight inside the activation, the
    recompute no longer needs the down projection's output, so its matrix multiply can be skipped in the recompute. Our
    prototype did this; this PR leaves it out so that it does not depend on AutoEP: opt-in routing weights inside the expert activation #8748.

Not a duplicate

A search of open and merged DeepSpeed PRs since September 2026 for micro-batch overlap and AutoEP DeepEP work found none
that overlaps transfers across micro-batches. Master's overlap_weight_grad (#8739) and skip_recompute_combine
(#8745) work inside one micro-batch; this PR rejects them in the same model because it replaces both.

AI assistance

This PR was prepared with AI assistance (Claude): the code, the tests, the measurements and this description.

pengdurice and others added 2 commits November 17, 2026 22:08
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: pengdurice <pengduhit@gmail.com>
…/backward schedule)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: pengdurice <pengduhit@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant