Repository navigation
[AutoEP] overlap on micro batch level of compute and communication - #9107
Draft
pengdurice wants to merge 2 commits into
Draft
pengdurice wants to merge 2 commits into
pengdurice wants to merge 2 commits into
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: pengdurice <pengduhit@gmail.com>
…/backward schedule) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: pengdurice <pengduhit@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of: #8672 (Q4 2026 roadmap), "Enable training of large scale models".
Depends on: PR 1 of this series (zero empty-expert weight gradients without a host sync). The diff below includes
it until it merges.
Summary
With a trained router, AutoEP's experts receive uneven numbers of rows, and every DeepEP dispatch and combine waits
for the rank that received the most. On Qwen3-30B-A3B with real weights and real text on 8 H100s, one rank receives up
to 2.36x the average rows, and about 8.6 s of DeepSpeed's 14.6 s of transfer time per step is this waiting (with
uniform random routing the same transfers take 6.0 s). Within one micro-batch there is little independent work to run
beside those transfers: master's
overlap_weight_gradhides the expert weight gradients, about 1.3 s of a step.Megatron-LM solves this across micro-batches (
--overlap-moe-expert-parallel-comm): the backward of micro-batch i runstogether with the forward of micro-batch i+1, so every transfer of one has a computation of the other beside it.
This PR adds the same schedule to AutoEP, as
MicroBatchOverlap:It is opt-in and changes nothing for code that does not construct it. Run with
EP_AVOID_RECORD_STREAM=1(DeepEP's ownenvironment variable; see "DeepEP's record_stream" below).
Results
Qwen3-30B-A3B with its released weights, its trained router (no forced routing, no dropped tokens) and real text
(PG-19, 4,096-token sequences); 1 node of 8 x H100 80GB HBM3, driver 595.112.07; NVIDIA's NeMo AutoModel 26.10
container (torch 2.14.0a0, CUDA 13.4, NCCL 2.30.7), DeepEP v2 2.1.0, transformers 5.18.0. Expert parallel 8,
micro-batch 4, 16 accumulation steps (512 sequences per step), ZeRO-2, full activation recompute, fused gate+up,
master's fused row weighting, fused RMSNorm and RoPE, Python GC off. All rows ran back to back in one job on one node;
14 steps each, step time is the mean of steps 5-14.
Precision is bf16 master weights and bf16 Adam states, because master cannot fit fp32 master weights with fp32 Adam
states at this micro-batch. In our study, precision changed step time by 1% or less; at learning rate 1e-5, bf16 master
weights learn more slowly (global loss 2.263 here against 2.234 with fp32 master weights), which is the same in every
row.
44661723eMicroBatchOverlap,EP_AVOID_RECORD_STREAM=1MicroBatchOverlap, variable unsetThe losses agree within 4e-5 (the mean over all 8 GPUs). The overlap uses 16 DeepEP SMs because the computation beside
each transfer gives up those SMs; in our prototype, 12 to 24 SMs changed step time by less than 0.25 s, and the plain
loop is fastest with more SMs (64).
Context from the study this PR comes from: the same schedule in our full prototype (with fp32 master weights and bf16
Adam states, and the routing weight inside the expert activation as in #8748, which lets the recompute skip the down
projection) took a step from 29.40 s to 24.63 s (-16.2%). In 30-step runs on one node with each framework's best
expert-parallel settings, that prototype took 24.90 s, Megatron-LM 26.33 s (with its own overlap and HybridEP),
NeMo AutoModel 28.60 s and TorchTitan 37.90 s.
Design
The schedule. Forward layer k of micro-batch i+1 is paired with backward layer L-1-k of micro-batch i. In each
paired layer step:
The first micro-batch's forward and the last one's backward run alone. Both micro-batches keep their full size, so no
matrix multiply shrinks.
How it plugs into the engine. Each micro-batch's loss is the output of an autograd Function whose backward runs that
micro-batch's backward together with the next micro-batch's forward.
forward_backwardcallsengine.backward(loss)for every micro-batch and marks the last one as the accumulation boundary, so loss scaling, ZeRO's gradient hooks and
the boundary bookkeeping are DeepSpeed's own. The engine's backward prologue hooks (which
engine.forwardattaches to amodel output) are attached to each loss. Each parameter is used by exactly one piece of each micro-batch, so its
gradient is accumulated once per micro-batch, as without the overlap. The caller makes the one
engine.step(), as withengine.set_gradient_accumulation_boundaryin a plain loop (and as accelerate drives DeepSpeed).Recompute. Every decoder layer is recomputed in its backward, keeping only its input and its DeepEP handle, as with
full activation checkpointing. The recompute replays the forward's dispatch layout (master's
replay_dispatchcontract) and does not repeat the forward's combine.Memory: saved-input slot buffer. Each micro-batch keeps every layer's input until its backward. Allocated one at a
time, the inputs of the micro-batch going forward land between the other micro-batch's short-lived recompute blocks,
and the allocator cache fragments step after step: in our study the first version took 34.7 s per step instead of
about 24.9 s, with about 0.25 s per allocator cache flush. The inputs now live in one allocation of L+1 slots: an even
micro-batch keeps layer l's input in slot l, an odd one in slot L-l, so each forward layer writes into exactly the slot
its paired backward layer has just finished with.
DeepEP's record_stream. DeepEP v2 reads
EP_AVOID_RECORD_STREAMwithstd::getenvon every asynchronous call(
stream_control_epilogue). Unset, it callsrecord_streamon every tensor of an in-flight transfer for both streams,so the allocator cannot reuse those blocks until each stream's later work has run. Set, it keeps the tensors alive in
the returned event instead.
MicroBatchOverlapwaits on every event before it drops it, so either setting is correct;we measured both (table above). Step time is the same (27.76 s with it, 27.80 s without), but without it the highest
GPU reserves 6.4 GiB more memory (73.6 against 67.2 GiB), and with less free memory that extra reserved memory turns
into cache flushes: in our fp32-master prototype, before the slot buffer existed, setting it cut flushes from 560 to
238-312 over 14 steps. The schedule logs a warning on rank 0 when the variable is unset. It does not set it itself,
because the variable is process-wide (open question 2).
Other details.
previous_event), so the host'swait for the received row counts overlaps the backward's expert work.
comm_num_smSMs withheld through PyTorch'sexperimental SM carveout (without it our prototype was 1.08 s per step slower).
layers ahead of the GPU while DeepEP's asynchronous calls hold their blocks.
pre-hook on the first decoder layer.
Code changes outside the new module.
DeepEPExchangegainscapture_event,dispatch_async,replay_dispatch_async,dispatch_with_handle_asyncandcombine_async: the existing calls withasync_with_compute_stream=True, returning DeepEP's event. Each pair sharesone private implementation (
_expanded_dispatch,_replayed_dispatch,_dispatch_with_handle), so the dispatcharguments and the recompute's routing check exist once; the synchronous methods behave as before.
auto_ep_comm.sm_carveout: a context-manager form of the existingrun_with_sm_carveout, which now uses it.AutoEPMoELayer._ensure_deepep_exchange: the buffer construction and capacity check, moved out of_deepep_routeunchanged so the schedule can call it.
Tests
Hardware and software as in "Results" (the distributed tests use 4 ranks), with
DEEPSPEED_RUN_H100_TESTS=1andDeepEP v2's
EP_DISABLE_GIN=1:cd tests python -m pytest -v unit/v1/moe/test_autoep_deepep_parity.py unit/module_inject/test_auto_ep_comm.py \ unit/v1/moe/test_autoep_expert_activation.py unit/v1/moe/test_autoep_unit.py EP_AVOID_RECORD_STREAM=1 python -m pytest -v unit/v1/moe/test_autoep_deepep_parity.py -k TestMicroBatchOverlaptest_autoep_deepep_parity.pytest_auto_ep_comm.pytest_autoep_expert_activation.pytest_autoep_unit.pyTestMicroBatchOverlapwithEP_AVOID_RECORD_STREAM=1New tests (
TestMicroBatchOverlapintest_autoep_deepep_parity.py):test_overlapped_micro_batches_match_the_plain_accumulation_loop[False/True]: a two-layer Qwen3-MoE on 4 GPUs withDeepEP, three micro-batches, even routing and skewed routing (one expert receives no rows). Each micro-batch's loss
is within 1% and every parameter's accumulated gradient within relative L2 2e-2 of the plain loop
(
engine(...)+engine.backward), and the schedule runs 4 paired layer steps. It catches a forward that readsanother micro-batch's rows or DeepEP handle, a backward that pairs the wrong layer's saved input, a lost gradient
piece (embedding, attention, router score, expert weight, output layer), and a gradient accumulated twice or not at
all in one micro-batch.
test_rejects_options_it_would_bypass:overlap_weight_grad=trueis rejected at construction.Changed test:
test_auto_ep_comm.py::TestBufferLifecycle::test_outgrowing_the_buffer_names_the_remedystill goesthrough
_deepep_route. Its stand-in layer is amock.Mock, so it now gets the real_ensure_deepep_exchange(wherethe capacity check moved unchanged); otherwise the route would call the mock's method and the check would not run.
pre-commit run --filespasses on every changed file (the new module was marked withgit add -Nso thatcheck-license, which usesgit grep, can see it).Compatibility
row_weighting_impleager (test default) and fused (timing),gate_up_implfusedscore_apply="pre";overlap_weight_gradandskip_recompute_combineare rejectedLimits
Rejected at construction, rather than silently ignored: models that are not Hugging Face causal LMs with
model.model.layers; layers whosemlpis not an AutoEP layer oncomm_backend="deepep"withautoep_size > 1;ZeRO stage 3 (the schedule runs each layer's pieces itself, outside the module hooks through which ZeRO-3 gathers
parameters); shared experts, router logits (the router auxiliary loss) and AutoEP+AutoTP folding; tied embeddings;
attention dropout (the recompute does not restore the random state); and
overlap_weight_gradandskip_recompute_combine, which act inside the layer's own forward that this schedule does not call.The training loop must hand all micro-batches of one optimizer step to
forward_backward. A standard accelerateloop, which calls backward once per micro-batch, does not use it without a small change.
Open questions for reviewers
engine.forward_backward(micro_batches)behind anexpert_parallelconfig flag, or an accelerate integration thatcollects the micro-batches of one step. Which would you prefer?
EP_AVOID_RECORD_STREAM. Should DeepSpeed set it when the schedule is constructed? DeepEP reads it on every call,so setting it for the process also changes master's
overlap_weight_gradpath, which also waits on its eventbefore dropping it. PR 2 of this series sets it only around that path's one asynchronous call; the same scoping
would work here around each asynchronous call, at the cost of touching the process environment more often.
recompute no longer needs the down projection's output, so its matrix multiply can be skipped in the recompute. Our
prototype did this; this PR leaves it out so that it does not depend on AutoEP: opt-in routing weights inside the expert activation #8748.
Not a duplicate
A search of open and merged DeepSpeed PRs since September 2026 for micro-batch overlap and AutoEP DeepEP work found none
that overlaps transfers across micro-batches. Master's
overlap_weight_grad(#8739) andskip_recompute_combine(#8745) work inside one micro-batch; this PR rejects them in the same model because it replaces both.
AI assistance
This PR was prepared with AI assistance (Claude): the code, the tests, the measurements and this description.