Skip to content

AutoEP: support comm_backend="deepep" with AutoEP+AutoTP folding - #9112

Draft
pengdurice wants to merge 1 commit into
deepspeedai:masterfrom
pengdurice:peng-autoep-deepep-folded-tp-v1
Draft

pengdurice wants to merge 1 commit into
deepspeedai:masterfrom
pengdurice:peng-autoep-deepep-folded-tp-v1

Conversation

@pengdurice

Copy link
Copy Markdown
Contributor

AutoEP: support comm_backend="deepep" with AutoEP+AutoTP folding

Part of: #8672 (Q4 2026 roadmap), "Mixture-of-Expert (MoE) Training Scaling and Kernel Development": "AutoEP Performance Optimization & Parallel Folding (AutoEP + AutoTP)" (PoC @tohtana @pengdurice). Follows #8670 and #8847.

Problem

comm_backend="deepep" was refused whenever tensor_parallel.autotp_size > 1:
AutoEPMoELayer.set_deepspeed_parallelism (deepspeed/module_inject/auto_ep_layer.py) raised ValueError. (Its
message also named the wrong setting, expert_tensor_parallel_size, for the AutoTP size.)

The reason was that folding and DeepEP exchange rows in different shapes:

  • Under folding, the tensor-parallel (TP) peers of a group hold the same tokens and split the routed assignments
    between them (partition_assignments in deepspeed/moe/ep_tp_dispatch.py: peer p keeps the assignments whose
    per-expert ordinal is p modulo tp_size). Each peer dispatches one row per kept assignment through the
    collective all-to-all; after the experts, restore_combined all-gathers the rows and their metadata over the TP
    group and does the weighted sum per token.
  • DeepEP takes the routing token-major ([tokens, top_k] expert indices and weights), dispatches whole tokens, and
    its combine returns one row per token, already summed over the token's slots. There are no per-assignment rows
    and no metadata for restore_combined to gather.

So a job that wanted DeepEP's dispatch and combine kernels could not fold tensor parallelism into AutoEP, and the
other way round.

Change

  • AutoEPMoELayer._forward: under folding, the DeepEP branch first calls the new _partition_routing_over_tp,
    which splits the assignments with the same partition_assignments rule as the collective path and expresses the
    split in DeepEP's own layout: the expert index of every assignment another peer keeps is set to -1, which DeepEP's
    dispatch documents as "no selection" (deep_ep/buffers/elastic.py, dispatch), and its routing weight to 0. A
    torch.where keeps the weight gradient flowing to the kept slots only. Each peer therefore sends only its share.
  • _deepep_route takes the restore context and, after DeepEP's combine has returned the peer's partial output (one
    row per token, summed over the assignments the peer kept), calls the new _restore_folded_deepep: an all-reduce
    of the [tokens, hidden] partials over the TP group (sum_partial_rows_over_tp, new in
    deepspeed/moe/ep_tp_dispatch.py). Its backward all-reduces the gradient too, as _AllGatherVariableRows does on
    the collective path, so every gradient carries the same tp_size factor and the folded AVERAGE and
    EXPERT_TP_CANCEL corrections apply unchanged. The dispatch counters (_last_folding_dispatch_counters) are set as
    on the collective path.
  • skip_recompute_combine: in the recorded recompute nothing reads the output, so the all-reduce is skipped in the
    forward and only its backward is recorded.
  • validate_folding_routing=true runs the same payload consistency check as the collective path and, in addition,
    checks that the peers' kept assignments add up to the total.
  • tp_keep_mask (new, ep_tp_dispatch.py) turns a peer's partition_assignments result into the [tokens, top_k]
    mask.
  • The payload construction of the collective folded branch moved into _partition_assignments_over_tp, which both
    branches call; the collective branch is otherwise unchanged.
  • The refusal in set_deepspeed_parallelism is removed. overlap_weight_grad is now refused with
    tensor_parallel.autotp_size > 1 by validate_autoep_config, because the weight gradients it returns from the
    DeepEP dispatch backward have not been validated under folding.
  • Docs: docs/code-docs/source/autoep.rst (communication backend section and the folding bullet) and
    docs/_pages/config-json.md (folding constraint and the overlap_weight_grad row).

What stays as it was: the collective path, DeepEP without folding, the DeepEP kernels and wrappers in
auto_ep_comm.py, the gradient reduction strategies in auto_ep_folding.py, the fused_weighted_sum refusal with
DeepEP (DeepEP already sums the rows).

Tests

Hardware: 1 node of 8 NVIDIA H100 80GB (driver 595.112.07), torch 2.8.0+cu128, Triton 3.4.0, DeepEP 2.1.0
(ElasticBuffer, EP_DISABLE_GIN=1), transformers 5.15.1, DEEPSPEED_RUN_H100_TESTS=1, use_grouped_mm: false
(the test fixture's default). The CPU runs: the login node, torch 2.8.0, gloo.

New tests:

Test What it checks Ranks Result
test_autoep_deepep_parity.py::TestDeepEPFoldedMatchesCollective::test_forward_and_backward_match_the_folded_collective_path (ZeRO 1 and 3, validate_folding_routing=true) one step with TP 2 x EP 2 through real DeepEP against the folded collective path: output, input gradient and every parameter gradient within the tolerances of TestDeepEPMatchesCollective 4 GPUs 2 passed
...::test_skip_recompute_combine_matches_the_folded_collective_path the same under Hugging Face per-layer reentrant checkpointing with skip_recompute_combine 4 GPUs passed
...::test_fused_row_weighting_matches_the_folded_collective_path the same with row_weighting_impl="fused" 4 GPUs passed
test_autoep_deepep_parity.py::test_cpu_gloo_folded_deepep_route_matches_the_folded_collective_path the DeepEP route on a fake exchange built from the collective primitives (dispatch honours -1, combine sums without weights), TP 2 x EP 2 in fp32, against the folded collective path: rtol=1e-4. Runs without DeepEP or CUDA. With the all-reduce's backward replaced by the identity, this test fails (input gradient off by up to 552x), so it does cover the gradient convention. 4 CPU processes passed (login node and GPU node)
test_autoep_autotp_dispatch.py::test_tp_keep_mask_gives_every_assignment_to_exactly_one_peer (two shapes, one with empty peers) the masks of the tp_size peers are disjoint, cover every assignment, and name the same (token, slot) pairs as partition_assignments' row list CPU 2 passed
test_autoep_autotp_dispatch.py::test_sum_partial_rows_over_tp_sums_forward_and_keeps_the_tp_factor_backward forward is the sum over peers; backward is tp_size times the upstream gradient; record_only skips the forward collective and keeps the backward 2 CPU processes passed
test_auto_ep_comm.py::TestAutoEPCommBackendSelection::test_deepep_accepts_folded_tensor_parallelism the config is accepted with tp_size=2 CPU passed
test_autoep_expert_activation.py::...::test_overlap_weight_grad_settings_are_validated (new case) overlap_weight_grad with tp_size=2 raises CPU passed

Existing tests, on the PR tree (every suite that runs through the shared payload refactor or the DeepEP route):

Suite Result on the PR tree Note
test_autoep_deepep_parity.py::TestDeepEPMatchesCollective::test_forward_and_backward_match_the_collective_path (DeepEP without folding) passed
the rest of test_autoep_deepep_parity.py (31 cases) 26 passed, 5 failed the 5 fail the same on master in the same job: the 4 test_overlapped_weight_gradients_match_the_serial_backward cases (overlap_weight_grad, CUDA "unspecified launch failure") and TestDeepEPColdStart (its mock.patch target deepspeed.comm.torch.known_world_size does not exist in this torch). The two test_cleanup_matches_legacy_preparation cases failed in the first run because the unfolded path passed a new keyword to _deepep_route, which the test's legacy stand-in does not take; the unfolded call now keeps its old form, and both cases pass
test_autoep_autotp_grad_parity.py 20 passed
test_autoep_autotp_runtime.py 6 passed
test_autoep_autotp_sharded_attention.py 69 passed, 2 deselected
test_autoep_autotp_bf16_folding_parity.py 5 passed
test_autoep_autotp_zero1_overlap_folding.py 3 passed
test_autoep_autotp_checkpoint.py 8 passed
test_autoep_autotp_dispatch.py 12 passed
test_autoep_unit.py 214 passed
test_auto_ep_comm.py 66 passed, 6 failed, 3 skipped the 6 (TestSkipRecomputeCombine) fail the same on master in the same job: they run torch._grouped_mm on CPU tensors, which this torch has no CPU kernel for

The test helper _snapshot_fp32_parameters in test_autoep_deepep_parity.py entered its stage-0 branch for ZeRO-3
too (ZeRO-3's optimizer also has fp16_groups) and failed; it now takes that branch only at stage 0, and reads an
fp32 parameter as its own master copy. _build_test_engine and _run_one_step take tp_size and
mixed_precision.

Validated cells: ZeRO 1 and 3 in bf16 with real DeepEP; ZeRO 0 in fp32 with the fake exchange; reentrant
per-layer checkpointing with skip_recompute_combine; eager and fused row weighting; score_apply="post"
(the fixture's preset). Not validated: ZeRO 2 with DeepEP under folding (the folded collective path covers ZeRO 2),
score_apply="pre" under folding, overlap_weight_grad (refused), optimizer offload, multi-node.

Notes

  • Communication: the collective folded restore all-gathers [tokens * top_k, hidden] rows over the TP group; the
    DeepEP folded restore all-reduces [tokens, hidden] partial outputs, top_k times fewer elements, and the
    all-to-all itself carries each token's row once per kept assignment, as on the collective path.
  • Precision: the partial outputs are summed in the activation dtype, as the collective folded restore sums its rows
    in the activation dtype.
  • Not validated: overlap_weight_grad under folding (refused), expert tensor parallelism (refused by folding
    itself), DeepEP's low-latency mode (not used by DeepSpeed), multi-node runs, and end-to-end throughput. No
    throughput or peak-memory numbers are reported: this change enables a configuration and changes no existing path.

Duplicate check

Searched open pull requests on 2026-11-17 ("DeepEP folding", "DeepEP AutoTP", "DeepEP tensor parallel"): none
enables DeepEP under AutoEP+AutoTP folding. #8847 enables the fused combine under folding on the collective path;
#8759 is unrelated (attention-output stash).

AI assistance

This change, its tests and this description were prepared with AI assistance (Claude).

Signed-off-by: pengdurice <pengduhit@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant