Repository navigation
AutoEP: support comm_backend="deepep" with AutoEP+AutoTP folding - #9112
Draft
pengdurice wants to merge 1 commit into
Draft
pengdurice wants to merge 1 commit into
pengdurice wants to merge 1 commit into
Conversation
Signed-off-by: pengdurice <pengduhit@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AutoEP: support
comm_backend="deepep"with AutoEP+AutoTP foldingPart of: #8672 (Q4 2026 roadmap), "Mixture-of-Expert (MoE) Training Scaling and Kernel Development": "AutoEP Performance Optimization & Parallel Folding (AutoEP + AutoTP)" (PoC @tohtana @pengdurice). Follows #8670 and #8847.
Problem
comm_backend="deepep"was refused whenevertensor_parallel.autotp_size > 1:AutoEPMoELayer.set_deepspeed_parallelism(deepspeed/module_inject/auto_ep_layer.py) raisedValueError. (Itsmessage also named the wrong setting,
expert_tensor_parallel_size, for the AutoTP size.)The reason was that folding and DeepEP exchange rows in different shapes:
between them (
partition_assignmentsindeepspeed/moe/ep_tp_dispatch.py: peerpkeeps the assignments whoseper-expert ordinal is
pmodulotp_size). Each peer dispatches one row per kept assignment through thecollective all-to-all; after the experts,
restore_combinedall-gathers the rows and their metadata over the TPgroup and does the weighted sum per token.
[tokens, top_k]expert indices and weights), dispatches whole tokens, andits combine returns one row per token, already summed over the token's slots. There are no per-assignment rows
and no metadata for
restore_combinedto gather.So a job that wanted DeepEP's dispatch and combine kernels could not fold tensor parallelism into AutoEP, and the
other way round.
Change
AutoEPMoELayer._forward: under folding, the DeepEP branch first calls the new_partition_routing_over_tp,which splits the assignments with the same
partition_assignmentsrule as the collective path and expresses thesplit in DeepEP's own layout: the expert index of every assignment another peer keeps is set to -1, which DeepEP's
dispatch documents as "no selection" (
deep_ep/buffers/elastic.py,dispatch), and its routing weight to 0. Atorch.wherekeeps the weight gradient flowing to the kept slots only. Each peer therefore sends only its share._deepep_routetakes the restore context and, after DeepEP's combine has returned the peer's partial output (onerow per token, summed over the assignments the peer kept), calls the new
_restore_folded_deepep: an all-reduceof the
[tokens, hidden]partials over the TP group (sum_partial_rows_over_tp, new indeepspeed/moe/ep_tp_dispatch.py). Its backward all-reduces the gradient too, as_AllGatherVariableRowsdoes onthe collective path, so every gradient carries the same
tp_sizefactor and the foldedAVERAGEandEXPERT_TP_CANCELcorrections apply unchanged. The dispatch counters (_last_folding_dispatch_counters) are set ason the collective path.
skip_recompute_combine: in the recorded recompute nothing reads the output, so the all-reduce is skipped in theforward and only its backward is recorded.
validate_folding_routing=trueruns the same payload consistency check as the collective path and, in addition,checks that the peers' kept assignments add up to the total.
tp_keep_mask(new,ep_tp_dispatch.py) turns a peer'spartition_assignmentsresult into the[tokens, top_k]mask.
_partition_assignments_over_tp, which bothbranches call; the collective branch is otherwise unchanged.
set_deepspeed_parallelismis removed.overlap_weight_gradis now refused withtensor_parallel.autotp_size > 1byvalidate_autoep_config, because the weight gradients it returns from theDeepEP dispatch backward have not been validated under folding.
docs/code-docs/source/autoep.rst(communication backend section and the folding bullet) anddocs/_pages/config-json.md(folding constraint and theoverlap_weight_gradrow).What stays as it was: the collective path, DeepEP without folding, the DeepEP kernels and wrappers in
auto_ep_comm.py, the gradient reduction strategies inauto_ep_folding.py, thefused_weighted_sumrefusal withDeepEP (DeepEP already sums the rows).
Tests
Hardware: 1 node of 8 NVIDIA H100 80GB (driver 595.112.07), torch 2.8.0+cu128, Triton 3.4.0, DeepEP 2.1.0
(
ElasticBuffer,EP_DISABLE_GIN=1), transformers 5.15.1,DEEPSPEED_RUN_H100_TESTS=1,use_grouped_mm: false(the test fixture's default). The CPU runs: the login node, torch 2.8.0, gloo.
New tests:
test_autoep_deepep_parity.py::TestDeepEPFoldedMatchesCollective::test_forward_and_backward_match_the_folded_collective_path(ZeRO 1 and 3,validate_folding_routing=true)TestDeepEPMatchesCollective...::test_skip_recompute_combine_matches_the_folded_collective_pathskip_recompute_combine...::test_fused_row_weighting_matches_the_folded_collective_pathrow_weighting_impl="fused"test_autoep_deepep_parity.py::test_cpu_gloo_folded_deepep_route_matches_the_folded_collective_pathrtol=1e-4. Runs without DeepEP or CUDA. With the all-reduce's backward replaced by the identity, this test fails (input gradient off by up to 552x), so it does cover the gradient convention.test_autoep_autotp_dispatch.py::test_tp_keep_mask_gives_every_assignment_to_exactly_one_peer(two shapes, one with empty peers)tp_sizepeers are disjoint, cover every assignment, and name the same (token, slot) pairs aspartition_assignments' row listtest_autoep_autotp_dispatch.py::test_sum_partial_rows_over_tp_sums_forward_and_keeps_the_tp_factor_backwardtp_sizetimes the upstream gradient;record_onlyskips the forward collective and keeps the backwardtest_auto_ep_comm.py::TestAutoEPCommBackendSelection::test_deepep_accepts_folded_tensor_parallelismtp_size=2test_autoep_expert_activation.py::...::test_overlap_weight_grad_settings_are_validated(new case)overlap_weight_gradwithtp_size=2raisesExisting tests, on the PR tree (every suite that runs through the shared payload refactor or the DeepEP route):
test_autoep_deepep_parity.py::TestDeepEPMatchesCollective::test_forward_and_backward_match_the_collective_path(DeepEP without folding)test_autoep_deepep_parity.py(31 cases)test_overlapped_weight_gradients_match_the_serial_backwardcases (overlap_weight_grad, CUDA "unspecified launch failure") andTestDeepEPColdStart(itsmock.patchtargetdeepspeed.comm.torch.known_world_sizedoes not exist in this torch). The twotest_cleanup_matches_legacy_preparationcases failed in the first run because the unfolded path passed a new keyword to_deepep_route, which the test's legacy stand-in does not take; the unfolded call now keeps its old form, and both cases passtest_autoep_autotp_grad_parity.pytest_autoep_autotp_runtime.pytest_autoep_autotp_sharded_attention.pytest_autoep_autotp_bf16_folding_parity.pytest_autoep_autotp_zero1_overlap_folding.pytest_autoep_autotp_checkpoint.pytest_autoep_autotp_dispatch.pytest_autoep_unit.pytest_auto_ep_comm.pyTestSkipRecomputeCombine) fail the same on master in the same job: they runtorch._grouped_mmon CPU tensors, which this torch has no CPU kernel forThe test helper
_snapshot_fp32_parametersintest_autoep_deepep_parity.pyentered its stage-0 branch for ZeRO-3too (ZeRO-3's optimizer also has
fp16_groups) and failed; it now takes that branch only at stage 0, and reads anfp32 parameter as its own master copy.
_build_test_engineand_run_one_steptaketp_sizeandmixed_precision.Validated cells: ZeRO 1 and 3 in bf16 with real DeepEP; ZeRO 0 in fp32 with the fake exchange; reentrant
per-layer checkpointing with
skip_recompute_combine; eager and fused row weighting;score_apply="post"(the fixture's preset). Not validated: ZeRO 2 with DeepEP under folding (the folded collective path covers ZeRO 2),
score_apply="pre"under folding,overlap_weight_grad(refused), optimizer offload, multi-node.Notes
[tokens * top_k, hidden]rows over the TP group; theDeepEP folded restore all-reduces
[tokens, hidden]partial outputs,top_ktimes fewer elements, and theall-to-all itself carries each token's row once per kept assignment, as on the collective path.
in the activation dtype.
overlap_weight_gradunder folding (refused), expert tensor parallelism (refused by foldingitself), DeepEP's low-latency mode (not used by DeepSpeed), multi-node runs, and end-to-end throughput. No
throughput or peak-memory numbers are reported: this change enables a configuration and changes no existing path.
Duplicate check
Searched open pull requests on 2026-11-17 ("DeepEP folding", "DeepEP AutoTP", "DeepEP tensor parallel"): none
enables DeepEP under AutoEP+AutoTP folding. #8847 enables the fused combine under folding on the collective path;
#8759 is unrelated (attention-output stash).
AI assistance
This change, its tests and this description were prepared with AI assistance (Claude).