Repository navigation
[AutoTP] Add support for vocab-parallel LM head and related configurations - #8309
Conversation
|
Hi @jinyouzhi when this PR is ready for review just ping me, thanks! |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
ab4139a to
8cf638a
Compare
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Require an explicit sequence-parallel group when the legacy wrapper gathers losses, and document the distinct backward conventions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8f03c27ea0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Replace the output head in heuristic AutoTP and preserve original tied-weight evidence before embedding traversal mutates parameters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Resolve and validate the heuristic output head before transformer partitioning, cache the candidate for replacement, and skip tied-weight scans when the feature is disabled. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Hi, this PR is ready for review now. Thank you for your patience! @delock During development, I identified and fixed some issues related to SP in a separate PR: #8457. |
|
Hi @jinyouzhi Thanks for the PR — the loss kernel and backward-compat handling for 1. License header (new file) 2. Suggestion (non-blocking): a minimal save → load → forward-loss parity test for the vocab-sharded head would be good to have, since checkpoint paths have no coverage for the new |
New files carry only the SPDX identifier and team line. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Fail when vocab_parallel_lm_head finds no supported head instead of degrading into an ordinary gathered head with no distributed loss, and warn when the flag supersedes an explicit lm_head spec from partition_config or a HuggingFace tp_plan. Also partition the embedding with a row spec in the tied-head regression so it exercises the captured tie identities; the previous embedding_rowwise plan converts to a skip spec, which left the embedding untouched and the guard unreached. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Save and reload a vocab-parallel LM head and compare the causal-LM loss, since no existing test exercises checkpoint paths for the VocabParallelLinear layout. The reloaded model starts from a different initialization so an unrestored checkpoint cannot pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
|
@delock Thank you for your detailed review! I have updated the PR according to your comments and guidance. Please take another look when you have a chance. |
101 rows over two ranks gives shards of 51 and 50, so the map has to carry the per-rank extents rather than assume an even split, and restoring at TP1 merges two unequal shards into one tensor. The fixture asserts the emitted map records that split before saving, so the test fails if conversion silently falls back to the vocabulary category. Adapted from @jinyouzhi's cross-tp vocab test in deepspeedai#8309, using the column partition already in tree so it carries no dependency on that PR. Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com>
101 rows over two ranks gives shards of 51 and 50, so the map has to carry the per-rank extents rather than assume an even split, and restoring at TP1 merges two unequal shards into one tensor. The fixture asserts the emitted map records that split before saving, so the test fails if conversion silently falls back to the vocabulary category. Adapted from @jinyouzhi's cross-tp vocab test in deepspeedai#8309, using the column partition already in tree so it carries no dependency on that PR. Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com>
|
LM-head name detection is duplicated in several coupled places — should all reuse The |
|
|
…equence The compatibility wrapper stays available at deepspeed.sequence.cross_entropy.vocab_sequence_parallel_cross_entropy for existing callers, but new callers should use the general vocab_parallel_cross_entropy(tp_group=..., sp_group=...) API instead. Signed-off-by: iLeGend <824040212@qq.com>
_vocab_metadata_cache was a module-level dict keyed on tp_group, so it held a strong reference to any process group ever passed in and never released it. Remove it: the bare vocab_parallel_cross_entropy function now always re-validates (no cache), and VocabParallelCausalLMLoss caches its own resolved (vocab_start_index, vocab_end_index, global_vocab_size) on self after validating once, since it already owns tp_group for its whole lifetime. Reusing one loss instance for a different shard now raises instead of silently reusing stale metadata. Signed-off-by: iLeGend <824040212@qq.com>
Good catch. I removed the reference to the I think this is safe enough for the multi-model/multi-shard/multi-loss case, since each loss instance owns its own |
The ("lm_head", "embed_out") check was hardcoded separately in
_replace (legacy_lm_head), _create_row_parallel_layer,
_configure_gathered_column_tie_fallbacks, and _replace_last_linear_module,
coupled to _is_lm_head_name without actually reusing it. Route them all
through _is_lm_head_name, which now reads AutoTPConfig.lm_head_patterns
when a partition_config is supplied and falls back to the legacy pair
otherwise, so there is one place that defines what counts as an LM head.
Signed-off-by: iLeGend <824040212@qq.com>
|
@jinyouzhi can you fix CI failure? Thanks! |
OK sure |
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Revert 53c4703 Keep _is_lm_head_name instance-bound so it can honor AutoTPConfig.lm_head_patterns, and update the stale static call in its regression test. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
ef7d23f
…#8538) ## Summary Implements PR-E from deepspeedai#8173: eager training for explicitly row-parallel, untied `lm_head` / `embed_out` output heads. This PR is independent of the vocabulary-parallel CE work in PR-B deepspeedai#8309 and contains no PR-D changes. - Slice replicated activations along the last dimension using the frozen weight-shard geometry, including uneven hidden dimensions. - Reuse the existing row-parallel forward output reduction and replicated bias behavior. - Reconstruct and all-reduce the complete input gradient in backward so preceding replicated layers receive the correct gradient. - Preserve the original parameter identity, existing inference specialization, and default gathered-column training behavior. - Reject tied output weights at TP > 1, reshaped/non-input-dimension partitions, and deferred compiler collectives for this path. ## Validation Executed on jk01 with two NVIDIA L20 GPUs (46068 MiB each), in an isolated container: Python 3.12.3, PyTorch 2.13.0+cu130, CUDA 13.0, Transformers 5.14.1. - Partition/config regression: **17 passed**. - Real two-rank DeepSpeed-engine integration: **three configurations each completed five optimizer updates**, with two accumulated micro-batches per update (SGD LR 0.05, ZeRO stage 0, FP32, clipping explicitly disabled in both paths). - Configurations cover `lm_head` / `embed_out`, bias / no bias, hidden dimensions 32 / 35, and flattened / sequence-shaped inputs. - Compared logits, CE loss, input gradients, replicated projection gradients, local head gradients, and updated weights against an independent unsharded PyTorch model after every update. Also checked the exact SGD update and that weights actually changed. - Changed-file pre-commit checks and `git diff --check` passed. The integration method `TestRowParallelOutputHeadTraining.test_five_optimizer_steps_match_unsharded_reference` was executed through two-rank torchrun on the real devices. It can also be run through pytest: ```sh PYTHONPATH=.:tests python3 -m pytest tests/unit/module_inject/test_tp_partition_config_path.py -q PYTHONPATH=.:tests python3 -m pytest tests/unit/v1/autotp/test_autotp_training.py::TestRowParallelOutputHeadTraining -v -s ``` This validates the minimal eager training contract, not large-model convergence, mixed-precision/ZeRO checkpoint round trips, or DeepCompile. Coupled tied-embedding sharding remains outside this PR's scope. Signed-off-by: 0z5a <dezhen.lu@uni-tuebingen.de> Co-authored-by: 0z5a <dezhen.lu@uni-tuebingen.de> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>
…eedai#8540) ## Dependency and review scope Depends on deepspeedai#8309 (PR-B from deepspeedai#8173), currently open. This branch is based on its head `1b4ba6b162e87b10bedbb7316466bd877e2cb278`; inherited PR-B commits retain their original authorship. The final commit is the optional Liger backend work. Please review that commit for this PR's independent changes. This is a draft until PR-B lands; the master comparison currently includes the dependency changes. This addresses the optional fused CE and memory-profiling direction in deepspeedai#8173. It does not include row-parallel output-head training (deepspeedai#8538) or coupled tied-embedding sharding. ## Changes - Add opt-in `tensor_parallel.vocab_parallel_ce_backend: "liger"` alongside `vocab_parallel_lm_head: true`; default `"torch"` remains dependency-free. - Lazy-load Liger's public vocabulary-parallel CE function (`liger-kernel>=0.8.1`) and retain the existing reduction / causal-label-shifting contract. - Use identical gathered shard metadata to select Liger on every TP rank only for equal vocabulary shards, nonempty CUDA tokens, and FP32/FP16/BF16 logits. Uneven shards or unsupported layouts uniformly fall back to the PyTorch reference. - Reject repeated retained-graph backward and higher-order gradients for Liger's destructive saved-buffer backward instead of silently producing incorrect gradients. - Add optional-backend numerical and real AutoTP training integration coverage, and document the compatibility boundaries. This fuses CE only, **not the linear projection**. Local vocabulary logits are still materialized. ## Real-device validation jk01: two NVIDIA L20 GPUs (46068 MiB each), isolated container, Python 3.12.3, PyTorch 2.13.0+cu130, CUDA 13.0, Transformers 5.14.1, Triton 3.7.1, liger-kernel 0.8.2. - Partition/config and non-distributed CE regressions: **42 passed**, 7 distributed cases deselected in that regression invocation. - Two-rank numerical tests: FP32/FP16/BF16, none/sum/mean, even/uneven vocab, ignored targets, unchanged input logits, retained-graph rejection, and higher-order rejection passed. - Real DeepSpeed-engine training: vocab 34 (Liger) and vocab 35 (uneven fallback), each **five optimizer updates**, with two accumulated micro-batches per update. SGD LR 0.05, FP32, ZeRO 0, clipping explicitly disabled. Logits, loss, gradients, updated sharded head weights/biases, and replicated projection weights/biases agree with an independent unsharded PyTorch model. - Changed-file pre-commit checks and `git diff --check` passed. The distributed integration methods in `TestLigerVocabParallelCE` were executed via two-rank torchrun on the actual devices. Reproduce with pytest: ```sh PYTHONPATH=.:tests python3 -m pytest tests/unit/v1/sequence_parallelism/test_liger_vocab_parallel_cross_entropy.py -v -s ``` ## Memory profile Compared local linear + vocabulary-parallel CE forward/backward + five SGD updates, two micro-batches each, after kernel warm-up and resetting weights/allocated peaks. BF16, TP=2, 1024 tokens per rank, hidden=128, global vocabulary=65536, local vocabulary=32768. Statistics are the maximum across ranks. | Backend | Peak allocated | Incremental peak over baseline | Short-run elapsed | | --- | ---: | ---: | ---: | | PyTorch | 512247296 bytes (488.52 MiB) | 478160384 bytes | 0.0606 s | | Liger | 310922240 bytes (296.52 MiB) | 276834304 bytes | 0.0779 s | Peak allocated decreases approximately 39.3%; incremental peak decreases approximately 42.1%. Losses and final local weights agree. This is a short single-sample memory comparison, **not evidence of throughput speedup**; allocated memory is not total device or reserved memory. Four-GPU TP x SP integration, mixed-precision/ZeRO checkpoint round trips, and large-model convergence are not validated by this two-GPU run. --------- Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
deepspeedai#8680) Follow-up to deepspeedai#8538 (part of deepspeedai#8173). ## Summary Row-parallel output-head training (deepspeedai#8538) rejected an `lm_head`/`embed_out` whose weight is tied to the input embedding, and that rejection could be bypassed silently (see *Bug* below). This PR fixes the bug and supports the tied case by sharding the tied `nn.Embedding` along the same hidden dimension as the row-parallel head, so both share one physical per-rank shard. ## Bug: tie silently broken when the embedding is visited first The tie was detected by scanning `self.module.modules()` *during* the replacement walk. The walk visits `embed_tokens` before `lm_head`, so when `partition_config` also matches the embedding, `_slice_embedding` swaps it for a new module with a new Parameter first. By the time `lm_head` was checked, no other module aliased its weight, the check passed, and the head was sharded while the embedding kept a separate copy — with no error. ```python AutoTPConfig(layer_specs=[ TPLayerSpec(patterns=[r".*embed_tokens\.weight$"], partition_type=PartitionType.ROW), TPLayerSpec(patterns=[r".*lm_head\.weight$"], partition_type=PartitionType.ROW), ]) # before: no error, model.lm_head.weight is model.embed_tokens.weight -> False # after: both modules share one hidden-dimension shard ``` Fixed by snapshotting weight ties from the original, pre-replacement model in `AutoTP.__init__`, following the approach the vocab-parallel path already takes for the same ordering issue (introduced in deepspeedai#8309, now `_tied_vocab_parallel_embedding_ids`). ## Changes - `HiddenParallelEmbedding` (`layers.py`): looks up each token's hidden-dimension slice and all-gathers the slices into the replicated activation (`GatherFromTensorParallelRegion`, so uneven hidden shards work). When tied, it reuses the row-parallel head's already-sharded weight `Parameter` and partition sizes; its universal-checkpoint metadata matches `LinearAllreduce`'s. `padding_idx`, `scale_grad_by_freq`, and Gemma3's scaled embedding are preserved; `max_norm`, `sparse`, and other custom forwards are rejected. - `AutoTP` (`auto_tp.py`): - Snapshots each module's weight-tie partners from the original model, and records embeddings tied to a row-parallel training head so `_slice_embedding` leaves them alone regardless of traversal order. - `_create_row_parallel_layer` validates all tie partners *before* partitioning the shared weight, then replaces every tied embedding alias with a `HiddenParallelEmbedding`. A tie to a non-`nn.Embedding` module still raises; a conflicting explicit embedding spec is superseded with a warning. - Docs: `autotp-training.md` describes the tied path and notes that `vocab_parallel_lm_head` is usually preferable for tied models, since a row-parallel head all-reduces full vocabulary logits. Not changed: untied embeddings that match a spec still go through `_slice_embedding`, whose un-gathered hidden slice is intentional for per-head embeddings such as T5's `relative_attention_bias`. ## Tests - Unit (`test_tp_partition_config_path.py`): tie is shared with/without an embedding spec visited first (the embedding-spec case reproduces the bug above on master); rejection for non-embedding ties and `max_norm` leaves the model untouched; Gemma3 scaled lookup matches the original module. - Integration (`TestTiedRowParallelOutputHeadTraining`, 2 GPUs): three optimizer steps against an unsharded reference comparing logits, the full tied gradient (lookup + projection contributions), and updated weights, covering even/uneven hidden (32/35), `padding_idx`, embedding-spec ordering, and ZeRO-0/2; universal-checkpoint affine map rebuilds the tied weight from uneven shards. Mutation check: detaching the embedding's weight in the lookup fails all four training cases. - `tests/unit/module_inject/` and `tests/unit/v1/autotp/`: 225 passed, 12 skipped on 3x NVIDIA GeForce RTX 5090 D. --------- Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>
Motivation
Changes
This pull request adds support for vocabulary-parallel language modeling (LM) heads in DeepSpeed's tensor parallelism (TP) system. It introduces a new
VocabParallelLinearlayer that keeps logits sharded across TP ranks for untied LM heads and installs a compatible distributed loss automatically. The changes ensure correct partitioning, validation, and integration with the rest of the DeepSpeed engine and configuration system.Key changes include:
Vocabulary-parallel LM head support
vocab_parallel_lm_headconfiguration option inTPTrainingConfigand passed it through the relevant APIs to enable this feature. [1] [2] [3] [4] [5]auto_tp.pyto detect and correctly replace untied LM heads with the newVocabParallelLinearlayer, along with validation to ensure the vocab head is not tied to the embedding. [1] [2] [3] [4]New layer and loss integration
VocabParallelLinearlayer inlayers.py, which ensures logits remain sharded and provides partitioning metadata. [1] [2]Miscellaneous
deepspeed.sequencemodule's public API.These changes collectively enable efficient, distributed training of models with untied, sharded vocabulary projection heads and ensure compatibility with DeepSpeed's tensor parallelism ecosystem.
Limitations