Skip to content

[AutoSP] Align cross entropy targets after an in-model causal label shift - #8811

Open
jinyouzhi wants to merge 4 commits into
deepspeedai:masterfrom
jinyouzhi:autosp-label-shift
Open

jinyouzhi wants to merge 4 commits into
deepspeedai:masterfrom
jinyouzhi:autosp-label-shift

Conversation

@jinyouzhi

@jinyouzhi jinyouzhi commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #8457. Only the last commit (6f7c57b9a) belongs to this PR; please review that commit. I will rebase once #8457 is merged.

Addresses #8457 (comment) (in-model causal label shift under AutoSP).

Problem

AutoSP shards the labels the same way as input_ids. A causal LM that shifts its labels inside the model (Hugging Face ForCausalLMLoss pads and shifts, labels = F.pad(labels, (0, 1), value=-100)[..., 1:]; Megatron-style code slices logits[..., :-1, :] / labels[..., 1:]) then shifts each shard on its own. Each rank pairs the last token of its shard with the wrong target, and the loss is silently wrong. #8457 documents pre-shifted labels as the workaround; this PR makes the in-model shift itself correct.

Approach

  • Labels are no longer sharded. Any shift the model applies to them runs on the full sequence, so the label subgraph computes exactly what it does without SP.
  • For each F.cross_entropy whose target derives from the labels:
    1. Walk from the CE input back to the sequence-sharded logits. Static sequence slices on the way (e.g. logits[..., :-1, :]) are removed, because each rank must keep its whole shard of logits. Their offsets give the window [start, S + stop) of global positions the CE input covered.
    2. Pad the full target with ignore_index back to the global positions of that window, then hand the CE this rank's shard [r*S/sp, (r+1)*S/sp). This needs no communication because the labels are replicated.
    3. The loss and gradient aggregation from [AutoSP] Aggregate the causal LM loss and gradients over the SP group #8457 is unchanged.
  • A static check makes sure the target lines up position by position with the logits window: the symbolic length of the target's sequence dimension must equal S + stop - start.
  • Unsupported patterns that the pass detects raise instead of silently misaligning (equivalent rewrites such as a torch.cat of shifted logits slices may go undetected; the docs say so):
    • Only casts, contiguous, view/reshape/flatten (after any slices) and static sequence slices are allowed between the logits and the CE.
    • The labels may only flow into CE targets. They cannot be combined with sharded tensors such as input_ids, or returned from the module.
  • Padding uses narrow + full_like + cat instead of F.pad/constant_pad_nd, which specialize the symbolic sequence length during fake-tensor propagation.

Validation (8× Intel Arc Pro B70, torch 2.14.0.dev+xpu)

  • TestAutoSPCausalLoss compares the loss and the parameters after two optimizer steps against single-device training on the full sequence, over 4 ranks (sp=2, dp=2). New cases:

    • slice (logits[..., :-1, :] / labels[..., 1:])
    • pad (HF style)
    • each combined with ZeRO-1, sum/mean, an explicit mesh, and an empty shard
  • test_rejects_labels_it_cannot_align covers a strided logits slice and labels masked by input_ids.

  • test_compile_autosp.py: 45 sequential and 20 non-sequential tests pass. TestAutoSPCompile and TestAutoSPEngineBackward were excluded because they fail on this machine for environmental reasons (dynamo tracing of synchronize(); the XPU C++20 op build, fixed in Use C++20 for XPU op builders on torch 2.12+ #8803), identically on the base branch.

  • Tiny HF LlamaForCausalLM (transformers 5.17), unshifted labels, sp=2, also checked with dynamic=True + attention_mask:

    loss
    single device 4.840665
    [AutoSP] Aggregate the causal LM loss and gradients over the SP group #8457 alone 4.845766 (wrong)
    this PR 4.840665

Notes

  • Pre-shifted labels (shift_labels) remain supported and are still the simplest path.

Contribution checklist

Related work. This PR is stacked on #8457 and addresses its review thread linked above. A search for "AutoSP label shift" found no other PR.

Sibling paths checked. AutoSP exists only as a DeepCompile pass, so there is no eager variant to fix. #8457 rejects ZeRO-2/3, and the label-shift handling here inherits that restriction.

Commands and results. Hardware and software: 4× Intel Arc Pro B70 (XPU, XCCL), Level Zero driver 1.15.39122 (compute runtime 26.27.39122.11), torch 2.14.0.dev20260804+xpu, transformers 5.17.0.

pytest -p no:randomly -m sequential tests/unit/v1/compile/test_compile_autosp.py
pytest -p no:randomly tests/unit/v1/compile/test_compile_autosp.py

Results are listed under Validation above.

Numerics, performance, memory.

  • Numerics: the loss and the parameters after two optimizer steps match single-device training exactly. The HF Llama table above shows the end-to-end loss.
  • Memory: labels are now replicated instead of sharded. That adds [B, S] integer ids per rank, negligible next to the logits.
  • Throughput: the CE target is sliced locally, so no new collectives are added.

Compatibility matrix.

Validated Not validated
ZeRO stage 0, stage 1 stage 2/3 (rejected by #8457); offload
Precision fp32 bf16/fp16 for the label-shift cases (TestAutoSPCompile was excluded, see above)
Parallelism sp=2 with dp=2 (4 ranks); sp=2 with an explicit mesh sp>2 for the label-shift cases
Accelerator XPU CUDA, HPU

AI assistance. This PR was developed with AI assistance (GitHub Copilot CLI). I reviewed every changed line and ran the tests above.

@jinyouzhi
jinyouzhi marked this pull request as ready for review October 13, 2026 02:58
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-18T02:14:24.778239Z 3372a81 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6f7c57b9a6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread deepspeed/__init__.py Resolved
Comment thread deepspeed/compile/passes/sp_compile.py Resolved
Comment thread deepspeed/compile/passes/sp_compile.py Resolved
Comment thread deepspeed/compile/passes/sp_compile.py Resolved
Comment thread deepspeed/compile/passes/sp_compile.py Resolved
@jinyouzhi
jinyouzhi marked this pull request as draft October 17, 2026 14:20
@jinyouzhi
jinyouzhi marked this pull request as ready for review November 2, 2026 15:23

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 496a3dbaeb

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread deepspeed/__init__.py Resolved
Comment thread deepspeed/__init__.py Resolved
Comment thread deepspeed/__init__.py Resolved
Comment on lines +228 to +230
if node not in sharded_nodes:
raise RuntimeError(f"AutoSP could not trace the cross entropy input back to sequence-sharded logits; "
f"stopped at {node.format_node()}")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject sequence reorders before accepting the logits root

When a model applies a sequence-wide transform such as torch.roll to logits before the allowed reshape/cross-entropy chain, this condition accepts the roll node because sharded_nodes contains every transitive consumer of the sharded inputs, not just the raw logits. The logits roll then runs independently inside each local S/P shard, whereas the replicated labels can be rolled globally and only then sharded, so boundary positions produce silently incorrect loss and gradients instead of the documented unsupported-pattern error; continue tracing or explicitly reject sequence-reordering operations.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b686c90: the pass now rejects logits reordered by roll/flip/fliplr/flipud (function or method form) where it accepts the logits root, since each rank would only reorder its own shard. Covered by the roll row of test_rejects_labels_it_cannot_align.

Comment on lines +222 to +224
elif _is_reshape(node) and slices:
# A reshape that runs before a slice would let the slice cut across the sequence dimension.
raise RuntimeError("AutoSP only supports slicing the cross entropy logits before they are reshaped")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject explicit logits reshapes after removed slices

The fresh remaining reshape case is on the logits side rather than the target-side symbolic reshape fixed earlier: for logits[..., :-1, :].reshape(batch * (seq_len - 1), vocab), traversal sees the reshape before it records the underlying slice, so this check permits it. The pass then erases the slice but leaves the explicit S-1 reshape size, and fake propagation or the first forward fails while trying to reshape B*S*V elements to B*(S-1)*V; require inferred reshape dimensions or rewrite the symbolic size when removing the slice.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b686c90: when a slice is removed, a reshape after it whose sizes are derived from the symbolic sequence length is rejected with a hint to use view(-1, vocab_size). Static sizes such as view(-1, logits.size(-1)) stay supported. Covered by the explicit_logits_view row (logits[..., :-1, :].reshape(B * (S - 1), V)).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix! The same rotation can be written in other ways, such as torch.cat((logits[:, 1:], logits[:, :1]), dim=1), which still runs and silently produces an incorrect loss.
Since it’s hard to detect every equivalent pattern, could we revise “AutoSP raises an error for any other pattern” to warn that some unsupported patterns may go undetected?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay thanks, I reword in f275054 as follow:

AutoSP raises for the unsupported patterns it detects, but some equivalent rewrites, such as  torch.cat((logits[:, 1:], logits[:, :1]), dim=1) , may go undetected and silently produce an incorrect loss.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tohtana updated.

@jinyouzhi jinyouzhi closed this Nov 4, 2026
@jinyouzhi jinyouzhi reopened this Nov 4, 2026
@jinyouzhi
jinyouzhi marked this pull request as draft November 4, 2026 15:57
@tohtana tohtana added needs-gpu-test GPU testing is recommended to validate the changes in this PR. and removed needs-gpu-test GPU testing is recommended to validate the changes in this PR. labels Nov 10, 2026
@jinyouzhi
jinyouzhi marked this pull request as ready for review November 18, 2026 02:06

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3372a81487

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread deepspeed/compile/passes/sp_compile.py Resolved
Comment thread deepspeed/compile/passes/sp_compile.py Resolved
Comment thread deepspeed/compile/passes/sp_compile.py Resolved

@tohtana tohtana left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @jinyouzhi,
Could you address comments from Codex bot? I don't see any other issues. Some P2s seem important to me. If you think we could ignore, please let me know.

@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 18, 2026
@jinyouzhi

Copy link
Copy Markdown
Contributor Author

Hi @tohtana, thanks for the review. I addressed all the Codex comments in b686c90 and replied to each inline. I agree the P2s shouldn't be ignored, I also rebased onto master now that #8457 is merged, so the PR is down to two commits. Thanks!

@jinyouzhi
jinyouzhi requested a review from tohtana November 18, 2026 10:57
jinyouzhi and others added 2 commits November 18, 2026 11:37
…odel label shift

Keep the labels whole on every rank so that any shift the model applies to them,
such as the Hugging Face F.pad(labels, (0, 1), value=-100)[..., 1:], runs on the
full sequence. For each cross entropy on the labels, remove static sequence slices
between the sharded logits and the loss (e.g. logits[..., :-1, :]), pad the full
target to the global positions those slices covered, and hand the loss this
rank's shard of the padded target.

Only rank-preserving casts, reshapes after the slices, and static sequence slices
are allowed between the logits and the cross entropy, and the labels may only flow
into cross entropy targets. Any other pattern raises instead of silently pairing
logits with the wrong targets.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Reject AutoSP without DeepCompile, read the boolean ZeRO config form, give the ranks of an SP group the same samples from the generated dataloader, reject reordered or explicitly resized sliced logits, give each cross entropy its own copy of a shared target transform, pad wide shifts without narrowing the target, and keep FP64 loss sums in FP64.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 19, 2026
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants