Repository navigation
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6f7c57b9a6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
6f7c57b to
b166cd8
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 496a3dbaeb
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if node not in sharded_nodes: | ||
| raise RuntimeError(f"AutoSP could not trace the cross entropy input back to sequence-sharded logits; " | ||
| f"stopped at {node.format_node()}") |
There was a problem hiding this comment.
Reject sequence reorders before accepting the logits root
When a model applies a sequence-wide transform such as torch.roll to logits before the allowed reshape/cross-entropy chain, this condition accepts the roll node because sharded_nodes contains every transitive consumer of the sharded inputs, not just the raw logits. The logits roll then runs independently inside each local S/P shard, whereas the replicated labels can be rolled globally and only then sharded, so boundary positions produce silently incorrect loss and gradients instead of the documented unsupported-pattern error; continue tracing or explicitly reject sequence-reordering operations.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in b686c90: the pass now rejects logits reordered by roll/flip/fliplr/flipud (function or method form) where it accepts the logits root, since each rank would only reorder its own shard. Covered by the roll row of test_rejects_labels_it_cannot_align.
| elif _is_reshape(node) and slices: | ||
| # A reshape that runs before a slice would let the slice cut across the sequence dimension. | ||
| raise RuntimeError("AutoSP only supports slicing the cross entropy logits before they are reshaped") |
There was a problem hiding this comment.
Reject explicit logits reshapes after removed slices
The fresh remaining reshape case is on the logits side rather than the target-side symbolic reshape fixed earlier: for logits[..., :-1, :].reshape(batch * (seq_len - 1), vocab), traversal sees the reshape before it records the underlying slice, so this check permits it. The pass then erases the slice but leaves the explicit S-1 reshape size, and fake propagation or the first forward fails while trying to reshape B*S*V elements to B*(S-1)*V; require inferred reshape dimensions or rewrite the symbolic size when removing the slice.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in b686c90: when a slice is removed, a reshape after it whose sizes are derived from the symbolic sequence length is rejected with a hint to use view(-1, vocab_size). Static sizes such as view(-1, logits.size(-1)) stay supported. Covered by the explicit_logits_view row (logits[..., :-1, :].reshape(B * (S - 1), V)).
There was a problem hiding this comment.
Thanks for the fix! The same rotation can be written in other ways, such as torch.cat((logits[:, 1:], logits[:, :1]), dim=1), which still runs and silently produces an incorrect loss.
Since it’s hard to detect every equivalent pattern, could we revise “AutoSP raises an error for any other pattern” to warn that some unsupported patterns may go undetected?
There was a problem hiding this comment.
Okay thanks, I reword in f275054 as follow:
AutoSP raises for the unsupported patterns it detects, but some equivalent rewrites, such as torch.cat((logits[:, 1:], logits[:, :1]), dim=1) , may go undetected and silently produce an incorrect loss.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3372a81487
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
tohtana
left a comment
There was a problem hiding this comment.
Hi @jinyouzhi,
Could you address comments from Codex bot? I don't see any other issues. Some P2s seem important to me. If you think we could ignore, please let me know.
3372a81 to
b686c90
Compare
…odel label shift Keep the labels whole on every rank so that any shift the model applies to them, such as the Hugging Face F.pad(labels, (0, 1), value=-100)[..., 1:], runs on the full sequence. For each cross entropy on the labels, remove static sequence slices between the sharded logits and the loss (e.g. logits[..., :-1, :]), pad the full target to the global positions those slices covered, and hand the loss this rank's shard of the padded target. Only rank-preserving casts, reshapes after the slices, and static sequence slices are allowed between the logits and the cross entropy, and the labels may only flow into cross entropy targets. Any other pattern raises instead of silently pairing logits with the wrong targets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Reject AutoSP without DeepCompile, read the boolean ZeRO config form, give the ranks of an SP group the same samples from the generated dataloader, reject reordered or explicitly resized sliced logits, give each cross entropy its own copy of a shared target transform, pad wide shifts without narrowing the target, and keep FP64 loss sums in FP64. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Addresses #8457 (comment) (in-model causal label shift under AutoSP).
Problem
AutoSP shards the labels the same way as
input_ids. A causal LM that shifts its labels inside the model (Hugging FaceForCausalLMLosspads and shifts,labels = F.pad(labels, (0, 1), value=-100)[..., 1:]; Megatron-style code sliceslogits[..., :-1, :]/labels[..., 1:]) then shifts each shard on its own. Each rank pairs the last token of its shard with the wrong target, and the loss is silently wrong. #8457 documents pre-shifted labels as the workaround; this PR makes the in-model shift itself correct.Approach
F.cross_entropywhose target derives from the labels:logits[..., :-1, :]) are removed, because each rank must keep its whole shard of logits. Their offsets give the window[start, S + stop)of global positions the CE input covered.ignore_indexback to the global positions of that window, then hand the CE this rank's shard[r*S/sp, (r+1)*S/sp). This needs no communication because the labels are replicated.S + stop - start.torch.catof shifted logits slices may go undetected; the docs say so):contiguous,view/reshape/flatten(after any slices) and static sequence slices are allowed between the logits and the CE.input_ids, or returned from the module.narrow+full_like+catinstead ofF.pad/constant_pad_nd, which specialize the symbolic sequence length during fake-tensor propagation.Validation (8× Intel Arc Pro B70, torch 2.14.0.dev+xpu)
TestAutoSPCausalLosscompares the loss and the parameters after two optimizer steps against single-device training on the full sequence, over 4 ranks (sp=2, dp=2). New cases:slice(logits[..., :-1, :]/labels[..., 1:])pad(HF style)sum/mean, an explicit mesh, and an empty shardtest_rejects_labels_it_cannot_aligncovers a strided logits slice and labels masked byinput_ids.test_compile_autosp.py: 45 sequential and 20 non-sequential tests pass.TestAutoSPCompileandTestAutoSPEngineBackwardwere excluded because they fail on this machine for environmental reasons (dynamo tracing ofsynchronize(); the XPU C++20 op build, fixed in Use C++20 for XPU op builders on torch 2.12+ #8803), identically on the base branch.Tiny HF
LlamaForCausalLM(transformers 5.17), unshiftedlabels, sp=2, also checked withdynamic=True+attention_mask:Notes
shift_labels) remain supported and are still the simplest path.Contribution checklist
Related work. This PR is stacked on #8457 and addresses its review thread linked above. A search for "AutoSP label shift" found no other PR.
Sibling paths checked. AutoSP exists only as a DeepCompile pass, so there is no eager variant to fix. #8457 rejects ZeRO-2/3, and the label-shift handling here inherits that restriction.
Commands and results. Hardware and software: 4× Intel Arc Pro B70 (XPU, XCCL), Level Zero driver 1.15.39122 (compute runtime 26.27.39122.11), torch 2.14.0.dev20260804+xpu, transformers 5.17.0.
Results are listed under Validation above.
Numerics, performance, memory.
[B, S]integer ids per rank, negligible next to the logits.Compatibility matrix.
TestAutoSPCompilewas excluded, see above)AI assistance. This PR was developed with AI assistance (GitHub Copilot CLI). I reviewed every changed line and ran the tests above.