Repository navigation
Conversation
…p_comm Oversized gradients bypass the contiguous IPG buffer, so the reduction reads the parameter's own storage. Non-owner ranks release it right after average_tensor() enqueues the reduction, and the caching allocator can hand the block to the compute stream while the reduction stream still reads it. Record the reduction stream on that storage before average_tensor(). Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: miaTian99 <1194695379@qq.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
pengdurice
approved these changes
Nov 19, 2026
pengdurice
enabled auto-merge
November 19, 2026 15:35
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 19, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
With
overlap_comm=true, ZeRO-1/2 enqueues gradient reduction onreduction_streamand returns without synchronizing. Parameters larger thanreduce_bucket_sizeskip the contiguous IPG buffer and keep their own gradient storage (extra_large_param_to_reduce). Afteraverage_tensor()queues the reduction, ranks that do not own such a parameter release it immediately:Regular parameters are safe here: their
grad.datawas rebound to the persistent IPG bucket buffer, so clearingparam.gradonly drops a reference. Oversized gradients never get that rebinding, so their storage goes back to the caching allocator while the reduction stream may still be reading it. The allocator can then reuse it for a tensor allocated on the compute stream (in our case an activation recomputed by gradient checkpointing), corrupting the gradient. In our 8-GPU run on 0.19.1 the weights turn NaN at step 3; on master the reduced gradient holds values up to 3.4e38 (bf16 max) and the update is skipped.Whether a block is exposed depends on how the gradient reaches the collective. In the attached reproducer, a gradient that lies inside one partition is passed to the collective as is, and its block is still
active_pending_freeafterreduce_ipg_grads()returns (checked withtorch.cuda.memory_snapshot()), so the compute stream cannot get it. A weight cut by a partition boundary whose lower slice is smaller thanreduce_bucket_sizebehaves differently:allreduce_and_scatter()batches the two slices,flatten()copies them on the reduction stream, and nothing records that stream on the source. This is the case that gets corrupted in the reproducer. It needs at least 3 ranks, because with 2 a straddling weight is owned by both and never released.Environment observed on:
gate_projgradient is 67,895,296 elements, 5.29xreduce_bucket_size.overlap_comm=true,contiguous_gradients=true,reduce_scatter=true,reduce_bucket_size=auto(= 12,845,056), no optimizer/parameter offload.b43c383f), where_offload_gradient_safety_enabledisFalsefor this config.Still broken on master: NaN becomes a silently frozen run
On master (
b43c383f, 0.19.10) the race is unchanged. What changed is the symptom: #8035 (bf16check_grad_overflowon by default) and #8638 (non-finite group norm is treated as overflow) detect the corrupted gradient and skip the update, so the weights stay finite but the model stops learning.Same 8-GPU run, same data order, 40 optimizer steps, only
overlap_commdiffers:overlap_comm=falseoverlap_comm=truegrad_norm=inflayers.19.mlp.gate_proj) abs sumThe skip discards the update; it does not repair the gradient.
Why the existing fixes don't cover this
extra_large_param_to_reduce, whose storage is the caller'sparam.grad.record_streams oversized gradients, but only whencpu_offloadis enabled (_offload_gradient_safety_enabled); non-offloaded paths are explicitly unchanged.Fix
Record the reduction stream on the oversized gradient storage before handing it to
average_tensor():extra_large_grad_reduc_for_averageshares storage withparam.grad, so the mark lands on the block thatclear_grad_attribute()later releases. The guards follow the existingrecord_streamcall sites in this file.Cost
One
record_streamper oversized parameter per step: the block stays alive until the reduction stream drains instead of being recycled immediately. Measured on 8 GPUs (bf16, fused AdamW, gradient checkpointing, 12 steps,record_streamapplied to the oversizedparam.grad, same storage as in the diff), against the same run without it: mean step time 3.542 s to 3.565 s (+0.6%), peak allocated 37.35 GB in both, peak reserved 45.0 to 46.6 GB per rank without and 45.3 to 45.6 GB with. NaN at step 3 is gone for 3 runs x 30 steps.Test plan
A standalone reproducer is attached (repro_oversized_grad_race.py, 3 ranks,
torchrun). It runs ZeRO-2 withoverlap_comm=true,contiguous_gradients=true, oversized weights and gradient checkpointing, and shifts the partition boundary so one weight has a lower slice belowreduce_bucket_size(the header prints the slices DeepSpeed produced). To make the race deterministic it delays the reduction stream withtorch.cuda._sleepbefore the collective, and--reuse emulatemakes the compute stream allocate the freed block and overwrite it, standing in for the recomputed activation. With--reuse naturalthe allocator is left alone, and this small model stays clean on unpatched code, so the script shows the missing protection rather than a natural occurrence.