Skip to content

ZeRO-1/2: record reduction stream on oversized gradients under overlap_comm - #9031

Open
miaTian99 wants to merge 2 commits into
deepspeedai:masterfrom
miaTian99:master
Open

miaTian99 wants to merge 2 commits into
deepspeedai:masterfrom
miaTian99:master

Conversation

@miaTian99

Copy link
Copy Markdown

Problem

With overlap_comm=true, ZeRO-1/2 enqueues gradient reduction on reduction_stream and returns without synchronizing. Parameters larger than reduce_bucket_size skip the contiguous IPG buffer and keep their own gradient storage (extra_large_param_to_reduce). After average_tensor() queues the reduction, ranks that do not own such a parameter release it immediately:

if not self.is_param_in_current_partition[param_id]:
    if self.overlap_comm and self.contiguous_gradients is False:
        self.previous_reduced_grads[comm_dtype].append(param)
    else:
        self.clear_grad_attribute(param)   # param.grad = None

Regular parameters are safe here: their grad.data was rebound to the persistent IPG bucket buffer, so clearing param.grad only drops a reference. Oversized gradients never get that rebinding, so their storage goes back to the caching allocator while the reduction stream may still be reading it. The allocator can then reuse it for a tensor allocated on the compute stream (in our case an activation recomputed by gradient checkpointing), corrupting the gradient. In our 8-GPU run on 0.19.1 the weights turn NaN at step 3; on master the reduced gradient holds values up to 3.4e38 (bf16 max) and the update is skipped.

Whether a block is exposed depends on how the gradient reaches the collective. In the attached reproducer, a gradient that lies inside one partition is passed to the collective as is, and its block is still active_pending_free after reduce_ipg_grads() returns (checked with torch.cuda.memory_snapshot()), so the compute stream cannot get it. A weight cut by a partition boundary whose lower slice is smaller than reduce_bucket_size behaves differently: allreduce_and_scatter() batches the two slices, flatten() copies them on the reduction stream, and nothing records that stream on the source. This is the case that gets corrupted in the reproducer. It needs at least 3 ranks, because with 2 a straddling weight is owned by both and never released.

Environment observed on:

  • Model: Qwen2.5-7B-class backbone (hidden_size 3584, intermediate_size 18944, 28 layers), the affected gate_proj gradient is 67,895,296 elements, 5.29x reduce_bucket_size.
  • Hardware: 8 GPUs, PCIe attached, no NVLink, bf16.
  • DeepSpeed 0.19.1, PyTorch 2.13 nightly, NCCL 2.30.4, fused AdamW, gradient checkpointing on, grad clip 1.0.
  • ZeRO-2 with overlap_comm=true, contiguous_gradients=true, reduce_scatter=true, reduce_bucket_size=auto (= 12,845,056), no optimizer/parameter offload.
  • The code path is unchanged on master (b43c383f), where _offload_gradient_safety_enabled is False for this config.

Still broken on master: NaN becomes a silently frozen run

On master (b43c383f, 0.19.10) the race is unchanged. What changed is the symptom: #8035 (bf16 check_grad_overflow on by default) and #8638 (non-finite group norm is treated as overflow) detect the corrupted gradient and skip the update, so the weights stay finite but the model stops learning.

Same 8-GPU run, same data order, 40 optimizer steps, only overlap_comm differs:

overlap_comm=false overlap_comm=true
skipped updates 0 / 40 39 / 40 (every step after the first)
reduced gradients finite, max abs 0.10 to 0.38 380 to 3.0k non-finite per step after step 1, max finite 3.4e38, grad_norm=inf
probe weight (layers.19.mlp.gate_proj) abs sum changes on 35 of 39 steps identical on all 39 later steps
mean loss, first 5 / last 5 steps 1.022 / 0.765 1.023 / 1.003

The skip discards the update; it does not repair the gradient.

Why the existing fixes don't cover this

Fix

Record the reduction stream on the oversized gradient storage before handing it to average_tensor():

@@ -2247,6 +2247,11 @@
                     extra_large_grad_reduc.data = extra_large_grad_reduc_for_average.data.view_as(extra_large_grad_reduc) if (not self.zenflow or self.extra_large_param_to_reduce[comm_dtype].dim() == 1) \
                         else extra_large_grad_reduc_for_average.data.view_as(extra_large_grad_reduc.transpose(0, 1))
 
+                    # Oversized gradients bypass the contiguous IPG buffer, so the
+                    # reduction reads the parameter's own storage, which non-owner
+                    # ranks release right after the reduction is enqueued.
+                    if self.overlap_comm and not get_accelerator().resolves_data_dependency():
+                        extra_large_grad_reduc_for_average.record_stream(self.reduction_stream)
                     self.average_tensor(extra_large_grad_reduc_for_average, comm_dtype)
                     del self.extra_large_param_to_reduce[comm_dtype]
                 else:

extra_large_grad_reduc_for_average shares storage with param.grad, so the mark lands on the block that clear_grad_attribute() later releases. The guards follow the existing record_stream call sites in this file.

Cost

One record_stream per oversized parameter per step: the block stays alive until the reduction stream drains instead of being recycled immediately. Measured on 8 GPUs (bf16, fused AdamW, gradient checkpointing, 12 steps, record_stream applied to the oversized param.grad, same storage as in the diff), against the same run without it: mean step time 3.542 s to 3.565 s (+0.6%), peak allocated 37.35 GB in both, peak reserved 45.0 to 46.6 GB per rank without and 45.3 to 45.6 GB with. NaN at step 3 is gone for 3 runs x 30 steps.

Test plan

A standalone reproducer is attached (repro_oversized_grad_race.py, 3 ranks, torchrun). It runs ZeRO-2 with overlap_comm=true, contiguous_gradients=true, oversized weights and gradient checkpointing, and shifts the partition boundary so one weight has a lower slice below reduce_bucket_size (the header prints the slices DeepSpeed produced). To make the race deterministic it delays the reduction stream with torch.cuda._sleep before the collective, and --reuse emulate makes the compute stream allocate the freed block and overwrite it, standing in for the recomputed activation. With --reuse natural the allocator is left alone, and this small model stays clean on unpatched code, so the script shows the missing protection rather than a natural occurrence.

DeepSpeed 0.19.1           : [result] WEIGHTS_NAN, step 1 (reused=2, grad_max_abs 3.3e29)
DeepSpeed 0.19.1 + this fix: [result] CLEAN, 6 steps (reused=0)
master b43c383f (0.19.10)  : [result] GRADIENTS_CORRUPTED, 6/6 updates skipped (grad_norm=inf, loss flat)

…p_comm

Oversized gradients bypass the contiguous IPG buffer, so the reduction
reads the parameter's own storage. Non-owner ranks release it right after
average_tensor() enqueues the reduction, and the caching allocator can hand
the block to the compute stream while the reduction stream still reads it.
Record the reduction stream on that storage before average_tensor().

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: miaTian99 <1194695379@qq.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-07T13:08:29.741121Z 32cc07b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@tohtana tohtana added needs-gpu-test GPU testing is recommended to validate the changes in this PR. and removed needs-gpu-test GPU testing is recommended to validate the changes in this PR. labels Nov 10, 2026
@pengdurice
pengdurice enabled auto-merge November 19, 2026 15:35
@pengdurice
pengdurice added this pull request to the merge queue Nov 19, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants