Skip to content

ZeRO-3 NVMe offload: give swapped-out parameter partitions the model dtype - #9099

Merged
xylian86 merged 2 commits into
deepspeedai:masterfrom
xylian86:xinyu/zero3-nvme-placeholder-dtype
Nov 17, 2026
Merged

xylian86 merged 2 commits into
deepspeedai:masterfrom
xylian86:xinyu/zero3-nvme-placeholder-dtype

Conversation

@xylian86

@xylian86 xylian86 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #9095. Since #8843, ZeRO-3 with NVMe parameter offload fails in deepspeed.initialize for bf16 and fp32 models:

NotImplementedError: ZeRO-3 parameter offload keeps parameter partitions in one dtype; found [torch.bfloat16, torch.float16]

A partition swapped out to NVMe keeps a 1-element placeholder as ds_tensor, and the swapper made that placeholder fp16 for every model dtype. #8843 reads ds_tensor.dtype at init. The partitions on NVMe then report fp16 while the small ones kept in CPU memory report bf16, so they look like two dtypes. Now the placeholder takes the swapper's dtype, which every swap buffer already has. fp16 models are unchanged. The PR also adds an fp32 variant of test_param_and_optimizer, since the existing tests use bf16 only.

…dtype

A parameter partition swapped out to NVMe keeps a 1-element placeholder
as ds_tensor. The swapper made it fp16 whatever the model dtype was.
deepspeedai#8843 reads ds_tensor.dtype at ZeRO-3 init, and with parameter offload
it raises when the partitions hold more than one dtype. With a bf16 or
fp32 model, the partitions on NVMe report fp16, while the partitions too
small to swap stay in CPU memory and report bf16 or fp32. So
deepspeed.initialize raises NotImplementedError: ZeRO-3 parameter
offload keeps parameter partitions in one dtype; found [torch.bfloat16,
torch.float16] ([torch.float16, torch.float32] for fp32). fp16 models
are not affected.

The placeholder now takes the swapper's dtype, which every swap buffer
already has. The test file trained in bf16 only, so add an fp32 variant
of test_param_and_optimizer.

Fixes deepspeedai#9095.

Signed-off-by: Xinyu Lian <lian7@illinois.edu>
@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 17, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T08:18:27.367131Z 34f2d14 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 17, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 34f2d14959

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 17, 2026
@delock
delock enabled auto-merge November 17, 2026 08:49
@delock
delock disabled auto-merge November 17, 2026 08:51
@delock

delock commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Hi @xylian86 thanks for the fix! Can you take a look at codex comments? Thanks!

The fp32 test does not need bf16, so a device without bf16 should still run it.

Signed-off-by: Xinyu Lian <lian7@illinois.edu>
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 17, 2026
@xylian86

Copy link
Copy Markdown
Collaborator Author

@delock For sure, I have solved it.

@xylian86
xylian86 enabled auto-merge November 17, 2026 09:14
@xylian86
xylian86 added this pull request to the merge queue Nov 17, 2026
Merged via the queue into deepspeedai:master with commit 54c4031 Nov 17, 2026
19 checks passed
@xylian86
xylian86 deleted the xinyu/zero3-nvme-placeholder-dtype branch November 17, 2026 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nightly regression: modal-torch-latest full suite failed

2 participants