Skip to content

Honor explicit dtype in ZeRO tensor constructors - #8959

Open
jinyouzhi wants to merge 5 commits into
deepspeedai:masterfrom
jinyouzhi:fix8948
Open

jinyouzhi wants to merge 5 commits into
deepspeedai:masterfrom
jinyouzhi:fix8948

Conversation

@jinyouzhi

@jinyouzhi jinyouzhi commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #8948.

Honor an explicit tensor-constructor dtype inside deepspeed.zero.Init, instead of unconditionally converting floating-point results to the context's dtype.

The unconditional conversion can round an explicitly fp32 torch.arange to bf16 before a model computes RoPE inverse frequencies. Converting the resulting buffer back to fp32 cannot recover the lost precision.

Changes

  • Preserve a non-None explicit dtype in both the wrapped torch factory functions and the torch.Tensor.__new__ hook.
  • Keep the existing context-dtype conversion when dtype is omitted or None. Device override and parameter-partitioning behavior are unchanged.
  • Document constructor dtype precedence in the zero.Init docstring.
  • Add a regression test comparing RoPE-style frequencies against a plain PyTorch fp32 reference. It also checks explicit fp32 torch.Tensor construction and the existing implicit bf16 behavior for torch.Tensor and torch.zeros.

This intentionally changes dtype precedence for explicitly typed constructor calls. Such tensors may use more initialization memory than before when their requested dtype is wider than the context dtype. This is not a change to the engine's subsequent model/buffer conversion policy.

Validation

Validated commit: 7122900a4404f8197fce0eb8908cb5d5554084fe.

Environment: CPU backend on a host with 2 Intel Xeon Platinum 8457C CPUs; Python 3.12.3; PyTorch 2.14.0.dev20260804+xpu. No GPU was used; accelerator/driver version is not applicable to these CPU runs.

DS_ACCELERATOR=cpu pytest -q tests/unit/runtime/zero/test_zero_context.py
# 12 passed

pre-commit run --files \
  deepspeed/runtime/zero/partition_parameters.py \
  tests/unit/runtime/zero/test_zero_context.py
# All applicable hooks passed

The previously missed torch.Tensor(4, dtype=torch.float32) path was reproduced on the CPU backend: it returned bf16 before the second constructor-hook fix. The regression now checks that it returns fp32.

Compatibility and remaining validation

Area Coverage
ZeRO Init, bf16 context, explicit fp32 and implicit constructor dtypes Regression passed on CPU
Existing ZeRO context/sharding/gather tests Entire test file passed on CPU, including its single- and two-rank cases
fp16/fp32 context targets for the new regression Not separately exercised
GPU backends, including NVIDIA L20 Not run
CPU/NVMe offload and tensor/pipeline/sequence/expert parallel combinations Not specifically validated
Full Transformers RoPE reproduction and end-to-end training convergence Not run
Throughput and peak memory Not measured; no performance or convergence claim

Sibling-path inspection covered both ZeRO constructor hooks. ZeRO-1/2 optimizer paths do not define these hooks. The independent deepspeed.OnDevice constructor wrappers retain their existing conversion policy and are outside this PR's ZeRO Init scope.

Duplicate check

Checked the issue discussion and open PR searches for 8948 in:body and zero.Init explicit dtype. No other PR addressing this constructor-dtype fix was found. Related #8843 preserves model-listed fp32 parameters; it does not fix precision lost while explicitly typed tensors are constructed.

AI assistance

The implementation, regression test, and this description were prepared with AI assistance.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
@jinyouzhi
jinyouzhi marked this pull request as ready for review October 30, 2026 15:33
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-30T15:37:54.718649Z 7122900 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7122900a44

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@vineethsaivs

Copy link
Copy Markdown
Contributor

Heads up on a side effect: with only this wrapper change applied to master, a module that builds nn.Parameter(torch.ones(8, dtype=torch.float32)) under Init(dtype=torch.bfloat16) keeps an fp32 parameter, and deepspeed.initialize for ZeRO-3 then fails an assert in _create_fp16_partitions_with_defragmentation (stage3.py:972). Master trains the same model fine. Casting floating-point params back to the Init dtype at the end of _post_init_method avoids it, and I can share the CPU repro.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
@jinyouzhi

Copy link
Copy Markdown
Contributor Author

Heads up on a side effect: with only this wrapper change applied to master, a module that builds nn.Parameter(torch.ones(8, dtype=torch.float32)) under Init(dtype=torch.bfloat16) keeps an fp32 parameter, and deepspeed.initialize for ZeRO-3 then fails an assert in _create_fp16_partitions_with_defragmentation (stage3.py:972). Master trains the same model fine. Casting floating-point params back to the Init dtype at the end of _post_init_method avoids it, and I can share the CPU repro.

Thank you for reviewing. Good catch on this issue, and I fixed in c198b20.

jinyouzhi added a commit to jinyouzhi/DeepSpeed that referenced this pull request Nov 4, 2026
This change is tracked separately in deepspeedai#8959 (fixes deepspeedai#8948).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 14, 2026
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 17, 2026
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. will-review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] zero.Init casts tensors created with an explicit dtype=torch.float (RoPE inv_freq computed in bf16)

3 participants