Repository navigation
Conversation
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7122900a44
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
Heads up on a side effect: with only this wrapper change applied to master, a module that builds |
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Thank you for reviewing. Good catch on this issue, and I fixed in c198b20. |
This change is tracked separately in deepspeedai#8959 (fixes deepspeedai#8948). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Summary
Fixes #8948.
Honor an explicit tensor-constructor
dtypeinsidedeepspeed.zero.Init, instead of unconditionally converting floating-point results to the context's dtype.The unconditional conversion can round an explicitly fp32
torch.arangeto bf16 before a model computes RoPE inverse frequencies. Converting the resulting buffer back to fp32 cannot recover the lost precision.Changes
Noneexplicitdtypein both the wrapped torch factory functions and thetorch.Tensor.__new__hook.dtypeis omitted orNone. Device override and parameter-partitioning behavior are unchanged.zero.Initdocstring.torch.Tensorconstruction and the existing implicit bf16 behavior fortorch.Tensorandtorch.zeros.This intentionally changes dtype precedence for explicitly typed constructor calls. Such tensors may use more initialization memory than before when their requested dtype is wider than the context dtype. This is not a change to the engine's subsequent model/buffer conversion policy.
Validation
Validated commit:
7122900a4404f8197fce0eb8908cb5d5554084fe.Environment: CPU backend on a host with 2 Intel Xeon Platinum 8457C CPUs; Python 3.12.3; PyTorch
2.14.0.dev20260804+xpu. No GPU was used; accelerator/driver version is not applicable to these CPU runs.The previously missed
torch.Tensor(4, dtype=torch.float32)path was reproduced on the CPU backend: it returned bf16 before the second constructor-hook fix. The regression now checks that it returns fp32.Compatibility and remaining validation
Sibling-path inspection covered both ZeRO constructor hooks. ZeRO-1/2 optimizer paths do not define these hooks. The independent
deepspeed.OnDeviceconstructor wrappers retain their existing conversion policy and are outside this PR's ZeRO Init scope.Duplicate check
Checked the issue discussion and open PR searches for
8948 in:bodyandzero.Init explicit dtype. No other PR addressing this constructor-dtype fix was found. Related #8843 preserves model-listed fp32 parameters; it does not fix precision lost while explicitly typed tensors are constructed.AI assistance
The implementation, regression test, and this description were prepared with AI assistance.