Repository navigation
ZeRO-3 NVMe offload: give swapped-out parameter partitions the model dtype - #9099
Conversation
…dtype A parameter partition swapped out to NVMe keeps a 1-element placeholder as ds_tensor. The swapper made it fp16 whatever the model dtype was. deepspeedai#8843 reads ds_tensor.dtype at ZeRO-3 init, and with parameter offload it raises when the partitions hold more than one dtype. With a bf16 or fp32 model, the partitions on NVMe report fp16, while the partitions too small to swap stay in CPU memory and report bf16 or fp32. So deepspeed.initialize raises NotImplementedError: ZeRO-3 parameter offload keeps parameter partitions in one dtype; found [torch.bfloat16, torch.float16] ([torch.float16, torch.float32] for fp32). fp16 models are not affected. The placeholder now takes the swapper's dtype, which every swap buffer already has. The test file trained in bf16 only, so add an fp32 variant of test_param_and_optimizer. Fixes deepspeedai#9095. Signed-off-by: Xinyu Lian <lian7@illinois.edu>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 34f2d14959
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
Hi @xylian86 thanks for the fix! Can you take a look at codex comments? Thanks! |
The fp32 test does not need bf16, so a device without bf16 should still run it. Signed-off-by: Xinyu Lian <lian7@illinois.edu>
|
@delock For sure, I have solved it. |
Fixes #9095. Since #8843, ZeRO-3 with NVMe parameter offload fails in
deepspeed.initializefor bf16 and fp32 models:A partition swapped out to NVMe keeps a 1-element placeholder as
ds_tensor, and the swapper made that placeholder fp16 for every model dtype. #8843 readsds_tensor.dtypeat init. The partitions on NVMe then report fp16 while the small ones kept in CPU memory report bf16, so they look like two dtypes. Now the placeholder takes the swapper's dtype, which every swap buffer already has. fp16 models are unchanged. The PR also adds an fp32 variant oftest_param_and_optimizer, since the existing tests use bf16 only.