Repository navigation
Allow bf16 Adam moments with fp32 master weights - #9106
Draft
pengdurice wants to merge 1 commit into
Draft
pengdurice wants to merge 1 commit into
pengdurice wants to merge 1 commit into
Conversation
…es without bf16 master weights) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: pengdurice <pengduhit@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of: #8672 (Q4 2026 roadmap), "Enable training of large scale models".
Summary
Today
bf16.bf16_optimizer_statesrequiresbf16.bf16_master_weights_and_grads: DeepSpeed keeps Adam's moments inbf16 only when the master weights are bf16 too. Users then choose between two settings, and both fail for large MoE
fine-tuning:
weight's magnitude rounds away. At a fine-tuning learning rate many updates are that small. Fine-tuning Qwen3-30B-A3B at lr 1e-5 (table below), the loss after
14 steps is 2.2630 with bf16 master weights against 2.2336 with fp32 master weights.
8 for the moments), on top of the bf16 weight and gradient. With expert parallelism equal to the number of GPUs, the
experts' state cannot be sharded across data-parallel ranks; Qwen3-30B-A3B on 8 GPUs with expert parallel 8 holds
3.62 B expert parameters per GPU, and this setting runs out of memory.
The other frameworks offer the setting in between, fp32 master weights with bf16 moments: Megatron-LM's precision-aware
optimizer (
--exp-avg-dtype bf16 --exp-avg-sq-dtype bf16with fp32 main parameters), NeMo AutoModel throughTransformer Engine's
FusedAdam(master_weights=True, exp_avg_dtype=torch.bfloat16, ...), and TorchTitan's Adam statedtype. This PR adds it to DeepSpeed:
Without
bf16_master_weights_and_grads, this now keeps fp32 master weights and stores FusedAdam's two moments in bf16:8 bytes of optimizer state per parameter instead of 12. The update itself is computed in fp32 from the stored moments,
which are rounded to bf16 when written back. With
bf16_master_weights_and_grads, nothing changes.Results
Qwen3-30B-A3B (48 layers, 128 experts, top 8), the released checkpoint with its trained router, PG-19 text,
1 node of 8 x H100 80GB, expert parallel 8, ZeRO-2, micro-batch 4 x 4,096 tokens, 16 accumulation steps, learning
rate 1e-5 constant, full activation recompute, DeepEP v2 on 64 SMs,
row_weighting_impl: "fused". 14 steps, step time= mean of steps 5-14, all runs in one job on one node:
bf16_master_weights_and_grads+bf16_optimizer_states)bf16_optimizer_statesonly)The two runs that fit use the same data and start from the same step-1 loss (2.476786); they separate from step 3 on.
Keeping fp32 master weights costs 7.1 GiB here (about 2 bytes per local parameter) and 0.1% of step time, and it
keeps the updates that bf16 master weights lose. The global loss is the mean over all 8 GPUs.
Changes
csrc/adam/multi_tensor_adam.cu:AdamFunctortakes the moments' type separately from the parameters' andgradients' type, through a second dispatch level. All arithmetic stays fp32 as before. A check rejects a call whose
two moment lists have different dtypes.
FusedAdam(exp_avg_dtype=...): allocates both moments in that dtype. It raises on accelerators other than CUDA/ROCm,whose FusedAdam builders assume one dtype for all four tensor lists.
load_state_dictrestores the moments' dtypeafter loading, because
torch.optim.Optimizer.load_state_dictcasts every floating-point state tensor to itsparameter's dtype and would bring them back as fp32 after a checkpoint reload.
bf16_optimizer_statesand fp32 master weights it builds FusedAdam withexp_avg_dtype=torch.bfloat16,and raises a
ValueErrorfor any other optimizer (torch_adam, Lamb, Muon, client optimizers). Those keep theirstates in the master weights' fp32, so the setting would otherwise save nothing without saying so.
offload_optimizerit isrejected (
DeepSpeedCPUAdamkeeps its moments in the parameters' dtype); offloading bf16 states still needsbf16_master_weights_and_grads, as today.config-json.md(bf16_optimizer_statesentry and a support matrix for fp32 master weights) and theprecision_config.pydocstring.Tests
test_fused_adam_low_precision_moments_with_fp32_params(tests/unit/ops/adam/test_adamw.py, bf16 and fp16moments): FusedAdam against the file's existing fp32 reference step, with the reference moments stored in the low
precision. Checks the moments' dtype, the parameters within 2% of the total update, and the moments within one
spacing of the moment dtype at their largest magnitude (the kernel's fused multiply-adds can round a value on a
rounding midpoint the other way).
TestFP32MasterWeightsWithBF16OptimizerStates(tests/unit/v1/half_precision/test_bf16.py, 2 GPUs, ZeRO 1, 2 and3): master weights stay fp32, moments are bf16, losses over 5 steps within 1% of the fp32-moment run; a checkpoint
reload keeps the moments in bf16, bit for bit.
TestFP32MasterWeightsWithBF16OptimizerStatesRejected: optimizer offload andtorch_adamare rejected.44661723e.2^31 + 1,000 elements, which takes the kernel's 64-bit indexing path: passes, with the parameters within 4e-6 of the total update and the moments within 1.5e-3 of their largest value at both ends of the tensor.
Hardware: 1 node, 8 x NVIDIA H100 80GB HBM3, driver 595.112.07; NVIDIA PyTorch container, torch 2.14.0a0 (nv26.08),
CUDA 13.4, transformers 5.18.0, pytest 9.0.3. FusedAdam built from this tree's
.cufile.test_bf16.py ran from the repository root. A tree that is not installed reads its version from
version.txtin thecurrent directory; from
tests/it reports 0.0.0, and the ZeRO-1 checkpoint loader refuses a checkpoint with thatversion, so the ZeRO-1 checkpoint-reload case fails there for that reason only.
pre-commit run --filespasses on the nine changed files (clang-format 18.1.3, as inrequirements-dev.txt).Compatibility
bf16_master_weights_and_gradswith or withoutbf16_optimizer_states, with or without offloadTestBF16MasterWeightsGradientscases passRelated work
multi_tensor_adam.cufor the non-ZeROFP16_Optimizer(fp16/bf16 gradients and output parameters, fp32 master weights and fp32 moments in one launch). It does not offer
low-precision moments, so the two changes do not duplicate each other; both add code to the same
.cufile, so thesecond to merge may need a small rebase.
bf16_optimizer_states,exp_avg_dtypeand "bf16 optimizer states").AI assistance
This change, its tests and this description were prepared with AI assistance (Claude).
🤖 Generated with Claude Code