Skip to content

Allow bf16 Adam moments with fp32 master weights - #9106

Draft
pengdurice wants to merge 1 commit into
deepspeedai:masterfrom
pengdurice:peng-add-fused-adam-bf16-moments-fp32-master-v1
Draft

pengdurice wants to merge 1 commit into
deepspeedai:masterfrom
pengdurice:peng-add-fused-adam-bf16-moments-fp32-master-v1

Conversation

@pengdurice

Copy link
Copy Markdown
Contributor

Part of: #8672 (Q4 2026 roadmap), "Enable training of large scale models".

Summary

Today bf16.bf16_optimizer_states requires bf16.bf16_master_weights_and_grads: DeepSpeed keeps Adam's moments in
bf16 only when the master weights are bf16 too. Users then choose between two settings, and both fail for large MoE
fine-tuning:

  • bf16 master weights lose small updates. bf16 has 8 bits of mantissa, so an update smaller than about 1/256 of a
    weight's magnitude rounds away. At a fine-tuning learning rate many updates are that small. Fine-tuning Qwen3-30B-A3B at lr 1e-5 (table below), the loss after
    14 steps is 2.2630 with bf16 master weights against 2.2336 with fp32 master weights.
  • fp32 master weights with fp32 moments need 12 bytes of optimizer state per parameter (4 for the master weight,
    8 for the moments), on top of the bf16 weight and gradient. With expert parallelism equal to the number of GPUs, the
    experts' state cannot be sharded across data-parallel ranks; Qwen3-30B-A3B on 8 GPUs with expert parallel 8 holds
    3.62 B expert parameters per GPU, and this setting runs out of memory.

The other frameworks offer the setting in between, fp32 master weights with bf16 moments: Megatron-LM's precision-aware
optimizer (--exp-avg-dtype bf16 --exp-avg-sq-dtype bf16 with fp32 main parameters), NeMo AutoModel through
Transformer Engine's FusedAdam(master_weights=True, exp_avg_dtype=torch.bfloat16, ...), and TorchTitan's Adam state
dtype. This PR adds it to DeepSpeed:

"bf16": {
    "enabled": true,
    "bf16_optimizer_states": true
}

Without bf16_master_weights_and_grads, this now keeps fp32 master weights and stores FusedAdam's two moments in bf16:
8 bytes of optimizer state per parameter instead of 12. The update itself is computed in fp32 from the stored moments,
which are rounded to bf16 when written back. With bf16_master_weights_and_grads, nothing changes.

Results

Qwen3-30B-A3B (48 layers, 128 experts, top 8), the released checkpoint with its trained router, PG-19 text,
1 node of 8 x H100 80GB, expert parallel 8, ZeRO-2, micro-batch 4 x 4,096 tokens, 16 accumulation steps, learning
rate 1e-5 constant, full activation recompute, DeepEP v2 on 64 SMs, row_weighting_impl: "fused". 14 steps, step time
= mean of steps 5-14, all runs in one job on one node:

Master weights / Adam moments Code Step time Peak allocated in step 14 (highest GPU) Global loss at step 14
fp32 / fp32 master out of memory in step 2, once the fp32 moments exist - -
bf16 / bf16 (bf16_master_weights_and_grads + bf16_optimizer_states) master 32.800 s 62.05 GiB 2.263046
fp32 / bf16 (bf16_optimizer_states only) this PR 32.843 s 69.15 GiB 2.233639

The two runs that fit use the same data and start from the same step-1 loss (2.476786); they separate from step 3 on.
Keeping fp32 master weights costs 7.1 GiB here (about 2 bytes per local parameter) and 0.1% of step time, and it
keeps the updates that bf16 master weights lose. The global loss is the mean over all 8 GPUs.

Changes

  • csrc/adam/multi_tensor_adam.cu: AdamFunctor takes the moments' type separately from the parameters' and
    gradients' type, through a second dispatch level. All arithmetic stays fp32 as before. A check rejects a call whose
    two moment lists have different dtypes.
  • FusedAdam(exp_avg_dtype=...): allocates both moments in that dtype. It raises on accelerators other than CUDA/ROCm,
    whose FusedAdam builders assume one dtype for all four tensor lists. load_state_dict restores the moments' dtype
    after loading, because torch.optim.Optimizer.load_state_dict casts every floating-point state tensor to its
    parameter's dtype and would bring them back as fp32 after a checkpoint reload.
  • Engine: with bf16_optimizer_states and fp32 master weights it builds FusedAdam with exp_avg_dtype=torch.bfloat16,
    and raises a ValueError for any other optimizer (torch_adam, Lamb, Muon, client optimizers). Those keep their
    states in the master weights' fp32, so the setting would otherwise save nothing without saying so.
  • Config: the combination is accepted for ZeRO 1, 2 and 3 without optimizer offload. With offload_optimizer it is
    rejected (DeepSpeedCPUAdam keeps its moments in the parameters' dtype); offloading bf16 states still needs
    bf16_master_weights_and_grads, as today.
  • Docs: config-json.md (bf16_optimizer_states entry and a support matrix for fp32 master weights) and the
    precision_config.py docstring.

Tests

  • test_fused_adam_low_precision_moments_with_fp32_params (tests/unit/ops/adam/test_adamw.py, bf16 and fp16
    moments): FusedAdam against the file's existing fp32 reference step, with the reference moments stored in the low
    precision. Checks the moments' dtype, the parameters within 2% of the total update, and the moments within one
    spacing of the moment dtype at their largest magnitude (the kernel's fused multiply-adds can round a value on a
    rounding midpoint the other way).
  • TestFP32MasterWeightsWithBF16OptimizerStates (tests/unit/v1/half_precision/test_bf16.py, 2 GPUs, ZeRO 1, 2 and
    3): master weights stay fp32, moments are bf16, losses over 5 steps within 1% of the fp32-moment run; a checkpoint
    reload keeps the moments in bf16, bit for bit.
  • TestFP32MasterWeightsWithBF16OptimizerStatesRejected: optimizer offload and torch_adam are rejected.
  • On master these new tests fail (the configuration is rejected at config time): all 10 fail on master 44661723e.
  • Not added to CI because of its memory (about 30 GB on one GPU): the kernel against the reference on a tensor of
    2^31 + 1,000 elements, which takes the kernel's 64-bit indexing path: passes, with the parameters within 4e-6 of the total update and the moments within 1.5e-3 of their largest value at both ends of the tensor.

Hardware: 1 node, 8 x NVIDIA H100 80GB HBM3, driver 595.112.07; NVIDIA PyTorch container, torch 2.14.0a0 (nv26.08),
CUDA 13.4, transformers 5.18.0, pytest 9.0.3. FusedAdam built from this tree's .cu file.

pytest tests/unit/v1/half_precision/test_bf16.py             # 41 passed (from the repository root, see below)
cd tests
pytest unit/ops/adam/test_adamw.py                           # 24 passed
pytest unit/v1/ops/muon/test_muon_partial_training.py        # 21 passed (Muon builds its Adam half through the changed path)
pytest unit/checkpoint/test_latest_checkpoint.py             # 2 passed

test_bf16.py ran from the repository root. A tree that is not installed reads its version from version.txt in the
current directory; from tests/ it reports 0.0.0, and the ZeRO-1 checkpoint loader refuses a checkpoint with that
version, so the ZeRO-1 checkpoint-reload case fails there for that reason only.

pre-commit run --files passes on the nine changed files (clang-format 18.1.3, as in requirements-dev.txt).

Compatibility

Cell Validated
ZeRO 1, 2, 3, bf16, fp32 master + bf16 moments, 2 GPUs yes (unit tests)
ZeRO-2, 8 GPUs, AutoEP + DeepEP, Qwen3-30B-A3B yes (the results above)
Checkpoint save and reload, ZeRO 1, 2, 3 yes (unit test)
bf16_master_weights_and_grads with or without bf16_optimizer_states, with or without offload unchanged; the existing TestBF16MasterWeightsGradients cases pass
fp16 training, optimizer offload, NVMe offload unchanged or rejected (see Changes)
Universal checkpointing not run. It loads moment fragments into states created by an optimizer step, which this change allocates in bf16
Accelerators other than CUDA/ROCm rejected with a clear error

Related work

  • Open PR Fuse mixed-precision gradients and parameter writeback into FusedAdam #8673 adds a separate CUDA entry point to multi_tensor_adam.cu for the non-ZeRO FP16_Optimizer
    (fp16/bf16 gradients and output parameters, fp32 master weights and fp32 moments in one launch). It does not offer
    low-precision moments, so the two changes do not duplicate each other; both add code to the same .cu file, so the
    second to merge may need a small rebase.
  • No open PR or issue adds fp32 master weights with bf16 moments (searched GitHub for bf16_optimizer_states,
    exp_avg_dtype and "bf16 optimizer states").

AI assistance

This change, its tests and this description were prepared with AI assistance (Claude).

🤖 Generated with Claude Code

…es without bf16 master weights)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: pengdurice <pengduhit@gmail.com>
@pengdurice pengdurice changed the title Allow bf16 Adam moments with fp32 master weights (as reminder) Allow bf16 Adam moments with fp32 master weights Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant