Repository navigation
Fix HybridEngineRollout KV caches under autocast when keys and values differ in dtype - #9124
ArchanaChetan07 wants to merge 1 commit into
Conversation
Under torch.autocast an fp32 Llama-style model produces fp32 keys (the rotary embedding multiplies the bf16 key projection by fp32 cos/sin) but bf16 values. DeepSpeedStaticLayer sizes its buffers from the first keys it sees and then rejected the bf16 values, and the static graph path's HF StaticCache prefill failed the same way. Cast incoming key/value states to the cache dtype in DeepSpeedStaticLayer.update, and prefill the static graph path into a DeepSpeedStaticCache instead of HF StaticCache. Fixes deepspeedai#9102 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ArchanaChetan07 <apatil@sandiego.edu>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Thanks for the fix and the clear reproduction test! Reviewed against master @ The fix looks correct. Blocker: needs a rebase. The PR currently conflicts with master ( Non-blocking suggestions:
Thanks! |
Fixes #9102.
Root cause
The issue attributes the failure to caches being created with the parameter dtype. Tracing it on master, the cause is slightly different: under
torch.autocast(dtype=bf16)an fp32 Llama model produces keys and values in different dtypes:k_proj/v_projrun under autocast and return bf16.apply_rotary_pos_embmultiplies the bf16 keys by the fp32cos/sin(the rotary module runs with autocast disabled), so the keys are promoted back to fp32. The values stay bf16.The rollout caches take their dtype from the first keys they see (fp32), so writing the bf16 values fails:
DeepSpeedStaticLayer.update(continuous batching, with or without graph capture):Index put requires the source and destination dtypes match, got Float ... BFloat16._generate_graph) prefills into HFStaticCache, whoseupdatehas the same limitation:index_copy_(): self and source expected to have the same dtype ... Float ... BFloat16.Allocating the caches in the autocast dtype would not fix this, because the keys really are fp32.
Change
DeepSpeedStaticLayer.update: cast the incoming key/value states to the cache dtype before writing. Here that is a lossless bf16 -> fp32 upcast of the values. When the dtypes already match,.to()is a no-op and returns the same tensor, so graph capture and the non-autocast paths are unchanged._generate_graph: prefill intoDeepSpeedStaticCacheinstead of HFStaticCache. With no write position set,DeepSpeedStaticLayerwrites the prompt at positions[0, prompt_len), and itsget_mask_sizes/get_query_offset/get_seq_lengthgive the same mask a fresh HFStaticCachegives for a single prefill forward. The existingtest_static_graph_matches_native_llama_generationcases still pass. The prefill KV is then copied into the decode cache exactly as before.Net: +36 / -3, two source files and one test.
Relation to other PRs
gh pr list --search "9102"and keyword searches on "rollout autocast" / "rollout kv cache dtype")._generate_graphblock where this PR changes the prefill cache (3 lines), so whichever lands second will need a small rebase. I'm happy to rebase onto Reuse the static rollout decode graph across calls #9077, or to drop the_generate_graphpart and leave it to Reuse the static rollout decode graph across calls #9077 if maintainers prefer.generate()uses HFDynamicCache(concatenation handles mixed dtypes) and already worked. The continuous prefill uses HFgenerate()for models without_supports_cache_class(e.g. Llama), and those KV tensors are copied intoDeepSpeedStaticLayer, so they are covered by theupdatechange.Tests
New test:
tests/unit/v1/test_hybrid_engine_rollout_v1.py::test_rollout_under_autocast_matches_hf_generate, parametrized over static graph / continuous graph / continuous eager. It runs greedy rollout of an fp32 tiny Llama under bf16 autocast and compares the tokens against HFmodel.generate()under the same autocast. It skips when bf16 or graph capture is unsupported.pytest unit/v1/test_hybrid_engine_rollout_v1.py -k autocastpytest unit/v1/test_hybrid_engine_rollout_v1.py unit/utils/test_static_cache.py unit/runtime/rolloutDS_ACCELERATOR=cpupytest -n 8 unit/runtime/rollout unit/utils/test_static_cache.py unit/v1/test_hybrid_engine_rollout_v1.pypre-commit run --files <changed files>passes (yapf, flake8, codespell, check-torchcuda, check-torchdist, license).Not validated: bf16-native GPUs (sm_80+; the T1000 runs bf16 through emulation), fp16 autocast, ZeRO-3 / AutoTP rollout under autocast, and multi-GPU.
AI assistance
AI assistance (Claude Code) was used to reproduce the bug, trace the root cause, draft the change and test, and run the commands above. I reviewed every changed line and ran the tests myself.
🤖 Generated with Claude Code