Skip to content

Fix HybridEngineRollout KV caches under autocast when keys and values differ in dtype - #9124

Open
ArchanaChetan07 wants to merge 1 commit into
deepspeedai:masterfrom
ArchanaChetan07:fix/9102-rollout-autocast-cache-dtype
Open

ArchanaChetan07 wants to merge 1 commit into
deepspeedai:masterfrom
ArchanaChetan07:fix/9102-rollout-autocast-cache-dtype

Conversation

@ArchanaChetan07

Copy link
Copy Markdown

Fixes #9102.

Root cause

The issue attributes the failure to caches being created with the parameter dtype. Tracing it on master, the cause is slightly different: under torch.autocast(dtype=bf16) an fp32 Llama model produces keys and values in different dtypes:

  • k_proj/v_proj run under autocast and return bf16.
  • apply_rotary_pos_emb multiplies the bf16 keys by the fp32 cos/sin (the rotary module runs with autocast disabled), so the keys are promoted back to fp32. The values stay bf16.

The rollout caches take their dtype from the first keys they see (fp32), so writing the bf16 values fails:

  • DeepSpeedStaticLayer.update (continuous batching, with or without graph capture): Index put requires the source and destination dtypes match, got Float ... BFloat16.
  • The static graph path (_generate_graph) prefills into HF StaticCache, whose update has the same limitation: index_copy_(): self and source expected to have the same dtype ... Float ... BFloat16.

Allocating the caches in the autocast dtype would not fix this, because the keys really are fp32.

Change

  • DeepSpeedStaticLayer.update: cast the incoming key/value states to the cache dtype before writing. Here that is a lossless bf16 -> fp32 upcast of the values. When the dtypes already match, .to() is a no-op and returns the same tensor, so graph capture and the non-autocast paths are unchanged.
  • _generate_graph: prefill into DeepSpeedStaticCache instead of HF StaticCache. With no write position set, DeepSpeedStaticLayer writes the prompt at positions [0, prompt_len), and its get_mask_sizes / get_query_offset / get_seq_length give the same mask a fresh HF StaticCache gives for a single prefill forward. The existing test_static_graph_matches_native_llama_generation cases still pass. The prefill KV is then copied into the decode cache exactly as before.

Net: +36 / -3, two source files and one test.

Relation to other PRs

Tests

New test: tests/unit/v1/test_hybrid_engine_rollout_v1.py::test_rollout_under_autocast_matches_hf_generate, parametrized over static graph / continuous graph / continuous eager. It runs greedy rollout of an fp32 tiny Llama under bf16 autocast and compares the tokens against HF model.generate() under the same autocast. It skips when bf16 or graph capture is unsupported.

Setup Command master this PR
1x NVIDIA T1000 8GB (sm_75), driver 596.51, torch 2.6.0+cu124, transformers 5.12.1, Windows 11, Python 3.11 pytest unit/v1/test_hybrid_engine_rollout_v1.py -k autocast 3 failed (the three errors from the issue) 3 passed
same pytest unit/v1/test_hybrid_engine_rollout_v1.py unit/utils/test_static_cache.py unit/runtime/rollout — 233 passed, 10 skipped
CPU (WSL2 Ubuntu, 24 cores), torch 2.15.0+cpu, transformers 5.23.0, Python 3.12, DS_ACCELERATOR=cpu pytest -n 8 unit/runtime/rollout unit/utils/test_static_cache.py unit/v1/test_hybrid_engine_rollout_v1.py autocast continuous case fails 200 passed, 43 skipped

pre-commit run --files <changed files> passes (yapf, flake8, codespell, check-torchcuda, check-torchdist, license).

Not validated: bf16-native GPUs (sm_80+; the T1000 runs bf16 through emulation), fp16 autocast, ZeRO-3 / AutoTP rollout under autocast, and multi-GPU.

AI assistance

AI assistance (Claude Code) was used to reproduce the bug, trace the root cause, draft the change and test, and run the commands above. I reviewed every changed line and ran the tests myself.

🤖 Generated with Claude Code

Under torch.autocast an fp32 Llama-style model produces fp32 keys (the
rotary embedding multiplies the bf16 key projection by fp32 cos/sin) but
bf16 values. DeepSpeedStaticLayer sizes its buffers from the first keys it
sees and then rejected the bf16 values, and the static graph path's
HF StaticCache prefill failed the same way.

Cast incoming key/value states to the cache dtype in
DeepSpeedStaticLayer.update, and prefill the static graph path into a
DeepSpeedStaticCache instead of HF StaticCache.

Fixes deepspeedai#9102

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ArchanaChetan07 <apatil@sandiego.edu>
@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 19, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T04:01:32.007090Z 9031d60 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@jinyouzhi

jinyouzhi commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Thanks for the fix and the clear reproduction test! Reviewed against master @ b5e0ba67f.

The fix looks correct. .to(self.dtype) after lazy init is the right root-cause fix for #9102: under torch.autocast(bf16) with an fp32 model, RoPE promotes keys back to fp32 while values stay bf16, so the lazily-initialized layer (dtype taken from the first keys) rejects the value writes. Since .to() is a no-op when dtypes already match, graph capture/replay and non-autocast paths are unaffected. Swapping the _generate_graph prefill cache to a lazy DeepSpeedStaticCache also matches the existing _generate_continuous pattern — the "write position ignored → writes land at [0, kv_length)" semantics are fine for the single full-prompt prefill used there, and the comment explaining why HF StaticCache is not used is helpful.

Blocker: needs a rebase. The PR currently conflicts with master (mergeable: CONFLICTING); the drift comes from a single commit, 97aa3c10d (#8602), which touched both source files (+521/−99). I verified that master still has the bug (lazy_initialization still derives self.dtype from key_states and update() has no cast), so this PR is still needed — please rebase onto master and re-run CI.

Non-blocking suggestions:

  1. Continuous-batch staging cache. _generate_continuous / _generate_continuous_graph still stage prefill through _create_static_cache(StaticCache, ..., dtype=model_dtype) in _continuous_prefill_cache. If HF StaticCache.update is dtype-strict, that is the same fp32-cache-gets-bf16-values hazard. The new (False, 2) parametrization should exercise it — worth confirming it actually reaches that branch (models with _supports_cache_class set).
  2. Static-cache-level unit test. A cheap addition to tests/unit/utils/test_static_cache.py: lazy-init a DeepSpeedStaticCache, then update() with fp32 keys + bf16 values, assert no error and values promoted to self.dtype. Existing tests always pass explicit dtypes + set_write_position, so this path is otherwise untested.
  3. GPU coverage. With skip-gpu-test, upstream CI only ran the (False, 2) parametrization on CPU (graph-capture cases skip without CUDA). Consider removing the label or linking a local CUDA run so the autocast + graph-capture path gets CI coverage.
  4. Heads-up on Reuse the static rollout decode graph across calls #9077 — it also touches ~3 lines in _generate_graph, so the rebase may have overlapping hunks.

Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] HybridEngineRollout fails under autocast when the autocast dtype differs from the model dtype

2 participants