Skip to content

[BUG] HybridEngineRollout fails under autocast when the autocast dtype differs from the model dtype #9102

Open

Description

@jinyouzhi

Describe the bug
HybridEngineRollout.generate() fails under torch.autocast when the autocast dtype differs from the model's parameter dtype (e.g. an fp32 model under bf16 autocast), on every path that pre-allocates a KV cache. The plain batched path (HF generate() with a dynamic cache) works.

The pre-allocated caches are created with the parameter dtype (model_dtype = next(module.parameters()).dtype), while the keys/values produced under autocast are bf16, so the cache write fails:

path result
batched, no graph ok
batched, use_graph_capture=True index_copy_(): self and source expected to have the same dtype, but got (self) Float and (source) BFloat16
continuous_batch_size set, no graph Index put requires the source and destination dtypes match, got Float for the destination and BFloat16 for the source.
continuous_batch_size set, use_graph_capture=True same as above

This setup comes up with the torch_autocast config, which keeps parameters in fp32 and runs the forward under autocast. The rollout calls engine.module directly, so getting a bf16 rollout means wrapping generate() in torch.autocast.

To Reproduce

import torch
from types import SimpleNamespace
from transformers import LlamaConfig, LlamaForCausalLM
from deepspeed.runtime.rollout.hybrid_engine_rollout import HybridEngineRollout, HybridEngineRolloutConfig
from deepspeed.runtime.rollout.base import RolloutRequest, SamplingConfig

cfg = LlamaConfig(vocab_size=64, hidden_size=64, intermediate_size=128, num_hidden_layers=2,
                  num_attention_heads=4, num_key_value_heads=2, max_position_embeddings=64)
cfg._attn_implementation = "sdpa"
model = LlamaForCausalLM(cfg).cuda().eval()
prompts = torch.randint(1, 64, (4, 8)).cuda()
rollout = HybridEngineRollout(SimpleNamespace(module=model), SimpleNamespace(pad_token_id=0, eos_token_id=-1),
                              cfg=HybridEngineRolloutConfig(use_graph_capture=True))
with torch.autocast("cuda", dtype=torch.bfloat16):
    rollout.generate(RolloutRequest(prompts, torch.ones_like(prompts)),
                     SamplingConfig(max_new_tokens=8, temperature=0))

Expected behavior
Either the cache follows the dtype the model actually produces under the active autocast context, or the rollout rejects this combination up front with a clear error.

System info

  • DeepSpeed master (fd8be8161)
  • torch 2.13.0+cu130, transformers 5.15.1, 1x NVIDIA H20

Additional context
Found while addressing review feedback on #9077, which adds the autocast state to the graph signature so a retained graph is recaptured when the autocast context changes. That covers the case where the autocast dtype matches the model dtype; this mismatch fails before and after that change.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions