You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] HybridEngineRollout fails under autocast when the autocast dtype differs from the model dtype #9102
Describe the bug HybridEngineRollout.generate() fails under torch.autocast when the autocast dtype differs from the model's parameter dtype (e.g. an fp32 model under bf16 autocast), on every path that pre-allocates a KV cache. The plain batched path (HF generate() with a dynamic cache) works.
The pre-allocated caches are created with the parameter dtype (model_dtype = next(module.parameters()).dtype), while the keys/values produced under autocast are bf16, so the cache write fails:
path
result
batched, no graph
ok
batched, use_graph_capture=True
index_copy_(): self and source expected to have the same dtype, but got (self) Float and (source) BFloat16
continuous_batch_size set, no graph
Index put requires the source and destination dtypes match, got Float for the destination and BFloat16 for the source.
continuous_batch_size set, use_graph_capture=True
same as above
This setup comes up with the torch_autocast config, which keeps parameters in fp32 and runs the forward under autocast. The rollout calls engine.module directly, so getting a bf16 rollout means wrapping generate() in torch.autocast.
Expected behavior
Either the cache follows the dtype the model actually produces under the active autocast context, or the rollout rejects this combination up front with a clear error.
Additional context
Found while addressing review feedback on #9077, which adds the autocast state to the graph signature so a retained graph is recaptured when the autocast context changes. That covers the case where the autocast dtype matches the model dtype; this mismatch fails before and after that change.
Activity