Skip to content

Reuse the static rollout decode graph across calls - #9077

Draft
jinyouzhi wants to merge 1 commit into
deepspeedai:masterfrom
jinyouzhi:static-graph-reuse
Draft

jinyouzhi wants to merge 1 commit into
deepspeedai:masterfrom
jinyouzhi:static-graph-reuse

Conversation

@jinyouzhi

@jinyouzhi jinyouzhi commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The default graph path (use_graph_capture=True without continuous_batch_size) builds a new DeepSpeedStaticCache, runs three warmup forwards and captures a new graph on every generate() call. In an RL loop that is a fixed cost per rollout round, although the decode graph depends only on the decode shape.

This PR keeps the static path's graph and cache across calls, as #8732 / #9009 do for the continuous path. Rebased onto #8602, so it covers the segment-KI decode path too.

Before After
Per generate() call Allocate cache, 3 warmups, capture Eager prefill; replay the retained graph
Recapture trigger Every call Graph signature changes: batch size, prompt width + max_new_tokens, device, dtype, parameter layout, training mode, autocast, b=1 step-kernel use, or a segki_refresh()
In-place weight updates n/a Without segKI: replayed without recapture. With segKI: sync_weights() / refresh_weights() / segki_refresh() force a recapture, because the injected kernels cache tensors derived from the weights
AutoTP n/a Ranks agree on recapture with an all-reduce, so no rank captures while its peers replay
shutdown() Releases the continuous state Releases both states

segKI (#8602)

  • The segKI kernels are installed once per capture. Hybrid (GDN) models keep the HF prefill cache that the decode cache's GDN slots are bound to, so the graph's state addresses stay fixed across calls.
  • segki_refresh() bumps a counter in the graph signature, so a graph that captured the freed derived tensors is never replayed. refresh_weights() additionally releases it right away.
  • Making segKI graphs survive a weight sync needs the derived tensors refilled in place (including the backend GLU concat cache). Left for a follow-up.

Fixes to the post-#8602 graph path

While rebasing I found three tests on master failing since #8602; they pass again here:

  • test_static_graph_uses_prefill_kv_shape: the decode cache is sized from the prefill's local KV tensors again (Size HybridEngine rollout KV caches from local prefill shapes under AutoTP #9008), not from the config divided by world size.
  • test_static_graph_matches_native_llama_generation[eager]: eager attention adds the mask to the scores, so it gets an additive mask built inside the graph from the boolean one.
  • test_decode_repetition_penalty_matches_hf_generate: the repetition penalty is applied on the graph prefill and Python decode paths again; the b=1 step kernel is used only without a penalty.

Tests

  • test_static_graph_is_reused_across_rollouts_with_the_same_decode_shape (Llama): capture once per decode shape, in-place update changes the tokens without recapture, train mode and autocast force recapture, shutdown() frees the cache.
  • TestAutoTPGraphRecapture (2 GPUs): ranks recapture together.
  • test_graph_reused_until_refresh[1, 2] (Qwen3.5-0.8B, segKI): one capture across two rollouts with the same tokens; segki_refresh() and refresh_weights() each force one recapture with the same tokens.

4× NVIDIA L20, torch 2.14.1+cu130, Transformers 5.17.0, test_segment_ki_kernels.py, test_graph_position_regression.py, runtime/rollout/test_hybrid_engine_rollout.py, v1/test_hybrid_engine_rollout_v1.py: 220 passed, 3 skipped, 3 failed. The 3 failures fail the same way on master on this machine (Qwen3RMSNorm has no eps in Transformers 5.17; torch.get_accelerator missing in this torch). Master on the same files: 6 more failures (listed above).

Qwen3.5-0.8B, segKI + graph, prompt 128, 128 new tokens, greedy: tokens identical to master at b=1 and b=4; per-call time 0.63 → 0.53 s (b=1) and 0.76 → 0.65 s (b=4) from skipping the recapture.

Not covered

  • Keeping segKI graphs across a weight sync (see above).
  • A pinned cache capacity so different total lengths share one graph.
  • ZeRO-3, offload, HybridEngine kernel injection.

This PR was prepared with AI assistance (GitHub Copilot); I reviewed and tested the changes.

@tohtana tohtana added the needs-gpu-test GPU testing is recommended to validate the changes in this PR. label Nov 13, 2026
@tohtana tohtana removed the needs-gpu-test GPU testing is recommended to validate the changes in this PR. label Nov 14, 2026
@jinyouzhi
jinyouzhi marked this pull request as ready for review November 15, 2026 07:00
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T12:49:10.095640Z 56ffdb3 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 968022173a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@jinyouzhi
jinyouzhi marked this pull request as draft November 17, 2026 08:53
@jinyouzhi
jinyouzhi marked this pull request as ready for review November 17, 2026 12:41

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 56ffdb3198

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +1021 to +1022
return (id(module), module.training, device.type, device_index, model_dtype, max_batch_size, max_cache_len,
tensor_signature)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Recapture graphs when autocast changes

When the newly retained static graph is reused across different autocast contexts (for example, first under torch.autocast('cuda', dtype=torch.bfloat16) and later with autocast disabled), every field in this signature remains unchanged because model_dtype is only the parameter dtype. Graph replay bypasses autocast dispatch and continues running the kernels captured under the first context, while the eager prefill follows the current context, so decode can silently use the wrong precision; include the accelerator autocast enabled state and dtype in the shared signature so both graph paths recapture.

AGENTS.md reference: AGENTS.md:L19-L19

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0f5e412: the autocast enabled state and dtype are now part of the shared graph signature, so both graph paths capture again when they change. The reuse test runs a bf16 copy of the model with and without bf16 autocast and expects a new capture; it fails without the change (autocast still switches kernels at the model's own dtype, e.g. softmax in fp32).

Rebase the static-graph reuse onto the segment-KI decode path from deepspeedai#8602.

- Keep the DeepSpeedStaticCache, decode inputs and captured graph across
  generate() calls with the same decode signature (batch, cache length,
  device, dtype, parameter layout, training mode, autocast, step-kernel
  use). AutoTP ranks agree on recapture through an all-reduce.
- Without segKI, in-place weight updates are replayed without recapture.
  With segKI, segki_refresh() bumps a generation in the signature and
  refresh_weights()/sync_weights() release the graphs, because the
  injected kernels cache tensors derived from the weights.
- Size the decode KV cache from the prefill's local KV tensors again, so
  AutoTP-sharded and narrowed-head models get the right shape.
- Apply the repetition penalty on the graph prefill and Python decode
  paths again; the b=1 step kernel is used only without a penalty.
- Give eager attention an additive mask built inside the graph from the
  boolean decode mask.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants