Skip to content

Run the whole static rollout decode step inside the CUDA graph - #9079

Draft
jinyouzhi wants to merge 2 commits into
deepspeedai:masterfrom
jinyouzhi:static-graph-decode-step
Draft

jinyouzhi wants to merge 2 commits into
deepspeedai:masterfrom
jinyouzhi:static-graph-decode-step

Conversation

@jinyouzhi

@jinyouzhi jinyouzhi commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #9077. Until it merges, this branch also contains its commit; only the last commit (Run the whole static rollout decode step inside the CUDA graph) is new here.

Summary

On the static graph path, #8602 added a decode_step_graph kernel that picks the token inside the graph, but only for b=1 greedy decoding without a repetition penalty. Every other case (b>1, sampling, repetition penalty, non-bf16 logits) captures only the forward: after every replay the host picks the next token, writes the position and mask, and checks for EOS.

This PR captures the whole decode step for every batch size:

Before After
b=1 greedy, bf16, no penalty Forward + decode_step_graph Unchanged
Every other case Forward only; token selection and updates on the host Forward + repetition penalty + token selection (argmax or HF warpers + multinomial) + EOS flag + position / mask / output-buffer update, in PyTorch ops
EOS check b>1: every step (device sync) Every 16 steps, on a device-side finished flag
Output width Always max_new_tokens Ends at the step where the last row produced EOS, as generate() does
Graph signature #9077's Plus sampling settings, repetition penalty and EOS ids
  • The in-graph repetition penalty marks the counted positions in a boolean buffer and scatters into a vocabulary + 1 mask, so it has fixed shapes and matches _apply_repetition_penalty bit for bit.
  • The RNG state is restored after warmup, so graph sampling draws the same numbers as generate().
  • Docs: inference-engine.rst updated.

Tests

  • New test_static_graph_stops_once_every_row_has_produced_eos: every row hits EOS on its third token; graph output equals the eager rollout, and the graph is replayed fewer than max_new_tokens - 1 times.
  • test_static_graph_capture_receives_a_prebuilt_4d_mask now replays the captured step, since the step itself advances the mask.
  • Existing greedy / sampling / repetition-penalty parity tests against model.generate() cover the in-graph selection.

4× NVIDIA L20, torch 2.14.1+cu130, Transformers 5.17.0, same files as #9077: 221 passed, 3 skipped, 3 failed (the same 3 environment failures as on master).

Qwen3.5-0.8B, segKI + graph: tokens identical to master at b=1 and b=4.

Performance

Qwen3-0.6B, no segKI, L20, prompt 128, 256 new tokens, EOS disabled; decode tok/s:

batch sampling #9077 this PR
1 greedy 209 209
1 T=1.0 205 207
8 greedy 1156 1162
8 T=1.0 1140 1149
32 greedy 2091 2096
32 T=1.0 2069 2073

Throughput is unchanged where decode is GPU-bound. The gain is that rollouts ending in EOS stop without a device sync on every token, for every batch size and sampling setup.

This PR was prepared with AI assistance (GitHub Copilot); I reviewed and tested the changes.

Rebase the static-graph reuse onto the segment-KI decode path from deepspeedai#8602.

- Keep the DeepSpeedStaticCache, decode inputs and captured graph across
  generate() calls with the same decode signature (batch, cache length,
  device, dtype, parameter layout, training mode, autocast, step-kernel
  use). AutoTP ranks agree on recapture through an all-reduce.
- Without segKI, in-place weight updates are replayed without recapture.
  With segKI, segki_refresh() bumps a generation in the signature and
  refresh_weights()/sync_weights() release the graphs, because the
  injected kernels cache tensors derived from the weights.
- Size the decode KV cache from the prefill's local KV tensors again, so
  AutoTP-sharded and narrowed-head models get the right shape.
- Apply the repetition penalty on the graph prefill and Python decode
  paths again; the b=1 step kernel is used only without a penalty.
- Give eager attention an additive mask built inside the graph from the
  boolean decode mask.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
The static graph path captured only the model forward for b>1 and for
sampling or a repetition penalty. After every replay the host chose the
next token, wrote the position and attention mask, and checked for EOS.

Capture the token selection with the forward for every batch size: the
graph applies the repetition penalty over the device token buffer, picks
the token (argmax, or HF warpers plus multinomial), records EOS, and
advances the position, mask and output buffer for the next replay. The
b=1 greedy decode_step_graph kernel from segment KI is still used where
it applies. The host only replays and checks for EOS every 16 steps,
then trims the output to the step where the last row produced EOS, as
generate() does. Sampling, repetition penalty and EOS ids join the graph
signature, and the generator state is restored after warmup so the graph
draws the same random numbers as generate().

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants