Repository navigation
Conversation
This was referenced Nov 13, 2026
jinyouzhi
force-pushed
the
static-graph-decode-step
branch
from
November 19, 2026 14:23
3fba9b5 to
7640fb3
Compare
Rebase the static-graph reuse onto the segment-KI decode path from deepspeedai#8602. - Keep the DeepSpeedStaticCache, decode inputs and captured graph across generate() calls with the same decode signature (batch, cache length, device, dtype, parameter layout, training mode, autocast, step-kernel use). AutoTP ranks agree on recapture through an all-reduce. - Without segKI, in-place weight updates are replayed without recapture. With segKI, segki_refresh() bumps a generation in the signature and refresh_weights()/sync_weights() release the graphs, because the injected kernels cache tensors derived from the weights. - Size the decode KV cache from the prefill's local KV tensors again, so AutoTP-sharded and narrowed-head models get the right shape. - Apply the repetition penalty on the graph prefill and Python decode paths again; the b=1 step kernel is used only without a penalty. - Give eager attention an additive mask built inside the graph from the boolean decode mask. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
The static graph path captured only the model forward for b>1 and for sampling or a repetition penalty. After every replay the host chose the next token, wrote the position and attention mask, and checked for EOS. Capture the token selection with the forward for every batch size: the graph applies the repetition penalty over the device token buffer, picks the token (argmax, or HF warpers plus multinomial), records EOS, and advances the position, mask and output buffer for the next replay. The b=1 greedy decode_step_graph kernel from segment KI is still used where it applies. The host only replays and checks for EOS every 16 steps, then trims the output to the step where the last row produced EOS, as generate() does. Sampling, repetition penalty and EOS ids join the graph signature, and the generator state is restored after warmup so the graph draws the same random numbers as generate(). Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On the static graph path, #8602 added a
decode_step_graphkernel that picks the token inside the graph, but only for b=1 greedy decoding without a repetition penalty. Every other case (b>1, sampling, repetition penalty, non-bf16 logits) captures only the forward: after every replay the host picks the next token, writes the position and mask, and checks for EOS.This PR captures the whole decode step for every batch size:
decode_step_graphmultinomial) + EOS flag + position / mask / output-buffer update, in PyTorch opsfinishedflagmax_new_tokensgenerate()does_apply_repetition_penaltybit for bit.generate().inference-engine.rstupdated.Tests
test_static_graph_stops_once_every_row_has_produced_eos: every row hits EOS on its third token; graph output equals the eager rollout, and the graph is replayed fewer thanmax_new_tokens - 1times.test_static_graph_capture_receives_a_prebuilt_4d_masknow replays the captured step, since the step itself advances the mask.model.generate()cover the in-graph selection.4× NVIDIA L20, torch 2.14.1+cu130, Transformers 5.17.0, same files as #9077: 221 passed, 3 skipped, 3 failed (the same 3 environment failures as on master).
Qwen3.5-0.8B, segKI + graph: tokens identical to master at b=1 and b=4.
Performance
Qwen3-0.6B, no segKI, L20, prompt 128, 256 new tokens, EOS disabled; decode tok/s:
Throughput is unchanged where decode is GPU-bound. The gain is that rollouts ending in EOS stop without a device sync on every token, for every batch size and sampling setup.
This PR was prepared with AI assistance (GitHub Copilot); I reviewed and tested the changes.