Repository navigation
Conversation
29e7658 to
1ffa78a
Compare
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 968022173a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 56ffdb3198
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| return (id(module), module.training, device.type, device_index, model_dtype, max_batch_size, max_cache_len, | ||
| tensor_signature) |
There was a problem hiding this comment.
Recapture graphs when autocast changes
When the newly retained static graph is reused across different autocast contexts (for example, first under torch.autocast('cuda', dtype=torch.bfloat16) and later with autocast disabled), every field in this signature remains unchanged because model_dtype is only the parameter dtype. Graph replay bypasses autocast dispatch and continues running the kernels captured under the first context, while the eager prefill follows the current context, so decode can silently use the wrong precision; include the accelerator autocast enabled state and dtype in the shared signature so both graph paths recapture.
AGENTS.md reference: AGENTS.md:L19-L19
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 0f5e412: the autocast enabled state and dtype are now part of the shared graph signature, so both graph paths capture again when they change. The reuse test runs a bf16 copy of the model with and without bf16 autocast and expects a new capture; it fails without the change (autocast still switches kernels at the model's own dtype, e.g. softmax in fp32).
0f5e412 to
26190f1
Compare
Rebase the static-graph reuse onto the segment-KI decode path from deepspeedai#8602. - Keep the DeepSpeedStaticCache, decode inputs and captured graph across generate() calls with the same decode signature (batch, cache length, device, dtype, parameter layout, training mode, autocast, step-kernel use). AutoTP ranks agree on recapture through an all-reduce. - Without segKI, in-place weight updates are replayed without recapture. With segKI, segki_refresh() bumps a generation in the signature and refresh_weights()/sync_weights() release the graphs, because the injected kernels cache tensors derived from the weights. - Size the decode KV cache from the prefill's local KV tensors again, so AutoTP-sharded and narrowed-head models get the right shape. - Apply the repetition penalty on the graph prefill and Python decode paths again; the b=1 step kernel is used only without a penalty. - Give eager attention an additive mask built inside the graph from the boolean decode mask. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Summary
The default graph path (
use_graph_capture=Truewithoutcontinuous_batch_size) builds a newDeepSpeedStaticCache, runs three warmup forwards and captures a new graph on everygenerate()call. In an RL loop that is a fixed cost per rollout round, although the decode graph depends only on the decode shape.This PR keeps the static path's graph and cache across calls, as #8732 / #9009 do for the continuous path. Rebased onto #8602, so it covers the segment-KI decode path too.
generate()callmax_new_tokens, device, dtype, parameter layout, training mode, autocast, b=1 step-kernel use, or asegki_refresh()sync_weights()/refresh_weights()/segki_refresh()force a recapture, because the injected kernels cache tensors derived from the weightsshutdown()segKI (#8602)
segki_refresh()bumps a counter in the graph signature, so a graph that captured the freed derived tensors is never replayed.refresh_weights()additionally releases it right away.Fixes to the post-#8602 graph path
While rebasing I found three tests on master failing since #8602; they pass again here:
test_static_graph_uses_prefill_kv_shape: the decode cache is sized from the prefill's local KV tensors again (Size HybridEngine rollout KV caches from local prefill shapes under AutoTP #9008), not from the config divided by world size.test_static_graph_matches_native_llama_generation[eager]: eager attention adds the mask to the scores, so it gets an additive mask built inside the graph from the boolean one.test_decode_repetition_penalty_matches_hf_generate: the repetition penalty is applied on the graph prefill and Python decode paths again; the b=1 step kernel is used only without a penalty.Tests
test_static_graph_is_reused_across_rollouts_with_the_same_decode_shape(Llama): capture once per decode shape, in-place update changes the tokens without recapture, train mode and autocast force recapture,shutdown()frees the cache.TestAutoTPGraphRecapture(2 GPUs): ranks recapture together.test_graph_reused_until_refresh[1, 2](Qwen3.5-0.8B, segKI): one capture across two rollouts with the same tokens;segki_refresh()andrefresh_weights()each force one recapture with the same tokens.4× NVIDIA L20, torch 2.14.1+cu130, Transformers 5.17.0,
test_segment_ki_kernels.py,test_graph_position_regression.py,runtime/rollout/test_hybrid_engine_rollout.py,v1/test_hybrid_engine_rollout_v1.py: 220 passed, 3 skipped, 3 failed. The 3 failures fail the same way on master on this machine (Qwen3RMSNormhas noepsin Transformers 5.17;torch.get_acceleratormissing in this torch). Master on the same files: 6 more failures (listed above).Qwen3.5-0.8B, segKI + graph, prompt 128, 128 new tokens, greedy: tokens identical to master at b=1 and b=4; per-call time 0.63 → 0.53 s (b=1) and 0.76 → 0.65 s (b=4) from skipping the recapture.
Not covered
This PR was prepared with AI assistance (GitHub Copilot); I reviewed and tested the changes.