Repository navigation
feat(rollout): segment-based kernel injection for hybrid-engine greedy decode - #8602
Conversation
11231a7 to
d41c914
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b31faddfc4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| from deepspeed.module_inject.kernel_reference import dual_gemv_silu_mul | ||
| return self.down_proj(dual_gemv_silu_mul(input, self.gate_proj.weight, self.up_proj.weight)) |
There was a problem hiding this comment.
Preserve the original MLP activation
The structural selector accepts every bias-free module with these three projection names, but this replacement always computes SiLU. A GELU-gated MLP such as Gemma satisfies that selector and will therefore silently produce different logits whenever use_segki=True; constrain detection to SiLU MLPs or retain/call the module's configured activation.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Several paths currently break gradients, padded prompts, fallback decoding, hybrid continuous batching, and native-op compilation or packaging.
Review effort: Balanced
Findings: 10
Open (18)
Missing CUDA bf16 and pybind functional headers · New Native gdn_gates path cuts off training gradients · New Tensor truth-value check breaks packed-sequence calls · New Custom attention ignores padding masks · New Fused add-norm loses gradients during training · New Unbound GDN slots are skipped without state initialization · New Prompt attention mask is dropped during prefill · New Fallback graph reuses already-consumed recurrent state · New Forward-only graph replay uses uncaptured token buffers · New EOS padding slice ignores prompt-length offset · New Dual-GEMV excludes normal 3D greedy-decode inputs · New Non-CUDA backends incorrectly load CUDA builder · New FP16 models incorrectly enter bf16-only decode kernels · New Decode loop builder is not registered for wheel builds · New Fused GLU builder is not registered for wheel builds · New CUDA-only tests run on unsupported accelerators · New Missing direct coverage for native decode fallback · New Gemma is documented as supported despite GELU rejection · New
What changed in this PR
Adds segment-based kernel injection and native decode acceleration for hybrid-engine greedy rollout.
Changes:
- Adds fused CUDA kernels, reference implementations, and op builders.
- Extends static-cache support for hybrid attention models.
- Integrates graph capture, native decode loops, documentation, and tests.
| File | Description |
|---|---|
csrc/module_inject/fused_glu.cu |
Implements fused CUDA kernels. |
csrc/module_inject/decode_loop.cu |
Implements the native decode loop. |
deepspeed/module_inject/segment_ki.py |
Detects and injects fused model segments. |
deepspeed/module_inject/kernel_reference.py |
Provides PyTorch kernel references. |
deepspeed/ops/module_inject/__init__.py |
Exports native-op builders. |
deepspeed/ops/module_inject/fused_glu.py |
Adds the fused-kernel builder. |
deepspeed/ops/module_inject/decode_loop.py |
Adds the decode-loop builder. |
deepspeed/utils/static_cache.py |
Adds hybrid cache slots and position semantics. |
deepspeed/runtime/rollout/hybrid_engine_rollout.py |
Integrates injection and accelerated decoding. |
tests/unit/module_inject/test_segment_ki_kernels.py |
Tests kernels and injection behavior. |
tests/unit/utils/test_static_cache.py |
Updates static-cache expectations. |
docs/code-docs/source/inference-engine.rst |
Documents segment-based injection. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Hi @delock, thanks for submitting this PR. I have left a few comments and questions. |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Graph compatibility, packed GDN execution, projection bias handling, and deterministic argmax behavior have unresolved correctness issues.
Review effort: Balanced
Findings: 3
Open (6)
Preserve packed-sequence boundaries through causal convolution · New Disable fused QKV projection when any bias is present · New Pass tensor masks for ordinary models during graph capture · New Match torch.argmax first-index tie breaking · New Add direct CUDA coverage for decode_step_graph · New Make live-weight integration test deterministically observe weight changes · New
Resolved since last review (18)
EOS padding slice ignores prompt-length offset Forward-only graph replay uses uncaptured token buffers Fallback graph reuses already-consumed recurrent state Prompt attention mask is dropped during prefill Unbound GDN slots are skipped without state initialization Fused add-norm loses gradients during training Custom attention ignores padding masks Tensor truth-value check breaks packed-sequence calls Native gdn_gates path cuts off training gradients Missing CUDA bf16 and pybind functional headers CUDA-only tests run on unsupported accelerators Fused GLU builder is not registered for wheel builds Decode loop builder is not registered for wheel builds FP16 models incorrectly enter bf16-only decode kernels Non-CUDA backends incorrectly load CUDA builder Dual-GEMV excludes normal 3D greedy-decode inputs Gemma is documented as supported despite GELU rejection Missing direct coverage for native decode fallback
Hi @nathon-lee , kindly reminder in case you didn't submit your review comments. Thanks! |
|
Hi, @delock Thanks for the PR and for addressing the earlier comments. I recommend addressing these two points before merging:
A few additional points to check; I am not treating these as confirmed merge blockers:
The position-handling regression is reproduced with the versions above. The other points are requests for testing or clarification, not confirmed runtime failures. |
|
Hi @nathon-lee , thanks for the review comments. I have followup them with subsequent commits. There is one finding, fused_add_norm didn't act correctly with BS>1, and it is found by comparing with HF forward.
|
|
Hi @delock, thanks for following up. That was a useful catch: the previous I did notice one issue at |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Hi @nathon-lee , sorry for the late reply, let me check with latest status of master. |
Hi @nathon-lee , the latest code should contain this missing implementation. Sorry for inconvience. |
| def total_memory(self, device_index=None): | ||
| ... | ||
|
|
||
| @abc.abstractmethod |
There was a problem hiding this comment.
I think we should not remove this?
|
The new |
…egrade Two hardening changes for silent-degradation paths found while validating on L20: 1. RoPE table install passes [3, 1, L] positions when the model's rotary is MRoPE (mrope_section). transformers 5.20/5.22 happen to expand 2-D positions internally, but strict MRoPE implementations raise -- and the install chain shares one try, so a raise silently skips install_decode_attention / install_fused_norm / install_lm_head and decode falls back to the HF forward with no signal. Verified a no-op on 5.20.2/5.22.1 (install counts and b=32 decode throughput unchanged: TP1 1244.1, TP2 1992.6 tok/s vs 1242.8/1993.0 medians). 2. That except now logs a warning with the exception instead of swallowing it (review feedback: silent fallbacks must be visible). Verified by injecting a simulated failure: warning fires and the fallback engages (attn install 0/8, fused_norm 0). 3. Same treatment for the GDN scan: without fla-core every scan (decode and prefill) silently runs the eager torch decomposition, measured 1.6x slower end-to-end on Qwen3.5-4B b=32 decode (1244 -> 645 tok/s). The per-layer FLA attribute is legitimately absent on transformers 5.19+, so the warning probes fla importability instead. Verified both ways: fla present -> no warning, 1244.5 tok/s; fla removed -> warning + 644.6 tok/s. Suggested by @jinyouzhi (MRoPE shape and the warning). Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
The amortized check (every 16 steps, from 8b05554) trades a per-step ~50us sync for up to 15 dead replays after the last row finishes. That wins only in EOS-free decoding (+0.1-0.3% measured at b=32); in EOS-heavy rollouts -- RL responses routinely end early and max_new_tokens is only a cap -- the dead replays cost ~20% per call at b=32 step times. Check every step again (the None fast path and the accumulating-mask break structure stay). Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
|
Hi @jinyouzhi thanks for the suggestions. I have corporated some of the suggestions and leave the others for future improvement. For optimizations that requires hook in refresh weight, maybe we should also leave them for future. This may make #9077 depend on this PR and refresh weight hook be implemented along with #9077 . Thanks! |
Thanks for incorporating the suggestions! Agreed. I'll study the refresh-weight path more carefully and revisit #9077's design accordingly, then update it after this PR lands. |
…ost model (deepspeedai#8786) ## November 11 update Rebased onto master `b9100c5`, which includes deepspeedai#8390 and deepspeedai#9008; new head `82ac56e`. Uses deepspeedai#8939's shared sampling kwargs, token selector and FP32 repetition-penalty helper. Manual graph and continuous-batching paths retain upstream temperature/top-k/top-p sampling. Adaptive prefill retains its single-sample greedy scope, including fixed-slot graph decode. deepspeedai#9004's pre-built 4-D graph mask is preserved. - **deepspeedai#8390 early stopping:** ordinary generation and Auto's batched fallback share deepspeedai#8390's generation kwargs, so they stop on natural EOS; ZeRO-3 keeps the group-aware synchronized stop. Auto remains single-process. - **Review fixes:** the graph path rejects `align_decode_fronts=True` only when Auto is disabled, because Auto always uses aligned bucketed prefill. The accelerator graph test covers both flag values. The documentation states that Auto takes precedence over `align_decode_fronts` on the graph path; cache trimming is still rejected. - **deepspeedai#9008 lazy KV layers:** the bucketed prefill copy initializes uninitialized cache layers through the shared `_continuous_cache_layer()` helper, like the equal-width path. - **Codex P2 (preparer findings):** after a custom `_prepare_generation_config()` runs, Auto rejects the prepared config if it changes any request override (token budget, EOS, PAD or greedy settings). Otherwise the batched route would honor the change inside `generate()` while the CB routes keep the request values. A preparer that replaces or edits the attention mask is rejected for the same reason. - **Codex P2 (mask lengths):** effective prompt lengths use `count_nonzero`, so BF16/FP16 masks longer than their exact-integer range no longer round the bucket width. - **Codex P1 (test reliability):** the route-planning profile test uses a frozen clock advanced by the model's generation-config preparer, instead of wall-clock bounds or mocked rollout internals. - The decode cost still uses the configured maximum generation budget. Natural EOS can shorten decode, so the extra-CB estimate can be too large; the documentation states this assumption. Validation on an 8 × H200 node, PyTorch 2.9.1+cu128, CUDA 12.8, Transformers 4.57.1: **232 passed, 3 skipped, 2 deselected** across the runtime rollout, scheduler, interface, static-cache and v1 accelerator suites, including deepspeedai#8390's 4-rank subgroup and 2-GPU ZeRO-3 synchronized-stop tests. Skips require the newer cache query-offset API; deselections are the native shared-prefill kernel and optional pretrained-model test. Pre-commit passed on all changed files. A small natural-EOS check (Qwen2.5-3B-Instruct, four short questions) ended every response at EOS, with Auto matching ordinary generation. This update used AI assistance. The performance tables below retain their original revision attribution; large-model throughput and peak memory were not remeasured for this rebase. ## Summary Closes deepspeedai#8785. Related tracking: [deepspeedai#8197](deepspeedai#8197), [deepspeedai#8462](deepspeedai#8462). **Performance follow-up to [deepspeedai#8497](deepspeedai#8497 Aligned CB avoids padding on long-tail prompts, but one prefill Forward per request is expensive for short prompts. A padding-ratio threshold cannot distinguish **128/4** from **512/16**, despite identical ratios and different fastest baselines. This change adds **one opt-in automatic policy** that compares ordinary no-CB generation with aligned CB using **DP-selected prefill buckets**. For 1×512 + 31×16, the selected buckets are `[1,512]` and `[31,16]`, reducing prefill Forwards from 32 to 2. **Auto is disabled by default.** Existing manual modes and positional configuration arguments are preserved. ```python cfg = HybridEngineRolloutConfig(adaptive_prefill=True) rollout = HybridEngineRollout(engine, tokenizer, cfg) sampling = SamplingConfig(max_new_tokens=16, temperature=0, continuous_batch_size=32) ``` ## Selection and configuration A length-sorted **O(n²)-time / O(n)-space DP** minimizes the additive estimated cost of contiguous buckets. It explores one batch, singleton batches and intermediate partitions without running the GPU for candidate evaluation. Each bucket uses ordinary left padding and a 2D attention mask; **no varlen kernel is needed**. Model geometry is read once per plan. ```text C(B,L) = fixed + token × B×L + attention × estimated_attention_GFLOPs + kv × estimated_KV_MiB attention_GFLOPs = 4 × B × L² × layers × query_heads × head_dim / 1e9 KV_MiB = 2 × B × L × layers × KV_heads × head_dim × element_bytes / 2^20 ``` | Parameter | Default | Meaning | | --- | ---: | --- | | `prefill_fixed_cost_ms` | 25.0 | Fixed milliseconds per prefill Forward | | `prefill_token_cost_ms` | 0.1 | Milliseconds per padded token position; projection/MLP term | | `prefill_attention_cost_ms` | 0.01 | Milliseconds per estimated attention GFLOP | | `prefill_kv_cost_ms` | 0.04 | Milliseconds per MiB of KV copied into the CB cache | | `continuous_decode_cost_ms` | 10.0 | Additional CB milliseconds per remaining Decode step | | `prefill_max_tokens` | 65536 | Maximum padded token positions per prefill Forward; `None` removes this limit | All cost coefficients must be finite and non-negative. **These are starting estimates, not universal GPU constants**; calibrate them for the model, device, precision and attention backend. Head dimension, query/KV heads, layers and parameter element size are read automatically. The DP is optimal for the configured estimates, not guaranteed globally optimal measured latency. Ordinary generation is eligible when every row fits `continuous_batch_size`, the physical input fits the prefill/context limits, and neither explicit static capacity nor trimming is requested. Its estimated cost omits CB cache-copy traffic; CB adds `continuous_decode_cost_ms × (max_new_tokens - 1)`. The first output token comes from Prefill. For CB, **every admission/refill replans the newly admitted requests**. Effective KV spans are copied into the existing right-aligned cache; surviving rows retain their layout. The policy does not change caller configuration or active-row bounds. Disabling Auto restores manual selection. Model/OOM errors propagate without trying another path. ## Correctness - Apply repetition penalty from the first token using **original padded history**, even when bucket model inputs are trimmed. - Preserve generated PAD-valued tokens through the first EOS, excluding structural padding after termination. - Include route planning and result normalization in end-to-end profiling; count **productive output tokens** and retain active/configured capacity fields. - Resolve HF's effective generation configuration before rejecting unsupported processors/output modes, including legacy model-config refresh; preserve explicit generation-config overrides. Auto requires single-process greedy rollout, one sample per prompt and cache-class support. With graph capture, Auto uses eager bucketed prefill and fixed-slot graph decode; cache trimming and shared prefill remain unsupported. ## Experiments Related work: [TurboTransformers, §5 (PPoPP 2021)](https://arxiv.org/html/2010.05680) already uses length sorting, profiled batch costs and an O(n²) DP to partition variable-length inference requests. [DynaPipe, §4](https://arxiv.org/html/2311.10418) applies cost-based DP partitioning to training micro-batches. This PR applies that established scheduling approach to HybridEngine rollout, with ordinary-generation comparison, refill planning and existing aligned-cache semantics; it makes no claim of a new DP algorithm. **Single GPU; Qwen/Qwen2.5-3B-Instruct; FP32 / SDPA; PyTorch 2.9.1+cu128; Transformers 4.57.1.** Each candidate receives **one warmup and five interleaved timed runs**, 32 requests and 16 output tokens per request. CB capacity is 32; trimming is disabled, cache capacity unset, TF32 disabled. The default cost coefficients above are unchanged across workloads. | Workload | Effective prompt lengths | Reason | | --- | --- | --- | | Short, similar | 20..13 repeated four times | Normal workload shape from deepspeedai#8497 | | Long tail | 1×512 + 31×16 | Long-tail workload from deepspeedai#8497 | | Short tail | 1×20 + 31×1 | Counterexample to ratio-only routing | | Moderate tail | 1×128 + 31×4 | Same padding ratio as 512/16, different best legacy path | | Spread | 16, 32, …, 512 | General distribution with intermediate buckets | Inputs repeat non-special token ID 14990. deepspeedai#8497 did not publish complete short-prompt sequences, so that case reconstructs its documented shape. Every timed response matches independent single-request eager generation; each workload produces **512 effective output tokens**. GPU-synchronized wall-clock timing includes the full `generate()` call, including Auto planning and normalization. Model loading and oracle work are outside timing; profiling is disabled during timed runs. All reported E2E values are **arithmetic means of five runs**. | Workload | No-CB (ms) | Equal-width CB (ms) | Aligned CB (ms) | Auto (ms) | Auto / best | Auto choice | | --- | ---: | ---: | ---: | ---: | ---: | --- | | Short, similar | **505.54** | 673.66 | 1549.74 | **501.77** | 0.9925× | No-CB | | 1×512 + 31×16 | 2458.46 | 2596.38 | **1682.10** | **840.06** | 0.4994× | 2 buckets | | 1×20 + 31×1 | **501.74** | 667.57 | 1331.82 | **497.57** | 0.9917× | No-CB | | 1×128 + 31×4 | **897.17** | 1053.26 | 1468.58 | **634.59** | 0.7073× | 2 buckets | | 16, 32, …, 512 | 2458.13 | 2595.33 | **2337.75** | **1984.86** | 0.8490× | 6 buckets | **All five workloads meet the 5% acceptance tolerance.** Auto chooses ordinary generation for both short cases; their small timing differences are not claimed as a kernel speedup. Relative to the fastest legacy baseline, mean latency falls by **50.06%** for 1×512 + 31×16, **29.27%** for 1×128 + 31×4, **15.10%** for 16, 32, …, 512. These improvements come from fewer prefill Forwards and reduced padded work. <details> <summary>Throughput and peak allocated memory for every path</summary> Throughput is 512 divided by mean latency. Memory is peak PyTorch allocated memory during timed calls, including model weights. | Workload | Path | Output tok/s | Peak allocated memory (MiB) | | --- | --- | ---: | ---: | | Short, similar | No-CB | 1012.77 | 11988.96 | | Short, similar | Equal-width CB | 760.03 | 12036.75 | | Short, similar | Aligned CB | 330.38 | 11928.77 | | Short, similar | **Auto** | 1020.40 | 11988.96 | | 1×512 + 31×16 | No-CB | 208.26 | 15558.69 | | 1×512 + 31×16 | Equal-width CB | 197.20 | 16747.20 | | 1×512 + 31×16 | Aligned CB | 304.38 | 13278.51 | | 1×512 + 31×16 | **Auto** | 609.48 | 13278.49 | | 1×20 + 31×1 | No-CB | 1020.46 | 11994.64 | | 1×20 + 31×1 | Equal-width CB | 766.96 | 12033.50 | | 1×20 + 31×1 | Aligned CB | 384.44 | 11925.07 | | 1×20 + 31×1 | **Auto** | 1029.00 | 11994.64 | | 1×128 + 31×4 | No-CB | 570.68 | 12742.62 | | 1×128 + 31×4 | Equal-width CB | 486.11 | 13066.75 | | 1×128 + 31×4 | Aligned CB | 348.64 | 12221.97 | | 1×128 + 31×4 | **Auto** | 806.82 | 12221.95 | | 16, 32, …, 512 | No-CB | 208.29 | 15558.69 | | 16, 32, …, 512 | Equal-width CB | 197.28 | 16747.20 | | 16, 32, …, 512 | Aligned CB | 219.01 | 13278.56 | | 16, 32, …, 512 | **Auto** | 257.95 | 13682.33 | </details> <details> <summary>Separate profiling breakdown</summary> One additional profiling call per candidate, outside acceptance timing. These diagnostics are not averages. Cache management is part of the generation overhead, not an additional independent total. | Workload | Path | Prefill (ms) | Decode (ms) | Cache management (ms) | Prefill / Decode Forwards | | --- | --- | ---: | ---: | ---: | ---: | | Short, similar | No-CB | 91.01 | 401.30 | N/A | 1 / 15 | | Short, similar | Equal-width CB | 91.99 | 499.53 | 26.80 | 1 / 15 | | Short, similar | Aligned CB | 955.52 | 500.30 | 35.25 | 32 / 15 | | Short, similar | **Auto** | 90.74 | 396.90 | N/A | 1 / 15 | | 1×512 + 31×16 | No-CB | 1918.22 | 531.89 | N/A | 1 / 15 | | 1×512 + 31×16 | Equal-width CB | 1919.76 | 595.14 | 27.09 | 1 / 15 | | 1×512 + 31×16 | Aligned CB | 981.04 | 594.52 | 35.02 | 32 / 15 | | 1×512 + 31×16 | **Auto** | 153.66 | 592.66 | 31.43 | 2 / 15 | | 1×20 + 31×1 | No-CB | 91.06 | 402.45 | N/A | 1 / 15 | | 1×20 + 31×1 | Equal-width CB | 91.84 | 494.84 | 26.88 | 1 / 15 | | 1×20 + 31×1 | Aligned CB | 743.30 | 491.93 | 34.48 | 32 / 15 | | 1×20 + 31×1 | **Auto** | 90.62 | 394.78 | N/A | 1 / 15 | | 1×128 + 31×4 | No-CB | 490.08 | 399.13 | N/A | 1 / 15 | | 1×128 + 31×4 | Equal-width CB | 490.85 | 482.08 | 27.09 | 1 / 15 | | 1×128 + 31×4 | Aligned CB | 889.01 | 483.21 | 34.83 | 32 / 15 | | 1×128 + 31×4 | **Auto** | 63.80 | 485.00 | 31.43 | 2 / 15 | | 16, 32, …, 512 | No-CB | 1918.03 | 531.64 | N/A | 1 / 15 | | 16, 32, …, 512 | Equal-width CB | 1918.95 | 594.68 | 27.32 | 1 / 15 | | 16, 32, …, 512 | Aligned CB | 1648.16 | 593.25 | 35.91 | 32 / 15 | | 16, 32, …, 512 | **Auto** | 1301.59 | 593.31 | 32.49 | 6 / 15 | The 128/4 and 512/16 workloads reduce prefill Forwards from 32 on aligned CB to **2 on Auto**. Spread uses **6 buckets**. All CB paths in these performance runs have zero trims and zero moved bytes; ordinary generation has no static-cache statistics. This makes no new trimming-speedup claim. </details> ## Validation and limits A separate microbenchmark measures **planner overhead**, without model Prefill: for 128/256 requests of 65,536 tokens, CPU DP takes **0.48/1.82 ms** on average with the default per-bucket token limit. It still checks all 8,256/32,896 candidate segments; over-limit segments skip cost evaluation. For 32 requests, DP takes 0.089 ms at length512 and 0.040 ms at length65,536 (0.078 ms with no token limit). GPU mask validation/length extraction for 32×65,536 takes 0.83 ms separately. Measurements use ten repeated timing samples after warmup. **These are not 64k model Prefill timings or the complete Auto-path overhead.** **The original validation passed 105 related rollout/cache tests**, including independent exhaustive partition comparisons for DP, device-backed tiny-Qwen tests for grouped prefill and refill, both cache APIs, effective generation-config/legacy refresh, repetition history, EOS/PAD masks, capacity bounds and external-wall-time checks for profiling. Existing tests cover retirement, compaction and trimming. Independent mixed-bucket checks also exercised live-survivor refill and trimming on/off against padded per-request HF generation. One pre-existing native shared-prefill extension test is excluded because the host has no CUDA compiler; **all added tests ran**. Changed-file pre-commit and `git diff --check` pass. Repeated local review ended with **no actionable findings**. ```sh python -m pytest -q \ tests/unit/runtime/rollout/test_hybrid_engine_rollout.py \ tests/unit/runtime/rollout/test_continuous_batching.py \ tests/unit/runtime/rollout/test_rollout_interface.py \ tests/unit/utils/test_static_cache.py \ -k 'not native_repeat_kv_cache_fp16_reverse_copy' ``` The prefill token limit does **not** bound allocated decode KV memory. A single prompt over that limit raises an explicit error; there is no chunked prefill. **64k prompts require a model context limit covering prompt plus output budget, sufficient KV capacity and temporary memory.** The measured checkpoint is configured for 32k. Planner/position-limit boundaries are tested; real 64k inference performance is not claimed. Performance evidence uses unsharded HF models behind `HybridEngineRollout`; the additional calibrated and real-question checks are listed below. Other model/device configurations, FP16, native kernel injection, ZeRO/TP training integration and production arrival/EOS distributions remain unmeasured. Controlled performance runs do not exercise staggered EOS or trimming; those have correctness coverage. No new dependency, kernel, online learner or trimming redesign. Related foundation: [deepspeedai#8368](deepspeedai#8368); profiling: [deepspeedai#8494](deepspeedai#8494). ## Reviewer follow-up [Commit 65a9115](deepspeedai@65a9115) propagates the resolved generation configuration through continuous Prefill, Decode and refill, and omits `output_logits` from legacy manual generation. **27 targeted tests pass**, including six new cases covering both Auto routes, both cache APIs and manual equal-width/aligned compatibility. The new tests reproduced four failures before the fix. The compatibility test uses a strict generation wrapper that rejects the unsupported keyword. Calibration and evaluation tooling moved to [DeepSpeedExamples deepspeedai#1018](deepspeedai/DeepSpeedExamples#1018), with its fitting regression test and usage documentation. **Merge the Examples companion before this PR so the documentation link is available.** The Examples companion includes MHA-head fallback and validates a source fingerprint covering rollout, planning and static-cache code. A seeded tiny-model GPU smoke check completed calibration and all four saved-config sweep workloads with matching output IDs on every path. Two independent local reviews returned no findings. The calibrated sweep from the preceding routing-overhead fix is separate from the original 16-token experiments above. It uses **two EOS-disabled output tokens**, SDPA and one GPU, with independent uniform calibration shapes and one unchanged coefficient set per configuration: | Configuration | 128×16 | 1×128 + 31×4 | 16..512 | 1×512 + 31×16 | Worst regression | | --- | ---: | ---: | ---: | ---: | ---: | | Tiny Qwen2 / FP32 | 1.0220× | 0.9685× | 0.9539× | 0.9647× | +2.205% | | Qwen2.5-3B / FP32 | 1.0005× | 0.2345× | 0.7808× | 0.2265× | +0.054% | | Qwen3-32B / BF16 | 1.0027× | 0.3881× | 0.7189× | 0.1811× | +0.273% | Values are **Auto E2E / fastest manual baseline E2E**. The two larger models select no-CB for 128×16 and bucketed CB for the other three cases; the tiny model selects no-CB throughout. Auto matches ordinary output IDs in all twelve cases. Target-model means use three interleaved repetitions after warmup; the tiny test uses ten. [Full sweep details and calibration limits](deepspeedai#8786 (comment)). A separate [Qwen3-32B real-question check](deepspeedai#8786 (comment)) selected no-CB on both workloads: MATH-500 averaged **31.452 s / 31.393 s** for ordinary/Auto over three runs; ARC-AGI-2 measured **237.560 s / 237.888 s** in one paired run. All eight responses matched. These checks cover complete generation calls, excluding tokenization, loading, rewards and training. A real tiny-OPT calibration additionally exposed a missing HF cache capability flag. [Commit ecacbce](deepspeedai@ecacbce) adds `is_compileable=True` to the static-cache container, matching its existing static layers. **Six targeted GPU checks pass**, including new OPT manual/Auto comparisons with padded prompts and refill against the HF oracle, plus existing Qwen bucket/cache-path checks. The three Examples regression tests cover fitting, complete OPT calibration and saved-source compatibility. Both local reviews found no issues in this follow-up. ## Final correctness and performance acceptance [Commit d4def3d](deepspeedai@d4def3d) includes the final fixes: non-default repetition penalties are calculated in FP32 like HF, and unknown public generation switches are rejected before routing. The default-penalty path retains its early return. **Five focused BF16 checks and six unknown-setting CPU cases pass.** The Examples CLI now requires complete metadata for saved calibration files. Both independent correctness and performance reviews returned no findings after these fixes. A new calibration and sweep were run against measured commit `d4def3d`. **Qwen3-32B, BF16, SDPA, one GPU; two EOS-disabled output tokens; one warmup plus three interleaved timed runs per path.** Calibration and held-out evaluation use separate shapes. The calibration file, sweep and that measured revision's rollout/planner/static-cache sources have the same combined SHA256: `f9d79a43d2e6c0e3fef164ef11e81a26c643820beb1a0d3f1a86f41a46dbdd0c`. | Workload | No-CB ms | Equal-width CB ms | Aligned CB ms | Auto ms | Auto / best | Auto path | | --- | ---: | ---: | ---: | ---: | ---: | --- | | 128×16 | **269.83** | 439.91 | 5717.89 | 270.03 | 1.000743× | No-CB | | 1×128 + 31×4 | **496.15** | 540.76 | 1501.57 | **191.96** | 0.386886× | 2 buckets | | 16,32,…,512 | 1962.87 | 2009.18 | **1864.55** | **1336.57** | 0.716828× | 6 buckets | | 1×512 + 31×16 | 1962.56 | 2005.33 | **1575.08** | **288.31** | 0.183045× | 2 buckets | **All four pass the 5% acceptance tolerance; worst measured regression is +0.07428%. Auto output IDs match ordinary generation in every case**, and both routes are exercised. Means and ratios were independently recomputed from raw samples. The legacy aligned spread baseline retains its previously recorded second-token difference on one row; Auto matches no-CB there. Fixed output length prevents this numerical-sensitive baseline difference from changing the amount of timed work. The fitted coefficients are fixed=28.791679 ms, token=0.093362064 ms, attention=0.010090454 ms/GFLOP, KV=0.056288540 ms/MiB and additional decode=17.144541 ms/step. The earlier tables remain historical measurements on their original versions. This final check does not establish long-context, natural-EOS, 256-request E2E or dynamic-arrival performance. Auto remains disabled by default. [Commit a0588b7](deepspeedai@a0588b7) then fixes manual aligned prefill to retain the original padded repetition history. Its focused GPU regression reproduced the mismatch and now passes. Auto already passed this same mapping, and the measured default-penalty paths do not read it; independent correctness and performance delta reviews returned no findings. The table remains attributed to `d4def3d`, with no timings or hashes relabeled as a new-HEAD measurement. The Examples benchmark explicitly requires Transformers >=4.56; the library's existing compatibility scope is unchanged. ## Review invitation cc @nathon-lee @delock — this follows your continuous-batching, profiling, and aligned-cache work in deepspeedai#8368, deepspeedai#8494, and deepspeedai#8497. Could you review the automatic route selection, bucket/refill cache semantics, and cost defaults? ## November refresh Rebased onto `b62afa8` and squashed to one commit; the PR is now mergeable. The refresh keeps deepspeedai#8732 graph state, deepspeedai#8839 last-position prefill logits, deepspeedai#8933 accelerator-neutral graph calls and deepspeedai#8852 `DeepSpeedStaticCache.get_query_offset()`. deepspeedai#8980's explicit accelerator opt-in is respected: a backend without `is_graph_capture_supported()` falls back to eager before any routing decision. *Superseded by 8a9641b:* Auto now combines eager bucketed prefill with fixed-slot graph decode (see the November 11 update). SegKI (deepspeedai#8602) is not part of this PR. Validation, 8×H200-class node with one GPU in use (NVIDIA H20Z, PyTorch 2.9.1+cu128, CUDA 12.8, Transformers 4.57.1): | Check | Result | | --- | --- | | rollout/cache suite | 159 passed, 3 skipped, 1 deselected | | Skips | Three Transformers 5.20-only query-offset cases added by deepspeedai#8852 | | Deselected | Native shared-prefill kernel test; the test host has no CUDA compiler | | Changed-file pre-commit | Passed | Graph replay, resolved-config propagation on both routes (FP32/BF16), DP bucket/refill oracles, Auto/Graph rejection (since replaced by joint routing) and manual graph dispatch all pass. No new performance numbers are claimed: the tables above were measured before deepspeedai#8839/deepspeedai#8732, so refreshed baselines need a separate calibration run if wanted. AI assistance was used for the rebase, conflict resolution, test execution and drafting of this description; the submitting human reviewed the diff and the results above. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: SuperGoodGame <985236470@qq.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
@jinyouzhi Thanks — the GLU epilogue point led to a nice find: the fp32 decomposition wasn't just a parity issue, it was also bandwidth (two big-tensor upcasts per layer). bf16-native F.silu is bitwise-equal to HF and +5.9% at b=64 decode (1667→1765 tok/s, L20/Ada, interleaved A/B) — landed in 23adb53. Routing m>32 through the native op measured -2.9% on Ada (its large-M path splits into row-chunked WMMA calls / the aten composite with the same fp32 round-trip), so the arch note in your comment is exactly right. |
The glu_epilogue fallback path (m > 32 decode and all prefill, where M = batch * prompt_len) computed silu via an fp32 decomposition: two upcasts of the [M, inter] gate/up tensors, five elementwise kernels, and a final downcast. Replacing it with F.silu in bf16 eliminates the cast round-trips and three kernels, and is bitwise-equal to HF's activation (the fp32 formula rounds differently -- max 0.5 at bf16 -- so m<=32, m>32 and prefill previously disagreed with HF and each other). Measured on L20 (Ada), Qwen3.5-4B, b=64 greedy decode, interleaved A/B (medians of 3): 1667.3 -> 1765.5 tok/s (+5.9%, non-overlapping ranges). The native op keeps the small-M fused kernel (m <= 32); on Ada its large-M path splits into row-chunked calls / an aten composite with the same fp32 round-trip, so the eager fallback with this epilogue is the fastest large-M implementation on this arch (1765 vs 1623 via the native dispatcher; Hopper keeps the concat-GEMM path). test_segment_ki_kernels.py (the only oracle consumer; compares with fp32 inputs under tolerance) passes: 25/25 on L20. Numerics note: the fp32 single-rounding formula stays documented in the docstring as the spec semantics; the executable path now matches HF instead of exceeding its precision. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
Eight conflict hunks in hybrid_engine_rollout.py, resolved semantically: - sampling kwargs -> master's _generation_kwargs helper (it builds on our _hf_sampling_kwargs, so our do_sample/temperature semantics carry over) - repetition penalty -> master's padding-aware upgrade (generation_config param, _repetition_penalty_ignores_padding probe, prompt_attention_mask arg); our old staticmethod copies removed to avoid silent double defs - profiling gate -> keep our ungated enable_profiling stats semantics; the same profile object feeds the bucket planner via the new graph signature (adaptive_prefill gates the planner, not the stats) - remaining hunks (signatures, prefill_cache kwargs) -> master verbatim Validated on L20 (cards 1-2, torch 2.15/tf 5.22): rollout v1 6 failures == the same pre-existing transformers-version set (+1 pass); module_inject 203 passed + 64 subtests, 0 failures; decode bench bit-identical (1242.2 tok/s, same tok_sum); TestTpGradNorm 4 passed locally. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
23adb53 to
2603bd3
Compare
HybridEngineRolloutConfig had use_segki and align_decode_fronts inserted before enable_cache_trimming, shifting the legacy positional constructor slots by two -- #8786's test_config_preserves_legacy_positional_arguments caught it (16 silently landing on enable_cache_trimming). align_decode_fronts is an upstream field and returns to its slot; use_segki moves to the end of the dataclass so the positional contract keeps holding. All in-tree construction is keyword-based, so the reorder is behavior-neutral. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
| handshake is still under design. | ||
| """ | ||
| from deepspeed.module_inject.segment_ki import segki_refresh | ||
| segki_refresh(self._rollout_module() if hasattr(self, "_rollout_module") else self.module) |
There was a problem hiding this comment.
Should we call refresh_weights from sync_weights()? It's currently no-op. I think we need to invalidate the cache.
We could also add a test to verify that the updated cache is used.
Review finding (Runguo): the script hardcoded the deployment box's checkout path; the installed deepspeed package is the import target. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Review findings from @runguo-li and @tohtana (all three verified): 1. refresh_weights() read self._rollout_module/self.module, which do not exist -- the rollout holds the model at engine.module. The arena e2e bypassed the method by calling segki_refresh() directly, so the entry point was never exercised. Resolve through the engine (mock-safe for the SimpleNamespace test engines), and state the retained-graph caveat honestly in the docstring (finding 2, design pending). 2. sync_weights() was a no-op: colocated weights are read live so there is nothing to push, but the injected forwards' derived caches go stale after an in-place write. The training-step hook now refreshes, so every sync source invalidates uniformly (tohtana's wiring point). 3. Rollout-path test: inject -> generate -> in-place weight write -> sync_weights -> generate must match a fresh injection bit-for-bit (the arena e2e logic through the public entry point). Replaces test_sync_weights_is_noop, which pinned the old no-op behavior. Validated on L20 (card 4, torch 2.15/tf 5.22): both new tests pass; full test_hybrid_engine_rollout.py 151 passed / 4 skipped. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
…im (a)) Runguo's finding 2: a retained continuous-batching graph captures the derived tensors' addresses (TP>1 cat weights, Ampere/Hopper GLU concat cache, GDN gates). refresh_weights drops those tensors, so a surviving graph would replay freed memory. Parameters are covered by the capture signature (in-place writes keep their addresses), but the derived caches are invisible to it, so nothing else invalidates them. Interim fix (a): refresh_weights releases _continuous_graph_state (synchronized, like shutdown) before dropping caches; the next rollout re-captures. Cost: one re-capture per sync. The end-state fix (b) -- refresh refills derived tensors in place so graphs survive by construction -- is deferred by design; remove this release when it lands. The new test pins the contract on any config (asserts the graph is released after sync, then post-sync generation matches a fresh injection). Red on the previous head (the graph survived the sync); config-dependent note: on Ada/dense/TP=1/small-batch the captured graph holds no derived tensor, which is why the hazard never surfaced there. Validated on L20 (card 4): red -> green; full rollout + module_inject 355 passed / 16 skipped / 0 failed. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
|
Thanks for the update! |
Hi @tohtana thanks for the comments. I'll fix it. In following PRs I'll investigate whether register a refresh hook for each SegKI op is possible. |
segki_refresh looked up the backend handle only through _ki_gdn_op, but GLU-only models carry theirs on the MLP modules as _ki_dual_op — for those, clear_glu_cat_cache() was never called and the process-wide gate|up concat served stale values after an in-place sync (review finding, Masahiro). Any handle reaches the process-wide cache; scan both attributes. New arena test covers the dense-model branch with a GLU-only model (DS_SEGMENT_KI_GLU_TEST_MODEL, default Qwen/Qwen3-0.6B). Signed-off-by: Guokai Ma <guokai.ma@intel.com>
…LUBuilder The CI formatting run flagged the generator/constructor wrapping in ed86967. The new dense-model test also needs to skip when the accelerator registers no FusedGLUBuilder (NotImplementedBuilder — the CPU accelerator): the injection never happens, so there is no backend cache to stale. A CUDA host whose JIT build fails is an environment defect — the test fails loudly there, not skips. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
36d8a2a to
0aeb3db
Compare
create_op_builder returns an instance; .__name__ exists on the class, not the object — the check raised AttributeError on the CPU CI runner. Read the type name instead (verified against the CPU accelerator directly; a real builder instance does not skip). Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Rebase the static-graph reuse onto the segment-KI decode path from deepspeedai#8602. - Keep the DeepSpeedStaticCache, decode inputs and captured graph across generate() calls with the same decode signature (batch, cache length, device, dtype, parameter layout, training mode, autocast, step-kernel use). AutoTP ranks agree on recapture through an all-reduce. - Without segKI, in-place weight updates are replayed without recapture. With segKI, segki_refresh() bumps a generation in the signature and refresh_weights()/sync_weights() release the graphs, because the injected kernels cache tensors derived from the weights. - Size the decode KV cache from the prefill's local KV tensors again, so AutoTP-sharded and narrowed-head models get the right shape. - Apply the repetition penalty on the graph prefill and Python decode paths again; the b=1 step kernel is used only without a penalty. - Give eager attention an additive mask built inside the graph from the boolean decode mask. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>



Summary
Adds a segment-KI framework that accelerates greedy decode in the hybrid-engine rollout by injecting native CUDA kernels into comm-free segments of the model, plus a C++ decode loop that runs the full autoregressive generation with near-zero per-step Python overhead.
Segment detection: structural, not per-model
Detection is attribute-name based: any HF family whose gated MLP uses
gate_proj/up_proj/down_proj(LLaMA, Qwen, Mistral, DeepSeek, Gemma...) or whose GatedDeltaNet block uses thein_proj_{qkv,z,b,a}set is picked up without per-model code. Projections carrying TP collectives are never fused (a defensiveisinstancecheck against the Allreduce layer hierarchy).Co-location by construction
Both GLU (dual-weight GEMV) and GDN (quad-weight GEMV) replacements read the original weight Parameters directly through
autograd.Functions with exact backward, so training and generation share one forward path: zero weight copies, no inject/eject switching between rollout and training. Verified end-to-end by an integration test (below): generate → train one step → generate, output changes, no sync calls.Components
csrc/module_inject/fused_glu.cucsrc/module_inject/decode_loop.custd::functionreplay callable (CUDA graph, eager forward, or future SYCL); argmax and buffer updates run as a private kernel; EOS checked via amortized D2H sync every 16 stepsdeepspeed/module_inject/segment_ki.pyapply_segment_ki)deepspeed/module_inject/kernel_reference.pydeepspeed/ops/module_inject/{fused_glu,decode_loop}.pydeepspeed/utils/static_cache.pydeepspeed/runtime/rollout/hybrid_engine_rollout.pyPerformance
Qwen3.5-4B, RTX 4080 SUPER, greedy, b=1:
Tests
tests/unit/module_inject/test_segment_ki_kernels.py— 11 tests, all passing (executed on RTX 4080 SUPER):kernel_reference.pyoracle on identical inputs (dual_gemv, quad_gemv, gdn_gates, triple_gemv, fused_add_norm, decode_attn)Notes
kernel_reference.pyserves as the executable spec;decode_loop'sstd::functioninterface is backend-agnostic (graph replay or eager forward); SYCL kernels would follow the same op-builder pattern.