Skip to content

feat(rollout): segment-based kernel injection for hybrid-engine greedy decode - #8602

Merged
tohtana merged 59 commits into
masterfrom
gma/qwen_ki_expr
Nov 19, 2026
Merged

tohtana merged 59 commits into
masterfrom
gma/qwen_ki_expr

Conversation

@delock

@delock delock commented Sep 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds a segment-KI framework that accelerates greedy decode in the hybrid-engine rollout by injecting native CUDA kernels into comm-free segments of the model, plus a C++ decode loop that runs the full autoregressive generation with near-zero per-step Python overhead.

Segment detection: structural, not per-model

Detection is attribute-name based: any HF family whose gated MLP uses gate_proj/up_proj/down_proj (LLaMA, Qwen, Mistral, DeepSeek, Gemma...) or whose GatedDeltaNet block uses the in_proj_{qkv,z,b,a} set is picked up without per-model code. Projections carrying TP collectives are never fused (a defensive isinstance check against the Allreduce layer hierarchy).

Co-location by construction

Both GLU (dual-weight GEMV) and GDN (quad-weight GEMV) replacements read the original weight Parameters directly through autograd.Functions with exact backward, so training and generation share one forward path: zero weight copies, no inject/eject switching between rollout and training. Verified end-to-end by an integration test (below): generate → train one step → generate, output changes, no sync calls.

Components

File Role
csrc/module_inject/fused_glu.cu dual/quad-weight GEMV (MLP / GDN input projections), GDN gates, decode attention (warp-per-head, KV split-K, online softmax), fused residual+RMSNorm, triple QKV GEMV, graph-capturable decode step
csrc/module_inject/decode_loop.cu C++ generation loop taking a backend-agnostic std::function replay callable (CUDA graph, eager forward, or future SYCL); argmax and buffer updates run as a private kernel; EOS checked via amortized D2H sync every 16 steps
deepspeed/module_inject/segment_ki.py segment detection + forward replacement (apply_segment_ki)
deepspeed/module_inject/kernel_reference.py pure-PyTorch executable specifications for every kernel — test oracle, runtime fallback, porting reference
deepspeed/ops/module_inject/{fused_glu,decode_loop}.py op builders (JIT compile, lazy load)
deepspeed/utils/static_cache.py hybrid-slot static cache: real static KV layers for full attention, pass-through slot reusing HF's cudagraph-safe GDN state buffers by reference
deepspeed/runtime/rollout/hybrid_engine_rollout.py prefill → cache setup → graph capture → C++ decode loop, with layered fallback (full-step graph for b=1, graph+argmax for b>1, C++ loop, Python loop)

Performance

Qwen3.5-4B, RTX 4080 SUPER, greedy, b=1:

tok/s relative
HF eager 22.5 1.0×
DeepSpeed (this PR) 69.4 3.1×
vLLM 74.9 —

Tests

tests/unit/module_inject/test_segment_ki_kernels.py — 11 tests, all passing (executed on RTX 4080 SUPER):

  • Kernel vs reference consistency (GPU, 6 tests): every kernel compared against its kernel_reference.py oracle on identical inputs (dual_gemv, quad_gemv, gdn_gates, triple_gemv, fused_add_norm, decode_attn)
  • Injection invariants (CPU-runnable, 4 tests): injected forward matches un-injected output, no weight copies installed, composite autograd path, backward scatters gradients to the original Parameters
  • Rollout → train → rollout integration (GPU, Qwen3.5-0.8B-Base): generate with graph capture + segKI, one SGD step through the same injected model, generate again with changed output — no inject/eject/sync calls anywhere in between
  • Multi-GPU (AutoTP) validation (planned follow-up)

Notes

  • SYCL/XPU porting: kernel_reference.py serves as the executable spec; decode_loop's std::function interface is backend-agnostic (graph replay or eager forward); SYCL kernels would follow the same op-builder pattern.
  • Related upstream fix split into a standalone PR: fix: attention_unfused alpha missing norm_factor squaring #8666 (attention_unfused alpha squaring).

@delock
delock force-pushed the gma/qwen_ki_expr branch 2 times, most recently from 11231a7 to d41c914 Compare September 26, 2026 07:00
@delock delock changed the title feat(rollout): HybridEngineRollout graph capture + segment-KI for Qwen3.5 hybrid — 67 tok/s (1.14× of vLLM) feat(rollout): segment-based kernel injection for hybrid-engine greedy decode Sep 26, 2026
@delock
delock marked this pull request as ready for review September 28, 2026 06:27

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b31faddfc4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread csrc/module_inject/decode_loop.cu Resolved
Comment thread deepspeed/module_inject/segment_ki.py Outdated
Comment on lines +184 to +185
from deepspeed.module_inject.kernel_reference import dual_gemv_silu_mul
return self.down_proj(dual_gemv_silu_mul(input, self.gate_proj.weight, self.up_proj.weight))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve the original MLP activation

The structural selector accepts every bias-free module with these three projection names, but this replacement always computes SiLU. A GELU-gated MLP such as Gemma satisfies that selector and will therefore silently produce different logits whenever use_segki=True; constrain detection to SiLU MLPs or retain/call the module's configured activation.

Useful? React with 👍 / 👎.

Comment thread deepspeed/module_inject/segment_ki.py Resolved
Comment thread deepspeed/module_inject/segment_ki.py Resolved
Comment thread deepspeed/module_inject/segment_ki.py Resolved
Comment thread csrc/module_inject/fused_glu.cu Resolved
Comment thread deepspeed/utils/static_cache.py Resolved
Comment thread csrc/module_inject/decode_loop.cu Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Several paths currently break gradients, padded prompts, fallback decoding, hybrid continuous batching, and native-op compilation or packaging.

Review effort: Balanced
Findings: 10 High severity · 6 Medium severity · 2 Low severity

Open (18)
What changed in this PR

Adds segment-based kernel injection and native decode acceleration for hybrid-engine greedy rollout.

Changes:

  • Adds fused CUDA kernels, reference implementations, and op builders.
  • Extends static-cache support for hybrid attention models.
  • Integrates graph capture, native decode loops, documentation, and tests.
File Description
csrc/​module_inject/​fused_glu.cu Implements fused CUDA kernels.
csrc/​module_inject/​decode_loop.cu Implements the native decode loop.
deepspeed/​module_inject/​segment_ki.py Detects and injects fused model segments.
deepspeed/​module_inject/​kernel_reference.py Provides PyTorch kernel references.
deepspeed/​ops/​module_inject/​__init__.py Exports native-op builders.
deepspeed/​ops/​module_inject/​fused_glu.py Adds the fused-kernel builder.
deepspeed/​ops/​module_inject/​decode_loop.py Adds the decode-loop builder.
deepspeed/​utils/​static_cache.py Adds hybrid cache slots and position semantics.
deepspeed/​runtime/​rollout/​hybrid_engine_rollout.py Integrates injection and accelerated decoding.
tests/​unit/​module_inject/​test_segment_ki_kernels.py Tests kernels and injection behavior.
tests/​unit/​utils/​test_static_cache.py Updates static-cache expectations.
docs/​code-docs/​source/​inference-engine.rst Documents segment-based injection.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread csrc/module_inject/decode_loop.cu Resolved
Comment thread deepspeed/module_inject/segment_ki.py Resolved
Comment thread deepspeed/module_inject/segment_ki.py Resolved
Comment thread deepspeed/module_inject/segment_ki.py Resolved
Comment thread deepspeed/module_inject/segment_ki.py Resolved
Comment thread csrc/module_inject/decode_loop.cu Resolved
@nathon-lee

Copy link
Copy Markdown
Contributor

Hi @delock, thanks for submitting this PR. I have left a few comments and questions.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Graph compatibility, packed GDN execution, projection bias handling, and deterministic argmax behavior have unresolved correctness issues.

Review effort: Balanced
Findings: 3 High severity · 1 Medium severity · 2 Low severity

Open (6)
Resolved since last review (18)

Comment thread deepspeed/module_inject/segment_ki.py Resolved
Comment thread deepspeed/module_inject/segment_ki.py Resolved
Comment thread csrc/module_inject/fused_glu.cu Resolved
Comment thread csrc/module_inject/fused_glu.cu Resolved
@delock

delock commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Hi @delock, thanks for submitting this PR. I have left a few comments and questions.

Hi @nathon-lee , kindly reminder in case you didn't submit your review comments. Thanks!

@nathon-lee

Copy link
Copy Markdown
Contributor

Hi, @delock Thanks for the PR and for addressing the earlier comments. I recommend addressing these two points before merging:

  1. _generate_graph() no longer passes cache_position or position_ids. A minimal CUDA test with PyTorch 2.9.1, Transformers 4.57.1, and a tiny LLaMA succeeds with explicit position tensors but fails without them with operation not permitted when stream is capturing. This path also runs with use_segki=False. Please preserve the existing compatibility and add a regression test.
  2. Most of the numerical tests compare the new kernels with the new reference implementations. Please also compare outputs and gradients directly against the original HF forward, with explicit BF16 tolerances. Bit-for-bit agreement is not necessary, but changes such as the intermediate rounding order in fused residual-add + RMSNorm should be evaluated against the original implementation.

A few additional points to check; I am not treating these as confirmed merge blockers:

  • The GDN softplus expression logf(1 + expf(x)) loses precision for negative inputs; for example, the FP32 expression produces 0 at x=-20. Please use a stable implementation and check the effect of changing g from FP32 to BF16.
  • The GDN convolution wrapper only forwards seq_idx, while the scan receives cu_seq_lens_q/cu_seqlens. Please confirm that the supported convolution implementation still receives the packed-sequence boundaries. If packed training is supported, a test comparing outputs and gradients from separate and packed execution would help. Otherwise, please document the limitation.
  • Is reusing a captured graph after a training step intended to be supported? The current test captures a new graph on each generate() call, so it does not cover that case. If graph reuse is out of scope, please clarify this in the documentation.
  • Some native-op loading failures fall back silently. Please log the reason at least once so users can tell which path is running.
  • Please clarify whether throughput counts executed decode steps or valid response tokens, and report capture overhead separately from steady-state decode performance. Please also update the PR description to reflect the removal of the standalone C++ decode loop.

The position-handling regression is reproduced with the versions above. The other points are requests for testing or clarification, not confirmed runtime failures.

@delock

delock commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

Hi @nathon-lee , thanks for the review comments. I have followup them with subsequent commits. There is one finding, fused_add_norm didn't act correctly with BS>1, and it is found by comparing with HF forward.

Hi, @delock Thanks for the PR and for addressing the earlier comments. I recommend addressing these two points before merging:

  1. _generate_graph() no longer passes cache_position or position_ids. A minimal CUDA test with PyTorch 2.9.1, Transformers 4.57.1, and a tiny LLaMA succeeds with explicit position tensors but fails without them with operation not permitted when stream is capturing. This path also runs with use_segki=False. Please preserve the existing compatibility and add a regression test.
  2. Most of the numerical tests compare the new kernels with the new reference implementations. Please also compare outputs and gradients directly against the original HF forward, with explicit BF16 tolerances. Bit-for-bit agreement is not necessary, but changes such as the intermediate rounding order in fused residual-add + RMSNorm should be evaluated against the original implementation.

A few additional points to check; I am not treating these as confirmed merge blockers:

  • The GDN softplus expression logf(1 + expf(x)) loses precision for negative inputs; for example, the FP32 expression produces 0 at x=-20. Please use a stable implementation and check the effect of changing g from FP32 to BF16.
  • The GDN convolution wrapper only forwards seq_idx, while the scan receives cu_seq_lens_q/cu_seqlens. Please confirm that the supported convolution implementation still receives the packed-sequence boundaries. If packed training is supported, a test comparing outputs and gradients from separate and packed execution would help. Otherwise, please document the limitation.
  • Is reusing a captured graph after a training step intended to be supported? The current test captures a new graph on each generate() call, so it does not cover that case. If graph reuse is out of scope, please clarify this in the documentation.
  • Some native-op loading failures fall back silently. Please log the reason at least once so users can tell which path is running.
  • Please clarify whether throughput counts executed decode steps or valid response tokens, and report capture overhead separately from steady-state decode performance. Please also update the PR description to reflect the removal of the standalone C++ decode loop.

The position-handling regression is reproduced with the versions above. The other points are requests for testing or clarification, not confirmed runtime failures.

@nathon-lee

nathon-lee commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Hi @delock, thanks for following up. That was a useful catch: the previous fused_add_norm kernel only indexed row 0, leaving the other output rows unwritten for BS > 1. The per-row indexing fix addresses that, and the multi-row kernel check passes here. Good to see the HF comparison caught it.

I did notice one issue at 6647fb30: static_cache.py:103 calls get_accelerator().is_stream_capturing(), but the accelerator interface and implementations do not define it. The existing per-row cache tests fail with AttributeError on both CPU and CUDA; the LLaMA graph path also fails with use_segki=False. Could you include the missing implementation?

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-22T03:22:42.556857Z
We couldn’t complete this review. Comment @codex review to try again.
cc3e804 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@delock

delock commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

Hi @delock, thanks for following up. That was a useful catch: the previous fused_add_norm kernel only indexed row 0, leaving the other output rows unwritten for BS > 1. The per-row indexing fix addresses that, and the multi-row kernel check passes here. Good to see the HF comparison caught it.

I did notice one issue at 6647fb30: static_cache.py:103 calls get_accelerator().is_stream_capturing(), but the accelerator interface and implementations do not define it. The existing per-row cache tests fail with AttributeError on both CPU and CUDA; the LLaMA graph path also fails with use_segki=False. Could you include the missing implementation?

Hi @nathon-lee , sorry for the late reply, let me check with latest status of master.

@delock

delock commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

Hi @delock, thanks for following up. That was a useful catch: the previous fused_add_norm kernel only indexed row 0, leaving the other output rows unwritten for BS > 1. The per-row indexing fix addresses that, and the multi-row kernel check passes here. Good to see the HF comparison caught it.
I did notice one issue at 6647fb30: static_cache.py:103 calls get_accelerator().is_stream_capturing(), but the accelerator interface and implementations do not define it. The existing per-row cache tests fail with AttributeError on both CPU and CUDA; the LLaMA graph path also fails with use_segki=False. Could you include the missing implementation?

Hi @nathon-lee , sorry for the late reply, let me check with latest status of master.

Hi @nathon-lee , the latest code should contain this missing implementation. Sorry for inconvience.

def total_memory(self, device_index=None):
...

@abc.abstractmethod

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should not remove this?

Comment thread accelerator/abstract_accelerator.py Resolved
@phoeenniixx

Copy link
Copy Markdown
Contributor

The new _ki_prompt_padded check routes every padded batch to HF generate, including with use_segki=False and for non-hybrid models. On master, the graph path handled padded prompts through the attention mask, so this looks like a regression for existing use_graph_capture users. Could the fallback be limited to the cases that need it (segKI kernels and hybrid/GDN models)? Since most rollout batches with different prompts are padded, supporting padding on the segKI path would also matter for real speedups, but that seems fine as a follow-up.

…egrade

Two hardening changes for silent-degradation paths found while validating
on L20:

1. RoPE table install passes [3, 1, L] positions when the model's rotary
   is MRoPE (mrope_section). transformers 5.20/5.22 happen to expand 2-D
   positions internally, but strict MRoPE implementations raise -- and
   the install chain shares one try, so a raise silently skips
   install_decode_attention / install_fused_norm / install_lm_head and
   decode falls back to the HF forward with no signal. Verified a no-op
   on 5.20.2/5.22.1 (install counts and b=32 decode throughput
   unchanged: TP1 1244.1, TP2 1992.6 tok/s vs 1242.8/1993.0 medians).

2. That except now logs a warning with the exception instead of
   swallowing it (review feedback: silent fallbacks must be visible).
   Verified by injecting a simulated failure: warning fires and the
   fallback engages (attn install 0/8, fused_norm 0).

3. Same treatment for the GDN scan: without fla-core every scan (decode
   and prefill) silently runs the eager torch decomposition, measured
   1.6x slower end-to-end on Qwen3.5-4B b=32 decode (1244 -> 645 tok/s).
   The per-layer FLA attribute is legitimately absent on transformers
   5.19+, so the warning probes fla importability instead. Verified both
   ways: fla present -> no warning, 1244.5 tok/s; fla removed ->
   warning + 644.6 tok/s.

Suggested by @jinyouzhi (MRoPE shape and the warning).

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
The amortized check (every 16 steps, from 8b05554) trades a per-step
~50us sync for up to 15 dead replays after the last row finishes. That
wins only in EOS-free decoding (+0.1-0.3% measured at b=32); in
EOS-heavy rollouts -- RL responses routinely end early and
max_new_tokens is only a cap -- the dead replays cost ~20% per call at
b=32 step times. Check every step again (the None fast path and the
accumulating-mask break structure stay).

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
@delock

delock commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator Author

Hi @jinyouzhi thanks for the suggestions. I have corporated some of the suggestions and leave the others for future improvement. For optimizations that requires hook in refresh weight, maybe we should also leave them for future. This may make #9077 depend on this PR and refresh weight hook be implemented along with #9077 . Thanks!

@jinyouzhi

Copy link
Copy Markdown
Contributor

Hi @jinyouzhi thanks for the suggestions. I have corporated some of the suggestions and leave the others for future improvement. For optimizations that requires hook in refresh weight, maybe we should also leave them for future. This may make #9077 depend on this PR and refresh weight hook be implemented along with #9077 . Thanks!

Thanks for incorporating the suggestions! Agreed. I'll study the refresh-weight path more carefully and revisit #9077's design accordingly, then update it after this PR lands.

banxingmjj pushed a commit to openanolis/DeepSpeed that referenced this pull request Nov 15, 2026
…ost model (deepspeedai#8786)

## November 11 update

Rebased onto master `b9100c5`, which includes deepspeedai#8390 and deepspeedai#9008; new head
`82ac56e`.

Uses deepspeedai#8939's shared sampling kwargs, token selector and FP32
repetition-penalty helper. Manual graph and continuous-batching paths
retain upstream temperature/top-k/top-p sampling. Adaptive prefill
retains its single-sample greedy scope, including fixed-slot graph
decode. deepspeedai#9004's pre-built 4-D graph mask is preserved.

- **deepspeedai#8390 early stopping:** ordinary generation and Auto's batched
fallback share deepspeedai#8390's generation kwargs, so they stop on natural EOS;
ZeRO-3 keeps the group-aware synchronized stop. Auto remains
single-process.
- **Review fixes:** the graph path rejects `align_decode_fronts=True`
only when Auto is disabled, because Auto always uses aligned bucketed
prefill. The accelerator graph test covers both flag values. The
documentation states that Auto takes precedence over
`align_decode_fronts` on the graph path; cache trimming is still
rejected.
- **deepspeedai#9008 lazy KV layers:** the bucketed prefill copy initializes
uninitialized cache layers through the shared
`_continuous_cache_layer()` helper, like the equal-width path.
- **Codex P2 (preparer findings):** after a custom
`_prepare_generation_config()` runs, Auto rejects the prepared config if
it changes any request override (token budget, EOS, PAD or greedy
settings). Otherwise the batched route would honor the change inside
`generate()` while the CB routes keep the request values. A preparer
that replaces or edits the attention mask is rejected for the same
reason.
- **Codex P2 (mask lengths):** effective prompt lengths use
`count_nonzero`, so BF16/FP16 masks longer than their exact-integer
range no longer round the bucket width.
- **Codex P1 (test reliability):** the route-planning profile test uses
a frozen clock advanced by the model's generation-config preparer,
instead of wall-clock bounds or mocked rollout internals.
- The decode cost still uses the configured maximum generation budget.
Natural EOS can shorten decode, so the extra-CB estimate can be too
large; the documentation states this assumption.

Validation on an 8 × H200 node, PyTorch 2.9.1+cu128, CUDA 12.8,
Transformers 4.57.1: **232 passed, 3 skipped, 2 deselected** across the
runtime rollout, scheduler, interface, static-cache and v1 accelerator
suites, including deepspeedai#8390's 4-rank subgroup and 2-GPU ZeRO-3
synchronized-stop tests. Skips require the newer cache query-offset API;
deselections are the native shared-prefill kernel and optional
pretrained-model test. Pre-commit passed on all changed files. A small
natural-EOS check (Qwen2.5-3B-Instruct, four short questions) ended
every response at EOS, with Auto matching ordinary generation. This
update used AI assistance.

The performance tables below retain their original revision attribution;
large-model throughput and peak memory were not remeasured for this
rebase.

## Summary

Closes deepspeedai#8785. Related tracking:
[deepspeedai#8197](deepspeedai#8197),
[deepspeedai#8462](deepspeedai#8462).

**Performance follow-up to
[deepspeedai#8497](deepspeedai#8497 Aligned
CB avoids padding on long-tail prompts, but one prefill Forward per
request is expensive for short prompts. A padding-ratio threshold cannot
distinguish **128/4** from **512/16**, despite identical ratios and
different fastest baselines.

This change adds **one opt-in automatic policy** that compares ordinary
no-CB generation with aligned CB using **DP-selected prefill buckets**.
For 1×512 + 31×16, the selected buckets are `[1,512]` and `[31,16]`,
reducing prefill Forwards from 32 to 2.

**Auto is disabled by default.** Existing manual modes and positional
configuration arguments are preserved.

```python
cfg = HybridEngineRolloutConfig(adaptive_prefill=True)
rollout = HybridEngineRollout(engine, tokenizer, cfg)
sampling = SamplingConfig(max_new_tokens=16, temperature=0, continuous_batch_size=32)
```

## Selection and configuration

A length-sorted **O(n²)-time / O(n)-space DP** minimizes the additive
estimated cost of contiguous buckets. It explores one batch, singleton
batches and intermediate partitions without running the GPU for
candidate evaluation. Each bucket uses ordinary left padding and a 2D
attention mask; **no varlen kernel is needed**. Model geometry is read
once per plan.

```text
C(B,L) = fixed + token × B×L + attention × estimated_attention_GFLOPs
         + kv × estimated_KV_MiB
attention_GFLOPs = 4 × B × L² × layers × query_heads × head_dim / 1e9
KV_MiB = 2 × B × L × layers × KV_heads × head_dim × element_bytes / 2^20
```

| Parameter | Default | Meaning |
| --- | ---: | --- |
| `prefill_fixed_cost_ms` | 25.0 | Fixed milliseconds per prefill
Forward |
| `prefill_token_cost_ms` | 0.1 | Milliseconds per padded token
position; projection/MLP term |
| `prefill_attention_cost_ms` | 0.01 | Milliseconds per estimated
attention GFLOP |
| `prefill_kv_cost_ms` | 0.04 | Milliseconds per MiB of KV copied into
the CB cache |
| `continuous_decode_cost_ms` | 10.0 | Additional CB milliseconds per
remaining Decode step |
| `prefill_max_tokens` | 65536 | Maximum padded token positions per
prefill Forward; `None` removes this limit |

All cost coefficients must be finite and non-negative. **These are
starting estimates, not universal GPU constants**; calibrate them for
the model, device, precision and attention backend. Head dimension,
query/KV heads, layers and parameter element size are read
automatically. The DP is optimal for the configured estimates, not
guaranteed globally optimal measured latency.

Ordinary generation is eligible when every row fits
`continuous_batch_size`, the physical input fits the prefill/context
limits, and neither explicit static capacity nor trimming is requested.
Its estimated cost omits CB cache-copy traffic; CB adds
`continuous_decode_cost_ms × (max_new_tokens - 1)`. The first output
token comes from Prefill.

For CB, **every admission/refill replans the newly admitted requests**.
Effective KV spans are copied into the existing right-aligned cache;
surviving rows retain their layout. The policy does not change caller
configuration or active-row bounds. Disabling Auto restores manual
selection. Model/OOM errors propagate without trying another path.

## Correctness

- Apply repetition penalty from the first token using **original padded
history**, even when bucket model inputs are trimmed.
- Preserve generated PAD-valued tokens through the first EOS, excluding
structural padding after termination.
- Include route planning and result normalization in end-to-end
profiling; count **productive output tokens** and retain
active/configured capacity fields.
- Resolve HF's effective generation configuration before rejecting
unsupported processors/output modes, including legacy model-config
refresh; preserve explicit generation-config overrides. Auto requires
single-process greedy rollout, one sample per prompt and cache-class
support. With graph capture, Auto uses eager bucketed prefill and
fixed-slot graph decode; cache trimming and shared prefill remain
unsupported.

## Experiments

Related work: [TurboTransformers, §5 (PPoPP
2021)](https://arxiv.org/html/2010.05680) already uses length sorting,
profiled batch costs and an O(n²) DP to partition variable-length
inference requests. [DynaPipe, §4](https://arxiv.org/html/2311.10418)
applies cost-based DP partitioning to training micro-batches. This PR
applies that established scheduling approach to HybridEngine rollout,
with ordinary-generation comparison, refill planning and existing
aligned-cache semantics; it makes no claim of a new DP algorithm.

**Single GPU; Qwen/Qwen2.5-3B-Instruct; FP32 / SDPA; PyTorch
2.9.1+cu128; Transformers 4.57.1.** Each candidate receives **one warmup
and five interleaved timed runs**, 32 requests and 16 output tokens per
request. CB capacity is 32; trimming is disabled, cache capacity unset,
TF32 disabled. The default cost coefficients above are unchanged across
workloads.

| Workload | Effective prompt lengths | Reason |
| --- | --- | --- |
| Short, similar | 20..13 repeated four times | Normal workload shape
from deepspeedai#8497 |
| Long tail | 1×512 + 31×16 | Long-tail workload from deepspeedai#8497 |
| Short tail | 1×20 + 31×1 | Counterexample to ratio-only routing |
| Moderate tail | 1×128 + 31×4 | Same padding ratio as 512/16, different
best legacy path |
| Spread | 16, 32, …, 512 | General distribution with intermediate
buckets |

Inputs repeat non-special token ID 14990. deepspeedai#8497 did not publish complete
short-prompt sequences, so that case reconstructs its documented shape.
Every timed response matches independent single-request eager
generation; each workload produces **512 effective output tokens**.

GPU-synchronized wall-clock timing includes the full `generate()` call,
including Auto planning and normalization. Model loading and oracle work
are outside timing; profiling is disabled during timed runs. All
reported E2E values are **arithmetic means of five runs**.

| Workload | No-CB (ms) | Equal-width CB (ms) | Aligned CB (ms) | Auto
(ms) | Auto / best | Auto choice |
| --- | ---: | ---: | ---: | ---: | ---: | --- |
| Short, similar | **505.54** | 673.66 | 1549.74 | **501.77** | 0.9925×
| No-CB |
| 1×512 + 31×16 | 2458.46 | 2596.38 | **1682.10** | **840.06** | 0.4994×
| 2 buckets |
| 1×20 + 31×1 | **501.74** | 667.57 | 1331.82 | **497.57** | 0.9917× |
No-CB |
| 1×128 + 31×4 | **897.17** | 1053.26 | 1468.58 | **634.59** | 0.7073× |
2 buckets |
| 16, 32, …, 512 | 2458.13 | 2595.33 | **2337.75** | **1984.86** |
0.8490× | 6 buckets |

**All five workloads meet the 5% acceptance tolerance.** Auto chooses
ordinary generation for both short cases; their small timing differences
are not claimed as a kernel speedup. Relative to the fastest legacy
baseline, mean latency falls by **50.06%** for 1×512 + 31×16, **29.27%**
for 1×128 + 31×4, **15.10%** for 16, 32, …, 512. These improvements come
from fewer prefill Forwards and reduced padded work.

<details>
<summary>Throughput and peak allocated memory for every path</summary>

Throughput is 512 divided by mean latency. Memory is peak PyTorch
allocated memory during timed calls, including model weights.

| Workload | Path | Output tok/s | Peak allocated memory (MiB) |
| --- | --- | ---: | ---: |
| Short, similar | No-CB | 1012.77 | 11988.96 |
| Short, similar | Equal-width CB | 760.03 | 12036.75 |
| Short, similar | Aligned CB | 330.38 | 11928.77 |
| Short, similar | **Auto** | 1020.40 | 11988.96 |
| 1×512 + 31×16 | No-CB | 208.26 | 15558.69 |
| 1×512 + 31×16 | Equal-width CB | 197.20 | 16747.20 |
| 1×512 + 31×16 | Aligned CB | 304.38 | 13278.51 |
| 1×512 + 31×16 | **Auto** | 609.48 | 13278.49 |
| 1×20 + 31×1 | No-CB | 1020.46 | 11994.64 |
| 1×20 + 31×1 | Equal-width CB | 766.96 | 12033.50 |
| 1×20 + 31×1 | Aligned CB | 384.44 | 11925.07 |
| 1×20 + 31×1 | **Auto** | 1029.00 | 11994.64 |
| 1×128 + 31×4 | No-CB | 570.68 | 12742.62 |
| 1×128 + 31×4 | Equal-width CB | 486.11 | 13066.75 |
| 1×128 + 31×4 | Aligned CB | 348.64 | 12221.97 |
| 1×128 + 31×4 | **Auto** | 806.82 | 12221.95 |
| 16, 32, …, 512 | No-CB | 208.29 | 15558.69 |
| 16, 32, …, 512 | Equal-width CB | 197.28 | 16747.20 |
| 16, 32, …, 512 | Aligned CB | 219.01 | 13278.56 |
| 16, 32, …, 512 | **Auto** | 257.95 | 13682.33 |

</details>

<details>
<summary>Separate profiling breakdown</summary>

One additional profiling call per candidate, outside acceptance timing.
These diagnostics are not averages. Cache management is part of the
generation overhead, not an additional independent total.

| Workload | Path | Prefill (ms) | Decode (ms) | Cache management (ms) |
Prefill / Decode Forwards |
| --- | --- | ---: | ---: | ---: | ---: |
| Short, similar | No-CB | 91.01 | 401.30 | N/A | 1 / 15 |
| Short, similar | Equal-width CB | 91.99 | 499.53 | 26.80 | 1 / 15 |
| Short, similar | Aligned CB | 955.52 | 500.30 | 35.25 | 32 / 15 |
| Short, similar | **Auto** | 90.74 | 396.90 | N/A | 1 / 15 |
| 1×512 + 31×16 | No-CB | 1918.22 | 531.89 | N/A | 1 / 15 |
| 1×512 + 31×16 | Equal-width CB | 1919.76 | 595.14 | 27.09 | 1 / 15 |
| 1×512 + 31×16 | Aligned CB | 981.04 | 594.52 | 35.02 | 32 / 15 |
| 1×512 + 31×16 | **Auto** | 153.66 | 592.66 | 31.43 | 2 / 15 |
| 1×20 + 31×1 | No-CB | 91.06 | 402.45 | N/A | 1 / 15 |
| 1×20 + 31×1 | Equal-width CB | 91.84 | 494.84 | 26.88 | 1 / 15 |
| 1×20 + 31×1 | Aligned CB | 743.30 | 491.93 | 34.48 | 32 / 15 |
| 1×20 + 31×1 | **Auto** | 90.62 | 394.78 | N/A | 1 / 15 |
| 1×128 + 31×4 | No-CB | 490.08 | 399.13 | N/A | 1 / 15 |
| 1×128 + 31×4 | Equal-width CB | 490.85 | 482.08 | 27.09 | 1 / 15 |
| 1×128 + 31×4 | Aligned CB | 889.01 | 483.21 | 34.83 | 32 / 15 |
| 1×128 + 31×4 | **Auto** | 63.80 | 485.00 | 31.43 | 2 / 15 |
| 16, 32, …, 512 | No-CB | 1918.03 | 531.64 | N/A | 1 / 15 |
| 16, 32, …, 512 | Equal-width CB | 1918.95 | 594.68 | 27.32 | 1 / 15 |
| 16, 32, …, 512 | Aligned CB | 1648.16 | 593.25 | 35.91 | 32 / 15 |
| 16, 32, …, 512 | **Auto** | 1301.59 | 593.31 | 32.49 | 6 / 15 |

The 128/4 and 512/16 workloads reduce prefill Forwards from 32 on
aligned CB to **2 on Auto**. Spread uses **6 buckets**. All CB paths in
these performance runs have zero trims and zero moved bytes; ordinary
generation has no static-cache statistics. This makes no new
trimming-speedup claim.

</details>

## Validation and limits

A separate microbenchmark measures **planner overhead**, without model
Prefill: for 128/256 requests of 65,536 tokens, CPU DP takes **0.48/1.82
ms** on average with the default per-bucket token limit. It still checks
all 8,256/32,896 candidate segments; over-limit segments skip cost
evaluation. For 32 requests, DP takes 0.089 ms at length512 and 0.040 ms
at length65,536 (0.078 ms with no token limit). GPU mask
validation/length extraction for 32×65,536 takes 0.83 ms separately.
Measurements use ten repeated timing samples after warmup. **These are
not 64k model Prefill timings or the complete Auto-path overhead.**

**The original validation passed 105 related rollout/cache tests**,
including independent exhaustive partition comparisons for DP,
device-backed tiny-Qwen tests for grouped prefill and refill, both cache
APIs, effective generation-config/legacy refresh, repetition history,
EOS/PAD masks, capacity bounds and external-wall-time checks for
profiling. Existing tests cover retirement, compaction and trimming.
Independent mixed-bucket checks also exercised live-survivor refill and
trimming on/off against padded per-request HF generation.

One pre-existing native shared-prefill extension test is excluded
because the host has no CUDA compiler; **all added tests ran**.
Changed-file pre-commit and `git diff --check` pass. Repeated local
review ended with **no actionable findings**.

```sh
python -m pytest -q \
  tests/unit/runtime/rollout/test_hybrid_engine_rollout.py \
  tests/unit/runtime/rollout/test_continuous_batching.py \
  tests/unit/runtime/rollout/test_rollout_interface.py \
  tests/unit/utils/test_static_cache.py \
  -k 'not native_repeat_kv_cache_fp16_reverse_copy'
```

The prefill token limit does **not** bound allocated decode KV memory. A
single prompt over that limit raises an explicit error; there is no
chunked prefill. **64k prompts require a model context limit covering
prompt plus output budget, sufficient KV capacity and temporary
memory.** The measured checkpoint is configured for 32k.
Planner/position-limit boundaries are tested; real 64k inference
performance is not claimed.

Performance evidence uses unsharded HF models behind
`HybridEngineRollout`; the additional calibrated and real-question
checks are listed below. Other model/device configurations, FP16, native
kernel injection, ZeRO/TP training integration and production
arrival/EOS distributions remain unmeasured. Controlled performance runs
do not exercise staggered EOS or trimming; those have correctness
coverage.

No new dependency, kernel, online learner or trimming redesign. Related
foundation: [deepspeedai#8368](deepspeedai#8368);
profiling: [deepspeedai#8494](deepspeedai#8494).

## Reviewer follow-up

[Commit
65a9115](deepspeedai@65a9115)
propagates the resolved generation configuration through continuous
Prefill, Decode and refill, and omits `output_logits` from legacy manual
generation. **27 targeted tests pass**, including six new cases covering
both Auto routes, both cache APIs and manual equal-width/aligned
compatibility. The new tests reproduced four failures before the fix.
The compatibility test uses a strict generation wrapper that rejects the
unsupported keyword.

Calibration and evaluation tooling moved to [DeepSpeedExamples
deepspeedai#1018](deepspeedai/DeepSpeedExamples#1018), with
its fitting regression test and usage documentation. **Merge the
Examples companion before this PR so the documentation link is
available.** The Examples companion includes MHA-head fallback and
validates a source fingerprint covering rollout, planning and
static-cache code. A seeded tiny-model GPU smoke check completed
calibration and all four saved-config sweep workloads with matching
output IDs on every path. Two independent local reviews returned no
findings.

The calibrated sweep from the preceding routing-overhead fix is separate
from the original 16-token experiments above. It uses **two EOS-disabled
output tokens**, SDPA and one GPU, with independent uniform calibration
shapes and one unchanged coefficient set per configuration:

| Configuration | 128×16 | 1×128 + 31×4 | 16..512 | 1×512 + 31×16 |
Worst regression |
| --- | ---: | ---: | ---: | ---: | ---: |
| Tiny Qwen2 / FP32 | 1.0220× | 0.9685× | 0.9539× | 0.9647× | +2.205% |
| Qwen2.5-3B / FP32 | 1.0005× | 0.2345× | 0.7808× | 0.2265× | +0.054% |
| Qwen3-32B / BF16 | 1.0027× | 0.3881× | 0.7189× | 0.1811× | +0.273% |

Values are **Auto E2E / fastest manual baseline E2E**. The two larger
models select no-CB for 128×16 and bucketed CB for the other three
cases; the tiny model selects no-CB throughout. Auto matches ordinary
output IDs in all twelve cases. Target-model means use three interleaved
repetitions after warmup; the tiny test uses ten. [Full sweep details
and calibration
limits](deepspeedai#8786 (comment)).

A separate [Qwen3-32B real-question
check](deepspeedai#8786 (comment))
selected no-CB on both workloads: MATH-500 averaged **31.452 s / 31.393
s** for ordinary/Auto over three runs; ARC-AGI-2 measured **237.560 s /
237.888 s** in one paired run. All eight responses matched. These checks
cover complete generation calls, excluding tokenization, loading,
rewards and training.



A real tiny-OPT calibration additionally exposed a missing HF cache
capability flag. [Commit
ecacbce](deepspeedai@ecacbce)
adds `is_compileable=True` to the static-cache container, matching its
existing static layers. **Six targeted GPU checks pass**, including new
OPT manual/Auto comparisons with padded prompts and refill against the
HF oracle, plus existing Qwen bucket/cache-path checks. The three
Examples regression tests cover fitting, complete OPT calibration and
saved-source compatibility. Both local reviews found no issues in this
follow-up.

## Final correctness and performance acceptance

[Commit
d4def3d](deepspeedai@d4def3d)
includes the final fixes: non-default repetition penalties are
calculated in FP32 like HF, and unknown public generation switches are
rejected before routing. The default-penalty path retains its early
return. **Five focused BF16 checks and six unknown-setting CPU cases
pass.** The Examples CLI now requires complete metadata for saved
calibration files. Both independent correctness and performance reviews
returned no findings after these fixes.

A new calibration and sweep were run against measured commit `d4def3d`.
**Qwen3-32B, BF16, SDPA, one GPU; two EOS-disabled output tokens; one
warmup plus three interleaved timed runs per path.** Calibration and
held-out evaluation use separate shapes. The calibration file, sweep and
that measured revision's rollout/planner/static-cache sources have the
same combined SHA256:
`f9d79a43d2e6c0e3fef164ef11e81a26c643820beb1a0d3f1a86f41a46dbdd0c`.

| Workload | No-CB ms | Equal-width CB ms | Aligned CB ms | Auto ms |
Auto / best | Auto path |
| --- | ---: | ---: | ---: | ---: | ---: | --- |
| 128×16 | **269.83** | 439.91 | 5717.89 | 270.03 | 1.000743× | No-CB |
| 1×128 + 31×4 | **496.15** | 540.76 | 1501.57 | **191.96** | 0.386886×
| 2 buckets |
| 16,32,…,512 | 1962.87 | 2009.18 | **1864.55** | **1336.57** |
0.716828× | 6 buckets |
| 1×512 + 31×16 | 1962.56 | 2005.33 | **1575.08** | **288.31** |
0.183045× | 2 buckets |

**All four pass the 5% acceptance tolerance; worst measured regression
is +0.07428%. Auto output IDs match ordinary generation in every case**,
and both routes are exercised. Means and ratios were independently
recomputed from raw samples. The legacy aligned spread baseline retains
its previously recorded second-token difference on one row; Auto matches
no-CB there. Fixed output length prevents this numerical-sensitive
baseline difference from changing the amount of timed work.

The fitted coefficients are fixed=28.791679 ms, token=0.093362064 ms,
attention=0.010090454 ms/GFLOP, KV=0.056288540 ms/MiB and additional
decode=17.144541 ms/step. The earlier tables remain historical
measurements on their original versions. This final check does not
establish long-context, natural-EOS, 256-request E2E or dynamic-arrival
performance. Auto remains disabled by default.


[Commit
a0588b7](deepspeedai@a0588b7)
then fixes manual aligned prefill to retain the original padded
repetition history. Its focused GPU regression reproduced the mismatch
and now passes. Auto already passed this same mapping, and the measured
default-penalty paths do not read it; independent correctness and
performance delta reviews returned no findings. The table remains
attributed to `d4def3d`, with no timings or hashes relabeled as a
new-HEAD measurement. The Examples benchmark explicitly requires
Transformers >=4.56; the library's existing compatibility scope is
unchanged.

## Review invitation

cc @nathon-lee @delock — this follows your continuous-batching,
profiling, and aligned-cache work in deepspeedai#8368, deepspeedai#8494, and deepspeedai#8497. Could you
review the automatic route selection, bucket/refill cache semantics, and
cost defaults?


## November refresh

Rebased onto `b62afa8` and squashed to one commit; the PR is now
mergeable. The refresh keeps deepspeedai#8732 graph state, deepspeedai#8839 last-position
prefill logits, deepspeedai#8933 accelerator-neutral graph calls and deepspeedai#8852
`DeepSpeedStaticCache.get_query_offset()`. deepspeedai#8980's explicit accelerator
opt-in is respected: a backend without `is_graph_capture_supported()`
falls back to eager before any routing decision.

*Superseded by 8a9641b:* Auto now combines eager bucketed prefill with
fixed-slot graph decode (see the November 11 update). SegKI (deepspeedai#8602) is
not part of this PR.

Validation, 8×H200-class node with one GPU in use (NVIDIA H20Z, PyTorch
2.9.1+cu128, CUDA 12.8, Transformers 4.57.1):

| Check | Result |
| --- | --- |
| rollout/cache suite | 159 passed, 3 skipped, 1 deselected |
| Skips | Three Transformers 5.20-only query-offset cases added by deepspeedai#8852
|
| Deselected | Native shared-prefill kernel test; the test host has no
CUDA compiler |
| Changed-file pre-commit | Passed |

Graph replay, resolved-config propagation on both routes (FP32/BF16), DP
bucket/refill oracles, Auto/Graph rejection (since replaced by joint
routing) and manual graph dispatch all pass. No new performance numbers
are claimed: the tables above were measured before deepspeedai#8839/deepspeedai#8732, so
refreshed baselines need a separate calibration run if wanted.

AI assistance was used for the rebase, conflict resolution, test
execution and drafting of this description; the submitting human
reviewed the diff and the results above.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: SuperGoodGame <985236470@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@delock

delock commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

@jinyouzhi Thanks — the GLU epilogue point led to a nice find: the fp32 decomposition wasn't just a parity issue, it was also bandwidth (two big-tensor upcasts per layer). bf16-native F.silu is bitwise-equal to HF and +5.9% at b=64 decode (1667→1765 tok/s, L20/Ada, interleaved A/B) — landed in 23adb53. Routing m>32 through the native op measured -2.9% on Ada (its large-M path splits into row-chunked WMMA calls / the aten composite with the same fp32 round-trip), so the arch note in your comment is exactly right.

The glu_epilogue fallback path (m > 32 decode and all prefill, where
M = batch * prompt_len) computed silu via an fp32 decomposition: two
upcasts of the [M, inter] gate/up tensors, five elementwise kernels,
and a final downcast. Replacing it with F.silu in bf16 eliminates the
cast round-trips and three kernels, and is bitwise-equal to HF's
activation (the fp32 formula rounds differently -- max 0.5 at bf16 --
so m<=32, m>32 and prefill previously disagreed with HF and each
other).

Measured on L20 (Ada), Qwen3.5-4B, b=64 greedy decode, interleaved A/B
(medians of 3): 1667.3 -> 1765.5 tok/s (+5.9%, non-overlapping
ranges). The native op keeps the small-M fused kernel (m <= 32); on
Ada its large-M path splits into row-chunked calls / an aten composite
with the same fp32 round-trip, so the eager fallback with this
epilogue is the fastest large-M implementation on this arch (1765 vs
1623 via the native dispatcher; Hopper keeps the concat-GEMM path).

test_segment_ki_kernels.py (the only oracle consumer; compares with
fp32 inputs under tolerance) passes: 25/25 on L20.

Numerics note: the fp32 single-rounding formula stays documented in
the docstring as the spec semantics; the executable path now matches
HF instead of exceeding its precision.

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
Eight conflict hunks in hybrid_engine_rollout.py, resolved semantically:
- sampling kwargs -> master's _generation_kwargs helper (it builds on our
  _hf_sampling_kwargs, so our do_sample/temperature semantics carry over)
- repetition penalty -> master's padding-aware upgrade (generation_config
  param, _repetition_penalty_ignores_padding probe, prompt_attention_mask
  arg); our old staticmethod copies removed to avoid silent double defs
- profiling gate -> keep our ungated enable_profiling stats semantics;
  the same profile object feeds the bucket planner via the new graph
  signature (adaptive_prefill gates the planner, not the stats)
- remaining hunks (signatures, prefill_cache kwargs) -> master verbatim

Validated on L20 (cards 1-2, torch 2.15/tf 5.22): rollout v1 6 failures
== the same pre-existing transformers-version set (+1 pass); module_inject
203 passed + 64 subtests, 0 failures; decode bench bit-identical
(1242.2 tok/s, same tok_sum); TestTpGradNorm 4 passed locally.

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
HybridEngineRolloutConfig had use_segki and align_decode_fronts inserted
before enable_cache_trimming, shifting the legacy positional constructor
slots by two -- #8786's test_config_preserves_legacy_positional_arguments
caught it (16 silently landing on enable_cache_trimming). align_decode_fronts
is an upstream field and returns to its slot; use_segki moves to the end of
the dataclass so the positional contract keeps holding. All in-tree
construction is keyword-based, so the reorder is behavior-neutral.

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>

@tohtana tohtana left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @delock! for the great improvement!
I found one suspicious behavior. Can you check it?

handshake is still under design.
"""
from deepspeed.module_inject.segment_ki import segki_refresh
segki_refresh(self._rollout_module() if hasattr(self, "_rollout_module") else self.module)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we call refresh_weights from sync_weights()? It's currently no-op. I think we need to invalidate the cache.
We could also add a test to verify that the updated cache is used.

delock and others added 3 commits November 17, 2026 09:28
Review finding (Runguo): the script hardcoded the deployment box's
checkout path; the installed deepspeed package is the import target.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Review findings from @runguo-li and @tohtana (all three verified):

1. refresh_weights() read self._rollout_module/self.module, which do not
   exist -- the rollout holds the model at engine.module. The arena e2e
   bypassed the method by calling segki_refresh() directly, so the entry
   point was never exercised. Resolve through the engine (mock-safe for
   the SimpleNamespace test engines), and state the retained-graph
   caveat honestly in the docstring (finding 2, design pending).

2. sync_weights() was a no-op: colocated weights are read live so there
   is nothing to push, but the injected forwards' derived caches go
   stale after an in-place write. The training-step hook now refreshes,
   so every sync source invalidates uniformly (tohtana's wiring point).

3. Rollout-path test: inject -> generate -> in-place weight write ->
   sync_weights -> generate must match a fresh injection bit-for-bit
   (the arena e2e logic through the public entry point). Replaces
   test_sync_weights_is_noop, which pinned the old no-op behavior.

Validated on L20 (card 4, torch 2.15/tf 5.22): both new tests pass;
full test_hybrid_engine_rollout.py 151 passed / 4 skipped.

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
…im (a))

Runguo's finding 2: a retained continuous-batching graph captures the
derived tensors' addresses (TP>1 cat weights, Ampere/Hopper GLU concat
cache, GDN gates). refresh_weights drops those tensors, so a surviving
graph would replay freed memory. Parameters are covered by the capture
signature (in-place writes keep their addresses), but the derived
caches are invisible to it, so nothing else invalidates them.

Interim fix (a): refresh_weights releases _continuous_graph_state
(synchronized, like shutdown) before dropping caches; the next rollout
re-captures. Cost: one re-capture per sync. The end-state fix (b) --
refresh refills derived tensors in place so graphs survive by
construction -- is deferred by design; remove this release when it
lands.

The new test pins the contract on any config (asserts the graph is
released after sync, then post-sync generation matches a fresh
injection). Red on the previous head (the graph survived the sync);
config-dependent note: on Ada/dense/TP=1/small-batch the captured graph
holds no derived tensor, which is why the hazard never surfaced there.

Validated on L20 (card 4): red -> green; full rollout + module_inject
355 passed / 16 skipped / 0 failed.

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
@delock

delock commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Hi @tohtana refresh_weights is now hooked into sync_weights(). Thanks for catching this.

@LiRunGuo For contiguous batching, I forced graph release after weight sync. Survive graph after weight sync can be left to further optimizations. Thanks!

@tohtana

tohtana commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the update!
I noticed one remaining case in segki_refresh(): it looks for the backend only through _ki_gdn_op, while GLU-only models store it in _ki_dual_op. For those models, clear_glu_cat_cache() is never called. This appears to leave the cached gate/up concatenation stale.

@delock

delock commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the update! I noticed one remaining case in segki_refresh(): it looks for the backend only through _ki_gdn_op, while GLU-only models store it in _ki_dual_op. For those models, clear_glu_cat_cache() is never called. This appears to leave the cached gate/up concatenation stale.

Hi @tohtana thanks for the comments. I'll fix it. In following PRs I'll investigate whether register a refresh hook for each SegKI op is possible.

segki_refresh looked up the backend handle only through _ki_gdn_op, but
GLU-only models carry theirs on the MLP modules as _ki_dual_op — for
those, clear_glu_cat_cache() was never called and the process-wide
gate|up concat served stale values after an in-place sync (review
finding, Masahiro). Any handle reaches the process-wide cache; scan
both attributes. New arena test covers the dense-model branch with a
GLU-only model (DS_SEGMENT_KI_GLU_TEST_MODEL, default Qwen/Qwen3-0.6B).

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
…LUBuilder

The CI formatting run flagged the generator/constructor wrapping in
ed86967. The new dense-model test also needs to skip when the
accelerator registers no FusedGLUBuilder (NotImplementedBuilder — the
CPU accelerator): the injection never happens, so there is no backend
cache to stale. A CUDA host whose JIT build fails is an environment
defect — the test fails loudly there, not skips.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
create_op_builder returns an instance; .__name__ exists on the class,
not the object — the check raised AttributeError on the CPU CI runner.
Read the type name instead (verified against the CPU accelerator
directly; a real builder instance does not skip).

Signed-off-by: Guokai Ma <guokai.ma@intel.com>

@tohtana tohtana left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @delock, it looks good to me!

@tohtana
tohtana added this pull request to the merge queue Nov 19, 2026
Merged via the queue into master with commit 97aa3c1 Nov 19, 2026
20 checks passed
@tohtana
tohtana deleted the gma/qwen_ki_expr branch November 19, 2026 04:09
jinyouzhi added a commit to jinyouzhi/DeepSpeed that referenced this pull request Nov 19, 2026
Rebase the static-graph reuse onto the segment-KI decode path from deepspeedai#8602.

- Keep the DeepSpeedStaticCache, decode inputs and captured graph across
  generate() calls with the same decode signature (batch, cache length,
  device, dtype, parameter layout, training mode, autocast, step-kernel
  use). AutoTP ranks agree on recapture through an all-reduce.
- Without segKI, in-place weight updates are replayed without recapture.
  With segKI, segki_refresh() bumps a generation in the signature and
  refresh_weights()/sync_weights() release the graphs, because the
  injected kernels cache tensors derived from the weights.
- Size the decode KV cache from the prefill's local KV tensors again, so
  AutoTP-sharded and narrowed-head models get the right shape.
- Apply the repetition penalty on the graph prefill and Python decode
  paths again; the b=1 step kernel is used only without a penalty.
- Give eager attention an additive mask built inside the graph from the
  boolean decode mask.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants