ROCm: accelerate GLM 5.3 prefill and reduce MTP overhead on gfx1151 - #1011
Open
kyuz0 wants to merge 6 commits into
Open
ROCm: accelerate GLM 5.3 prefill and reduce MTP overhead on gfx1151#1011kyuz0 wants to merge 6 commits into
kyuz0 wants to merge 6 commits into
Conversation
Appending a prompt prefix or evaluating an external token can leave the private MTP window spanning target-only rows without draft KV. Invalidate draft history and its minimum position so the next seed starts at a populated row; preserve internal speculative evaluation and equal-length sync. Read-only GLM 5.3 diagnostics reproduce three unwritten rows on main and none with the fix. Resident and streamed-expert append/ordinary-gap controls pass with exact live/reset vectors. CUDA, Metal, TP and GLM 5.2 are not hardware-qualified; exact-sampling replacement semantics remain outside this correction.
Admit resident GLM 5.3 topology with 288 experts, top-eight routing and 4096/2048/4096 dimensions to the existing IQ2 gate/up MMQ path. Preserve the previous admission for other models, streaming exclusions and token-tail handling. Production-shape reference/canary checks pass. The isolated candidate improves 4K prefill from 76.61 to 127.58 tok/s; the final combined change passes the official and practical panels. Activation requantization changes arithmetic; quality evidence is model-level, not bit parity.
Route the resident GLM K4096/M8192 Q4 projection through existing MMQ only for 128 to 2048 rows, retaining the original small-prompt path and quality/streaming fallbacks. Production-layout reference and tail/canary checks pass. The isolated candidate improves 4K prefill from 76.61 to 85.08 tok/s. Combined final short100 scores match main exactly and long8 weighted NLL improves 1.96%, with mixed per-case changes.
Assign one wave per pooled key group for the FP16-key, 32-head, dimension-128 GLM 5.3 indexer on resident gfx1151 bulk prefill. Preserve causal and partial-group masking and existing fallbacks. Independent full-output boundary/canary checks pass. Actual ISA uses paired FMAs and the comment acknowledges rounding drift. The isolated candidate improves 8K prefill from 70.91 to 77.91 tok/s; final combined model scoring and practical checks pass.
Quantize two activation rows once and reuse packed weights across their accumulators for twelve production shapes on resident, quality-off gfx1151 GLM 5.3. Preserve GLM 5.2 and other-model dispatch and exclude the slower K128/M8192 shape. This also covers ordinary two-token prefill and appended tails. Full independent references and canaries pass; the fused operator matches the existing prequantized path. Final combined code passes 138 practical checks, 12 vision fact cases and exact-source official panels, with state replay checks intact. Completed coding MTP improves 12.21 to 14.52 tok/s versus main; raw 4K MTP remains slower at 6.25 versus 7.79. Q8 activation arithmetic changes trajectories and acceptance, so this is not a universal MTP speed or distribution-equivalence claim.
|
Tested this independently on my own Strix Halo box (AMD Ryzen AI Max+ 395 / Radeon 8060S, gfx1151, 128GB unified memory, ROCm 10.0 / HIP 7.15) — same GLM-5.3-Flash-Q2.gguf, on top of Speed (
Matches your numbers closely (168.41/2.20x @4k, 156.89/2.46x @16k). Correctness — after getting burned by the non-determinism in #923, I checked this one thoroughly:
No issues found. This one looks solid — nice work on tracking down the invalidation bug from the last round. |
Contributor
Author
|
Thank you for checking and testing this PR! @antirez , whenever you have a second, this looks like a good one to marge. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GLM-5.3-Flash-Q2 prefill improves 2.20–2.46× on Strix Halo (gfx1151), and embedded-MTP decode improves 22.6% on the completed coding example. Ordinary decode is essentially unchanged.
Resident weights, ROCm 10, greedy native benchmark measurements against main
6289c51; MTP uses a benchmark adapter. Rates are tokens/s:Single-run measurements. Coding answers contain 241→267 useful tokens, so throughput is not answer latency. Prefill/ordinary figures precede the final MTP-only commit, which leaves those paths unchanged.
Changes:
Quality evidence:
CUDA comparison used the identical GGUF and 4,096 raw token IDs at the same context frontier. Across all 154,880 logits:
All three select the same top token and top-ten set. Distances are mixed; CUDA also uses approximate arithmetic, so this checks consistency rather than establishing higher precision or quality equivalence.
Limits: gains are workload-specific; a separate raw-text MTP control is 19.3% slower than main MTP. Vision was not rerun after the final MTP-only commit, and its checks grade facts rather than strict formatting. Performance paths are guarded for resident ROCm/gfx1151 GLM 5.3.