Repository navigation
[muse_spark][perf] Split the packed qkv inside the q/k region - #5562
Draft
felipemello1 wants to merge 1 commit into
Draft
felipemello1 wants to merge 1 commit into
felipemello1 wants to merge 1 commit into
Conversation
QKVLinear split wqkv's output and made three contiguous copies in eager before Muse Spark's q/k region. The split now lives in QKVLinear.split, and Muse Spark passes the packed qkv into its region, so the copies fuse into the region's kernels. Other models call forward = wqkv + split and run the same ops as before. Muse Spark 4-layer unit, GB300, CUDA graphs: decode -0.9% to -2.4% (T=1..256); train 16k -0.14% by kernel time (glue 1.69 -> 1.21 ms per unit; wall rows are min of 2 with ~0.5 ms noise). Outputs and grads bitwise identical.
felipemello1
added this pull request to stack #5499
October 20, 2026 08:59
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR: Muse Spark's
muse_spark_attention_qkregion now takes the packedwqkvoutput and splits it itself. The q/k copies fuse into the per-head norms, and the backward writes the packed grad without a cat. One 4-layer unit: decode -0.9% to -2.4%, prefill -0.4% to -1.1%, train 16k -0.14% by kernel time (wall rows within noise). Outputs and grads are bitwise identical.Problem
QKVLinear.wqkvreturns q, k and v as one tensor, viewed as[T, 16 kv heads, 10, 64]: per kv head, 8 q heads + 1 k + 1 v.QKVLinear.forwardsplits it and copies q, k and v out (q's reshape can't be a view; vLLM needs contiguous k and v).muse_spark_attention_qkthen reads the copied q and k for the norms and RoPE.Solution
MuseSparkAttentioncallsqkv_linear.wqkv(x)and passes the packed tensor to the region, now_prepare_query_key_value, which returns q, k and v.QKVLinear.split. The split moves out ofQKVLinear.forwardinto its own method, with the samelocal_mapoutput types; the region calls it.wqkvnow runs in global SPMD mode, as Muse'soutput_gatealready does (SPMD typecheck fwd+bwd passes before and after for Muse Spark, Muse Glimmer, Llama 3 and GPT-OSS at TP=2 with and without SP, and Muse Spark and Llama 3 at CP=2).QKVLinear.forwardruns the same ops for Llama 3, Qwen3, GPT-OSS and Muse Glimmer; q, k and v keep their contiguous layouts; the region's options and graph count.Results
You may ask: why not Llama 3, Qwen3 and GPT-OSS too? Their RoPE region is the RoPE module's own forward, which takes q and k. Fusing the split needs a new q/k region in
GQAttention(split -> norm -> RoPE): a follow-up. The copies and cat cost Qwen3 8B 127 us per layer at 16k.You may ask: why is training's q norm kernel slower (114 vs 76 us per layer at 16k)? It also writes the split q, which the norm backward needs; before, the eager copy provided that tensor.
NOTE: reaches every Muse Spark recipe (the region is on by default): the training recipes and the RL DAPO-math recipe (Dist-MoE experts, so the microseconds carry over, the percentages don't).
NOTE: not tested: loss curve, multi-GPU training runs.
Test plan
pytest tests/unit_tests/gpu/models/muse_spark/test_attention_compile.py(the q/k region now takes packed qkv; checks q, k, v and the packed grad against eager)pytest tests/unit_tests/cpu/models/common/test_qkv_linear.pypytest tests/unit_tests/gpu/models/common/test_async_linear.py(2 GPUs; the asyncwqkvnow runs under global type checking)SPMD typecheck on a 2-rank CPU mesh, fwd+bwd, before and after (one-off checks): Muse Spark (RoPE and NoPE layers), Muse Glimmer, Llama 3 and GPT-OSS (bias) at TP=2 with and without SP; Muse Spark and Llama 3 at CP=2.