AProjQ4: imatrix-guided Q4_K attention and GPU runtime optimizations - #952
AProjQ4: imatrix-guided Q4_K attention and GPU runtime optimizations#952GiorgioOppo wants to merge 50 commits into
Conversation
|
I would like to thank @adamlawi, @iammac2, @trueimage, @kyuz0, @evandhoffman and @OPS-NeoRetro for their help. |
|
Third-party Metal validation of this branch, for the record — no action needed on this PR. Measured on an M5 Max 128 GB at
One caveat worth having if numbers from different people are compared: within a session the ratio narrows from ~1.19 (first rep, cold) to ~1.13 (third), because q4 loses more to drift than q8 (−10.0% vs −4.1%). A cold single-rep run reporting ~1.19 is consistent with these results, not a disagreement. Die temperature was logged throughout and the decline is not thermal — the machine peaks at 78 °C after four minutes and then runs cooler while throughput keeps falling. Full method, per-rep tables, the complete temperature log and raw CSVs: |
|
@GiorgioOppo You asked in #621 whether our extended-range CUDA result was noise or solid. Answering here, since this is where the work continues, and carrying over the two findings from that thread that are still useful — one of them for anyone benchmarking this branch, not just for us. Noise or solid: it is solid, and it is negligible. Those are different questions.Statistically solid. 10 measurement windows over 2 hours, 20 paired samples per context point, 160 runs, zero failures, on GB10 / sm_121 at
All four intervals exclude zero (3.2–5.1σ), and the series had converged: the tenth window moved every mean by ≤0.02 pp. It is not noise. Practically negligible. The effects are two to three tenths of a percent. Nobody will feel that, and no decision about this PR should turn on it. If you need one sentence for the PR: on CUDA/GB10, Q4 and Q8 prefill are at parity from 8k to 64k, with Q4 marginally ahead at 8k and marginally behind above it. The honest framing is that we can now resolve an effect too small to matter — which is a fine place to end up, and better than the earlier −1.0% figure that turned out to be an artefact of our own method. The finding worth carrying into this PR: measurement order biases the comparisonOn this box, throughput declines monotonically over a measurement window. In two windows, all 24 arm-to-arm comparisons drifted downward. If the arms always run in the same order, whichever model is loaded later is penalised. Measured directly:
The bias is larger than three of the four effects being measured. At 32768 the sign of the result depends only on which model you load first. This is not thermal on our side: SM clock held 2405 MHz for the full two hours, no throttling flags, 31–61 °C. @evandhoffman reports something consistent above — his ratio narrows from ~1.19 on the first rep to ~1.13 by the third within a session. Same class of effect, different platform. The fix is free: alternate the order between rounds ( What carried over from #621 and still holds at
|
can you also post the decode speed? |
|
@GiorgioOppo Decode numbers below. One caveat first, because it matters for how much weight to put on them: these are not from the balanced series. That series measured pure prefill ( Decode, steady-state t/s, GB10 / sm_121,
Q4 decode leads Q8 by roughly 14% from 4k to 8k, and by about 17.5% at 2048. Two reasons to trust this despite the weaker design:
Scope limits, stated plainly: ctx only up to 8192 (these windows predate the extended sweep), prompt file was the 135 kB one rather than the 405 kB used later, and the figures are from So, combining with the prefill series: on CUDA/GB10, Q4 is at parity with Q8 on prefill and roughly 14% ahead on decode. That matches what @evandhoffman reports on Metal, where the advantage is also decode-only ( If you want decode measured with the balanced design — alternating order, paired ratios, confidence intervals like the prefill table — say so and I will run it. It is about 40 minutes of windows and would give decode the same statistical footing as the prefill numbers. |
Throughput declines across a measurement window, so a fixed arm order penalises whichever arm always runs last. @adamlawi measured this on antirez/ds4#952: the positional bias was larger than three of the four effects being compared, and at one frontier the SIGN of the result depended only on which model loaded first. Their figures are quoted as reported and are not verified here -- the reason to act is the method. Three places, matching the issue's three items. **The suite runner** ran `backends.items()` in the same order every trial, so one backend was always last. `trial_order` now reverses on even trials. Extracted rather than left inline in main(), so the property that matters -- no backend holds the last position in every trial -- is a test rather than a comment. **decode_ab.sh** had the same fixed order. decode_ab_engine.sh has alternated since it was written; this script predates the finding. It now alternates and appends a run-order.txt beside the CSVs. **The row** records `run_position` and `run_arms`. Alternating removes the bias going forward; it does nothing for the 979 rows already written, and those cannot be retro-corrected for a bias nobody wrote down. Absent means the order was not recorded -- deliberately not defaulted to 1, which would claim every existing row ran first, which is the bias being looked for. Neither field is in REQUIRED, for the same reason as client_version: every existing row predates them and `validate` runs on read. With an odd trial count the split is uneven -- 2 of 3 in first position, not 1.5 -- which is better than 3 of 3 and is why the position is worth recording rather than assumed to cancel. 629 passed, 3 skipped. The 5 ruff findings in run.py predate this change.
… retyped Both questions came up five or six times today and each time I answered them with a throwaway heredoc. One of those retypings computed a ratio of medians instead of a median of ratios -- the exact defect the same session was correcting in this file. A committed function cannot make that mistake twice. `per_rep_ratio` answers "does the paired ratio narrow within a session", which we told antirez/ds4#952 that it does. `per_arm_drift` answers "does one arm lose more than the other", which we told #952 is q4 over q8. Both are now one command, with the reasoning in the docstrings and the pairing pinned by tests -- including one where an arm halves across reps while the other stays flat, which a ratio of medians would get wrong. Ad-hoc analysis also leaves a claim with nothing behind it: the number reaches an issue and the derivation does not. Verified against the #118 data, printed rather than transcribed: paired ratio by rep: rep1=1.157 rep2=1.202 rep3=1.162 drift first->last rep, per arm: main=-7.5% pr964=-6.9% 668 passed, 3 skipped.
…was wrong Re-measurement of what we posted to antirez/ds4#952, prompted by an audit that turned out to be checking the wrong engine. decode q4/q8 1.158 1.139 1.152 1.159 median 1.155, spread 1.5 pp prefill q4/q8 1.006 1.022 0.991 0.992 median 0.999, spread 3.1 pp The posted figures were 1.155 and 1.003. They reproduce. Method: ~/git/ds4-pr621 at 6a20b13 -- the commit the original comment names -- with both GGUFs re-downloaded (#135) and the q4 SHA-256 verified against the 413cf0a6...c767 recorded in RESULTS.md, because a replication against a different file is not one. 32 frontiers, 3 reps, arm order alternating, ratios paired within each rep, run lock held throughout, fans on auto and never set. Machine state captured at the start and end of every run. **The audit on #91 was wrong and is corrected there.** It called the within-session narrowing "refuted" and the drift mechanism "inverted", having tested both against #52's August datasets at engine 2669a8e -- 226 commits from the engine the claim was made about. At 6a20b13 the narrowing appears in 3 of 4 runs and q4 loses ground to q8 in the same 3. The engine difference was recorded in RESULTS.md two lines above the numbers I quoted. What does still need correcting upstream is narrower than the audit claimed: the "3-rep paired median" label (the statistic divided two independent medians, and landed on the right answer by luck), and "reproduces the 2026-08-30 figure of 1.155" (those passes give 1.157 and 1.141 paired, so close rather than exact). A draft is on #91 awaiting review; nothing has been sent to antirez/ds4. For #136: this A/B carries 1.5 pp of between-run spread against #118's 4.7 pp, same machine and harness -- so #118's run-2 outlier is a property of that comparison, not of the instrument. 32 files. 684 passed, 3 skipped.
|
Correcting my comment above, and re-measuring it. The statistic was mislabelled. I called it a "3-rep paired median". It was not paired — our script took each arm's median independently and divided them, so the two medians could come from different repetitions. With 7–10% drift inside a session, that re-admits the drift as noise. Fixed on our side; the ratio is now taken within each repetition. Re-measured, because a wrong statistic landing on the right answer is luck rather than evidence. Four fresh runs, same M5 Max, same commit
1.155 stands, and prefill remains indistinguishable from parity. The figures were right; the method that produced them was not, and that seemed worth saying while they are being cited. The within-session narrowing is directionally right but less steady than I wrote. I said the ratio narrows ~1.19 → ~1.13 because q4 loses more to drift than q8, quoting −10.0% against −4.1%. Across four runs:
Three of four narrow, and the same three show q4 losing ground. In runs 2 and 3 the gap opens because q8 speeds up, not because q4 slows — a different mechanism from the one I described. Anyone comparing this against their own drift measurements should take the direction rather than the size: on the same arm the magnitude ranges from −0.7% to −9.3%. One smaller correction. I wrote that this "reproduces the 2026-08-30 figure of 1.155 across 226 commits". Recomputed with the paired statistic those August passes give 1.157 and 1.141 — close, not exact to three decimals. Full method, per-run and per-rep tables, and the raw CSVs for all four runs: evanwtf/local-llm#91 |
|
@GiorgioOppo Heads-up in case you do not see it in your own repo: this PR's head does not link on CUDA, and I have opened GiorgioOppo#12 with the fix.
All five binaries fail — Two practical consequences while it is unmerged:
Worth noting the same defect is still present upstream: |
|
@GiorgioOppo Decode, measured properly this time — balanced design, on the clean head Decode, Q4 vs Q85 windows, 10 paired samples per context, 80 runs, zero failures. Arms adjacent in time, order alternated between rounds (
83–105σ. This confirms the earlier fixed-order figures almost exactly (+17.5/+14.1/+14.2/+14.1), which is what I expected: at a 14–17% effect the positional bias is irrelevant. Here it measured 0.15–0.48 pp, in line with before. Q4 decode leads Q8 by ~14% from 4k to 8k, and ~17% at 2048. That is the solid, large result in this PR on CUDA. Prefill came along in the same runs — and it resolves a discrepancy I flagged earlier
That looks inconsistent with the extended sweep, where I reported prefill parity (−0.34…+0.33%). It is not. Lining up every prefill measurement I have at ctx 8192:
The benchmark prompt decides it, not the commit and not I do not have an explanation yet — plausibly cache residency or how the longer input distributes across prefill chunks. What it means practically is that "does Q4 beat Q8 on prefill" is not well-posed without stating the prompt, and my own earlier posts in #621 quoted both kinds of number without noticing. The extended sweep used the long prompt throughout, so its internal comparisons stand; it is the cross-run comparisons that were mixing two regimes. If you want a specific prompt treated as canonical for this PR, say which and I will re-run everything against it. Conditions: SM clock 2405 MHz throughout, 45–52 °C, no throttling. Raw CSVs with per-run clock and temperature kept on our side. |
|
@GiorgioOppo Correction to my last comment: the prompt file is not the explanation. I checked, and it cannot be. The two files I had been treating as different inputs are not different: The "long" file is the short one repeated three times. At What the comparison actually isThe two windows overlap at exactly one context point,
Arm ordering does not explain the gap, and the direction is worth noting: the earlier window used a fixed arm order, which we measured to be biased in Q4's favour by 0.38–0.55 pp — and it still shows the smaller lead. The bias works against the observation, so it can shrink it, not manufacture it. That leaves exactly two variables: the commit and Which one it is matters differently for you:
A 2×2×2 run is on the machine now — commit × Separate finding, useful to anyone comparing sweepsAbove So a On the ROCm prefill@iammac2 — this bears on your −2%. My earlier "try a different prompt" suggestion is withdrawn, since the file cannot matter. The live hypothesis is now the flags: if |
OPS-NeoRetro
left a comment
OPS-NeoRetro
left a comment
There was a problem hiding this comment.
I think we shouldn't rename this PR @GiorgioOppo, but please squash this PR into one commit after the fixes are applied and before @antirez reviews this...
| if (sel) { | ||
| uint8_t seen[1024] = {0}; | ||
| uint32_t distinct = 0; | ||
| for (uint32_t r = 0; r < eval_rows; r++) | ||
| for (uint32_t k = 0; k < DS4_N_EXPERT_USED; k++) { | ||
| const int32_t id = sel[r * DS4_N_EXPERT_USED + k]; | ||
| if (id >= 0 && id < 1024 && !seen[id]) { seen[id] = 1; distinct++; } | ||
| } | ||
| fprintf(stderr, "ds4: DSpark verify block: %u rows, %u distinct experts of %u slots (last layer)\n", | ||
| eval_rows, distinct, eval_rows * DS4_N_EXPERT_USED); |
There was a problem hiding this comment.
I can't seem to find where n_tokens is defined, so replacing n_tokens to eval_rows is considered a fix
| gpu() { | ||
| if command -v nvidia-smi >/dev/null 2>&1; then | ||
| nvidia-smi --query-gpu=clocks.sm,temperature.gpu --format=csv,noheader,nounits 2>/dev/null \ | ||
| | head -1 | tr -d ' ' | ||
| else | ||
| echo "," | ||
| fi | ||
| } |
There was a problem hiding this comment.
Did you forgot about rocm-smi and other stuff?
There was a problem hiding this comment.
It's difficult to test and support hardware I don't own.
There was a problem hiding this comment.
Okay, @adamlawi and @kyuz0 can make a PR then.
EDIT: @GiorgioOppo, you use a Mac. Why you don't use systemctl then and some string manipulation?
| rm -f speed-bench/metal_iq2_moe_top8_pair_bench | ||
| rm -f ds4 ds4-server ds4-bench ds4-eval ds4-agent ds4_cpu ds4_native ds4_server_test ds4_test ds4_agent_test gguf-tools/quality-testing/score_official gguf-tools/quality-testing/score_official.o speed-bench/metal_decode_schedule_bench speed-bench/metal_prefill_variant_bench speed-bench/metal_q4_dense_pair_bench speed-bench/metal_q4_prefill_pair_bench speed-bench/metal_q4_mm_tail_cull_bench speed-bench/metal_q4_attn_out_a_direct_bench speed-bench/metal_iq2_moe_tail_cull_bench speed-bench/gpu_iq2_moe_prefill_bench_rocm speed-bench/gpu_iq2_moe_prefill_bench_cuda speed-bench/rocm_q4_prefill_bench speed-bench/cuda_q4_prefill_bench speed-bench/*.o tests/test_q4k_dot tests/test_mxfp4_dot tests/test_quantizer_indexer_q4 tests/test_mxfp4_metal tests/test_mxfp4_rocm tests/bench_mxfp4_rocm tests/test_mxfp4_cuda tests/test_rocm_q4_dense_pair tests/test_metal_session_batch tests/test_metal_q4_streams tests/test_metal_q4_prefill_pair tests/test_metal_indexer_q4 tests/test_metal_q4_attn_exactn tests/test_metal_q4_attn_out_a_direct tests/test_metal_q4_qb_f16_cache tests/test_metal_exactn_oracle tests/test_metal_dspark_capture tests/test_metal_argmax_top1 tests/test_metal_iq2_midonly tests/test_metal_iq2_ssd_grouped_mm tests/test_metal_iq2_live_index tests/test_glm53_kda tests/test_glm53_kda_rocm tests/test_glm53_vision_engine tests/test_glm53_vision_prompt tests/test_deepseek4_vision_image tests/test_gpu_xdev tests/test_gpu_model_cache tests/test_gpu_lookup_cache_strict tests/test_engine_mgpu_refusal tests/test_engine_mgpu_runtime tests/test_engine_correctness tests/test_sampling tests/test_cuda_session_batch tests/test_cuda_mixed_batch tests/*.o *.o cuda/mmq/*.o cuda/mmq/test/*.o tests/cuda_long_context_smoke tests/cuda_long_context_smoke.o | ||
| rm -f ds4 ds4-server ds4-bench ds4-eval ds4-agent ds4_cpu ds4_native ds4_server_test ds4_test ds4_agent_test gguf-tools/quality-testing/score_official gguf-tools/quality-testing/score_official.o speed-bench/metal_decode_schedule_bench speed-bench/metal_prefill_variant_bench speed-bench/metal_q4_dense_pair_bench speed-bench/metal_q4_prefill_pair_bench speed-bench/metal_q4_mm_tail_cull_bench speed-bench/metal_q4_attn_out_a_direct_bench speed-bench/metal_iq2_moe_tail_cull_bench speed-bench/gpu_iq2_moe_prefill_bench_rocm speed-bench/gpu_iq2_moe_prefill_bench_cuda speed-bench/rocm_q4_prefill_bench speed-bench/cuda_q4_prefill_bench speed-bench/*.o tests/test_q4k_dot tests/test_mxfp4_dot tests/test_quantizer_indexer_q4 tests/test_mxfp4_metal tests/test_mxfp4_rocm tests/bench_mxfp4_rocm tests/test_mxfp4_cuda tests/test_rocm_q4_dense_pair tests/test_metal_session_batch tests/test_metal_q4_streams tests/test_metal_q4_prefill_pair tests/test_metal_indexer_q4 tests/test_metal_q4_attn_exactn tests/test_metal_q4_attn_out_a_direct tests/test_metal_q4_qb_f16_cache tests/test_metal_exactn_oracle tests/test_metal_dspark_capture tests/test_metal_argmax_top1 tests/test_metal_iq2_midonly tests/test_metal_iq2_ssd_grouped_mm tests/test_metal_iq2_live_index tests/test_glm53_kda tests/test_glm53_kda_rocm tests/test_glm53_vision_engine tests/test_glm53_vision_prompt tests/test_deepseek4_vision_image tests/test_prompt_prefix tests/test_gpu_xdev tests/test_gpu_model_cache tests/test_gpu_lookup_cache_strict tests/test_engine_mgpu_refusal tests/test_engine_mgpu_runtime tests/test_engine_correctness tests/test_sampling tests/test_cuda_session_batch tests/test_cuda_mixed_batch tests/*.o *.o cuda/mmq/*.o cuda/mmq/test/*.o tests/cuda_long_context_smoke tests/cuda_long_context_smoke.o |
There was a problem hiding this comment.
We can clean this up into:
| rm -f speed-bench/metal_iq2_moe_top8_pair_bench | |
| rm -f ds4 ds4-server ds4-bench ds4-eval ds4-agent ds4_cpu ds4_native ds4_server_test ds4_test ds4_agent_test gguf-tools/quality-testing/score_official gguf-tools/quality-testing/score_official.o speed-bench/metal_decode_schedule_bench speed-bench/metal_prefill_variant_bench speed-bench/metal_q4_dense_pair_bench speed-bench/metal_q4_prefill_pair_bench speed-bench/metal_q4_mm_tail_cull_bench speed-bench/metal_q4_attn_out_a_direct_bench speed-bench/metal_iq2_moe_tail_cull_bench speed-bench/gpu_iq2_moe_prefill_bench_rocm speed-bench/gpu_iq2_moe_prefill_bench_cuda speed-bench/rocm_q4_prefill_bench speed-bench/cuda_q4_prefill_bench speed-bench/*.o tests/test_q4k_dot tests/test_mxfp4_dot tests/test_quantizer_indexer_q4 tests/test_mxfp4_metal tests/test_mxfp4_rocm tests/bench_mxfp4_rocm tests/test_mxfp4_cuda tests/test_rocm_q4_dense_pair tests/test_metal_session_batch tests/test_metal_q4_streams tests/test_metal_q4_prefill_pair tests/test_metal_indexer_q4 tests/test_metal_q4_attn_exactn tests/test_metal_q4_attn_out_a_direct tests/test_metal_q4_qb_f16_cache tests/test_metal_exactn_oracle tests/test_metal_dspark_capture tests/test_metal_argmax_top1 tests/test_metal_iq2_midonly tests/test_metal_iq2_ssd_grouped_mm tests/test_metal_iq2_live_index tests/test_glm53_kda tests/test_glm53_kda_rocm tests/test_glm53_vision_engine tests/test_glm53_vision_prompt tests/test_deepseek4_vision_image tests/test_gpu_xdev tests/test_gpu_model_cache tests/test_gpu_lookup_cache_strict tests/test_engine_mgpu_refusal tests/test_engine_mgpu_runtime tests/test_engine_correctness tests/test_sampling tests/test_cuda_session_batch tests/test_cuda_mixed_batch tests/*.o *.o cuda/mmq/*.o cuda/mmq/test/*.o tests/cuda_long_context_smoke tests/cuda_long_context_smoke.o | |
| rm -f ds4 ds4-server ds4-bench ds4-eval ds4-agent ds4_cpu ds4_native ds4_server_test ds4_test ds4_agent_test gguf-tools/quality-testing/score_official gguf-tools/quality-testing/score_official.o speed-bench/metal_decode_schedule_bench speed-bench/metal_prefill_variant_bench speed-bench/metal_q4_dense_pair_bench speed-bench/metal_q4_prefill_pair_bench speed-bench/metal_q4_mm_tail_cull_bench speed-bench/metal_q4_attn_out_a_direct_bench speed-bench/metal_iq2_moe_tail_cull_bench speed-bench/gpu_iq2_moe_prefill_bench_rocm speed-bench/gpu_iq2_moe_prefill_bench_cuda speed-bench/rocm_q4_prefill_bench speed-bench/cuda_q4_prefill_bench speed-bench/*.o tests/test_q4k_dot tests/test_mxfp4_dot tests/test_quantizer_indexer_q4 tests/test_mxfp4_metal tests/test_mxfp4_rocm tests/bench_mxfp4_rocm tests/test_mxfp4_cuda tests/test_rocm_q4_dense_pair tests/test_metal_session_batch tests/test_metal_q4_streams tests/test_metal_q4_prefill_pair tests/test_metal_indexer_q4 tests/test_metal_q4_attn_exactn tests/test_metal_q4_attn_out_a_direct tests/test_metal_q4_qb_f16_cache tests/test_metal_exactn_oracle tests/test_metal_dspark_capture tests/test_metal_argmax_top1 tests/test_metal_iq2_midonly tests/test_metal_iq2_ssd_grouped_mm tests/test_metal_iq2_live_index tests/test_glm53_kda tests/test_glm53_kda_rocm tests/test_glm53_vision_engine tests/test_glm53_vision_prompt tests/test_deepseek4_vision_image tests/test_prompt_prefix tests/test_gpu_xdev tests/test_gpu_model_cache tests/test_gpu_lookup_cache_strict tests/test_engine_mgpu_refusal tests/test_engine_mgpu_runtime tests/test_engine_correctness tests/test_sampling tests/test_cuda_session_batch tests/test_cuda_mixed_batch tests/*.o *.o cuda/mmq/*.o cuda/mmq/test/*.o tests/cuda_long_context_smoke tests/cuda_long_context_smoke.o | |
| rm -f tests/*.o *.o cuda/mmq/*.o cuda/mmq/test/*.o | |
| rm -f speed-bench/*.o | |
| rm -f gguf-tools/*.o | |
| rm -f gguf-tools/quality-testing/*.o | |
| rm -f ds4 ds4-server ds4-bench ds4-eval ds4-agent | |
| rm -f ds4_cpu ds4_native | |
| rm -f ds4_server_test ds4_test ds4_agent_test | |
| rm -f gguf-tools/quality-testing/score_official | |
| rm -f speed-bench/metal_decode_schedule_bench | |
| rm -f speed-bench/metal_prefill_variant_bench | |
| rm -f speed-bench/metal_q4_dense_pair_bench | |
| rm -f speed-bench/metal_q4_prefill_pair_bench | |
| rm -f speed-bench/metal_q4_mm_tail_cull_bench | |
| rm -f speed-bench/metal_q4_attn_out_a_direct_bench | |
| rm -f speed-bench/metal_iq2_moe_tail_cull_bench | |
| rm -f speed-bench/metal_iq2_moe_top8_pair_bench | |
| rm -f speed-bench/gpu_iq2_moe_prefill_bench_rocm | |
| rm -f speed-bench/gpu_iq2_moe_prefill_bench_cuda | |
| rm -f speed-bench/rocm_q4_prefill_bench | |
| rm -f speed-bench/cuda_q4_prefill_bench | |
| rm -f tests/test_q4k_dot tests/test_mxfp4_dot | |
| rm -f tests/test_quantizer_indexer_q4 | |
| rm -f tests/test_mxfp4_metal | |
| rm -f tests/test_mxfp4_rocm tests/bench_mxfp4_rocm | |
| rm -f tests/test_mxfp4_cuda | |
| rm -f tests/test_rocm_q4_dense_pair | |
| rm -f tests/test_metal_session_batch | |
| rm -f tests/test_metal_q4_streams | |
| rm -f tests/test_metal_q4_prefill_pair | |
| rm -f tests/test_metal_indexer_q4 | |
| rm -f tests/test_metal_q4_attn_exactn | |
| rm -f tests/test_metal_q4_attn_out_a_direct | |
| rm -f tests/test_metal_q4_qb_f16_cache | |
| rm -f tests/test_metal_exactn_oracle | |
| rm -f tests/test_metal_dspark_capture | |
| rm -f tests/test_metal_argmax_top1 | |
| rm -f tests/test_metal_iq2_midonly | |
| rm -f tests/test_metal_iq2_ssd_grouped_mm | |
| rm -f tests/test_metal_iq2_live_index | |
| rm -f tests/test_glm53_kda | |
| rm -f tests/test_glm53_kda_rocm | |
| rm -f tests/test_glm53_vision_engine | |
| rm -f tests/test_glm53_vision_prompt | |
| rm -f tests/test_deepseek4_vision_image | |
| rm -f tests/test_prompt_prefix | |
| rm -f tests/test_gpu_xdev | |
| rm -f tests/test_gpu_model_cache | |
| rm -f tests/test_gpu_lookup_cache_strict | |
| rm -f tests/test_engine_mgpu_refusal | |
| rm -f tests/test_engine_mgpu_runtime | |
| rm -f tests/test_engine_correctness | |
| rm -f tests/test_sampling | |
| rm -f tests/test_cuda_session_batch | |
| rm -f tests/test_cuda_mixed_batch | |
| rm -f tests/cuda_long_context_smoke |
It makes the Makefile more navigable for humans
| ``` | ||
| md5(prompt.txt) = 3c2d6993... 135 000 B | ||
| md5(head -c 135000 prompt-long.txt) = 3c2d6993... | ||
| ``` |
There was a problem hiding this comment.
MD5 is outdated and now cracked, SHA-1 is shattered
There was a problem hiding this comment.
Forging a collision is one thing, using it to verify an output is another.
There was a problem hiding this comment.
Um, okay then, treat MD5 and SHA-1 like CRC32 then
|
|
||
| The long file was the short one repeated three times. Every sweep short enough | ||
| to fit inside the first copy was reading a byte-identical prefix. Checksum the | ||
| prefix your sweep actually consumes, not the whole file. |
There was a problem hiding this comment.
| prefix your sweep actually consumes, not the whole file. | |
| prefix your sweep actually consumes, not the whole file: | |
| ```sh | |
| sha512sum prompt.txt | |
| # prompt.txt 4cae1eb2... | |
| # If your prefix is 135000 bytes (not tokens): | |
| head -c 135000 prompt.txt | sha512sum | |
| # prompt.txt 4cae1eb2... | |
| ``` |
|
@GiorgioOppo The reason I am writing this up rather than just reporting the count: the four are not one problem. They are two, and only one of them looks like a defect. The criterionThe pass/fail test on this path is Normally that would make the test over-specified, since two dispatches may accumulate in a different order. Not here: the comment at the call site states the intent plainly — the same kernel is run twice on the same data, differing only in whether the expert table is compacted and the ids remapped, and "bitwise equality with the compact result validates the full-expert copies". Bit equality is the design assumption, so a difference is meaningful. Group 1: bit-inequality only (2 cases)
The Not one element differs by more than 1e-3. The large Group 2: a real numeric disagreement (2 cases)
At a magnitude of ~145 one ULP is about 1.5e-5, so 0.34 is tens of thousands of ULP of genuine disagreement. It affects 2.0% and 2.7% of elements respectively — rare but large, which is not the signature of accumulation-order noise. It is also seed-dependent: the neighbouring seed at the same shape ( Everything else is healthy in all four: SuggestionGroup 1 is arguably a question about whether bit-exactness is the right gate for compact-vs-global, and could be closed by a decision rather than a fix. Group 2 is a numeric question in the fused MoE I have an interest to declare: since 5 Sep this branch is what serves our production, so group 2 matters to us directly rather than academically. Happy to run any instrumented build you want on GB10 — reproduce with |
|
@GiorgioOppo Resolved, and it turned out to be worth the chase. Two of my earlier attributions in this thread were wrong; the real answer changes how the prefill numbers here should be read, including yours. What I got wrong, twice
What is actually going on
So "prefill_tps at ctx 8192" is not one quantity. Same model, same machine, same context — different operation depending on how you got there. I isolated this with three conditions, reading only the ctx 8192 row, 108 runs, no failures:
A and B differ by 8x in The finding that matters for this PREvery point, sorted by increment:
On CUDA / GB10, the AProjQ4 prefill advantage exists for small prefill chunks and disappears for large ones. At increments of 16k and 32k it is indistinguishable from zero. A possible mechanism, offered as a hypothesis and not a result: the logs report What this impliesEvery prefill figure in this thread — mine, and I think several others — is implicitly a statement about one increment size. "AProjQ4 is faster on prefill" and "AProjQ4 is at parity on prefill" can both be true measurements of the same build. That may also be worth checking against @iammac2's ROCm -2%: if that came from a different sweep shape than the CUDA numbers it is being compared with, the two are not measuring the same operation. I withdraw my earlier "try a different prompt" suggestion — the file cannot matter — but the invocation can, a lot more than I expected. Pinning Decode is untouched by all of this and remains the strong result: +13.92% and +13.93% at ctx 8192 on Method and toolingBalanced design throughout — arms adjacent in time, order alternated between rounds, ratios paired within a round, clock and temperature recorded around every run, production stopped only for the length of a block. I put the corrected rule and an enforcement check into GiorgioOppo#14: Raw CSVs and logs available for any of the above. |
|
@iammac2 can you retest it? |
In progress.... |
ROCm AProjQ4 retest — replication with CIs (reply to "can you retest it?")@GiorgioOppo — replicated on head Decode3 paired samples per context.
Holds at larger context (8192-step sweep): 32768 → +11.11% [+10.99%, +11.22%], and 65536 → +10.44% (single run). Q4 decode leads Q8 by ~11–12% across the range — reproducing my earlier runs and in the CUDA ~14% direction. Prefill — chunk size decides the sign
At 2048-token chunks Q4 is barely faster (+0.6–1.0%) — a real but tiny effect, far below CUDA's +2.5–5%. At 8192-token chunks Q4 is ~7% slower, a large, tight deficit that CUDA does not show (parity). So the prefill picture is chunk-dependent on ROCm too, but with the opposite sign to CUDA: no meaningful small-chunk win, and a real large-chunk loss. ConditionsSCLK held at 1000 MHz; peak temp 74 °C; peak power 100 W (small-chunk ~80 W, large-chunk ~100 W); Method and tooling
|
If you open it in my codebase, I can merge it from GitHub and give you credit for the fix. |
Link ds4_image.o so the test resolves ds4_deepseek4_attention_bounds and can run on CUDA hosts. (cherry picked from commit 929563d)
|
AProjQ4 numbers on the current head Setup
Arms are our three production builds (each is the branch commit plus the still-open #704 cherry-pick, which touches
|
| metric | overall (n=48) |
|---|---|
| decode | +0.14% [+0.06, +0.23] |
| incremental prefill | +0.07% [−0.11, +0.24] |
Nothing moves, at any frontier. 0edc5534 (execution phases), 35da3323 and 809386d2 (indexer candidates) are neutral here — same conclusion @iammac2 reached on gfx1151, from an independent rig.
7a5002de vs 8c22d667 — cumulative
| ctx | 2048 | 4096 | 6144 | 8192 | overall |
|---|---|---|---|---|---|
| decode | +9.86% | +7.42% | +7.61% | +7.54% | +8.11% [+7.80, +8.42] |
| prefill | −2.16% | −1.08% | −1.03% | −1.12% | −1.35% [−1.57, −1.13] |
Absolute decode at 8192: 8c22d667 18.53 t/s → 7a5002de 19.93 t/s. Baseline session drift over the 43 min: 0.6%.
Context
We run this branch in production on a DGX Spark (switched to d0bc6f07 on 11 Sep, then to 7a5002de the same evening). The quality gate — score_official against a 6a20b131 reference — has come out bit-identical on all three of those builds, which is the check that actually gates our deployments.
One earlier observation of mine is now fully retracted: the 0x4 SW-power-cap samples appeared in the 7a5002de arm this time, in the main arm last time, and across four arms in the run before. It is a transient GB10 event, unrelated to which binary runs — please disregard it as a branch property.
Integrate codex/ds4.1-metal-ssd at 8cf2383 into the V4 Q4 branch at f6f036c, preserving both histories and the CUDA smoke-test link fix. This includes upstream V4.1 support, the SSD prefill/cache improvements, BF16 boundaries, typed Q4 attention, calibration and conversion tools. Reconcile the overlapping implementations rather than replacing either backend's existing attention and streaming paths: - Keep V4 Q4 direct output-A, HC, sidecar and indexer implementations. Deduplicate the identical direct output-A kernel shared with V4.1. - Preserve V4 compact top-k for noncausal calls and V4.1 causal selection, shuffle sorting, prefix pruning and matching host/shader argument layouts. - Give Q8 reduction templates independent single-barrier and BF16 options, retaining the V4 single-barrier and V4.1 BF16 specializations. - Preserve the M32/K64 and expert-tail kernels; size the shared A/B boundary using sizeof(S0), including the F32 router case. - Move V4.1 diagnostic bits to 13..15 to avoid the V4 indexer flag collision. - Exclude V4.1 sessions from V4 sidecar preflight and register successful V4.1 session creation with the common live-session counter. - Adapt V4.1's Q4 slice call to the resident_decode argument, passing zero and treating only positive returns as success. - Retain full CUDA Q4 header dependencies and the V4.1 test targets. Extend integration coverage for session ownership and extracted shaders: - Check V4.1 preflight exclusion without changing session state, and idempotent live-session registration in the host sanitizer fixture. - Add a bounded --session-lifecycle check: two real V4.1 Metal sessions, context 256, SSD streaming, public preflight and exact cleanup, without evaluating the model. Extend the existing session accounting fixtures. - Teach the host indexer harness to replay production SIMD shuffles from lane snapshots and retain causal-width handling. Add 11,904 causal leaf cases with exact scalar/shuffle permutations, tails and masked future rows. Validation on Apple M1 Max, 32 GiB: - Full production build and make -q all passed. - 20 focused Metal test groups passed with API validation, covering V4/V4.1 Q4 and Q8/BF16, direct output-A, prefill pairs, Q-B cache, top-k, SSD expert formats, streamed prefill lifetime, calibration in release flags, snapshot identity, dispatch and session lifecycle/state. - 14 host/frontend/protocol groups passed, including mocked RDMA completion accounting. No actual distributed model inference was run. - Host ASan/UBSan checks passed for preflight, indexer/shuffle/streaming and selected CUDA/ROCm policies and arithmetic. CUDA smoke dry-run retains ds4_image.o. CUDA/ROCm production source files are unchanged by the merge. - Seven conversion, four APFS requantization and two vision conversion Python fixtures passed. Six default-path SSD runs compared the exact source branch and merge with the same 42-token Rome prompt and 50 generated tokens at temperature zero. Outputs are byte-identical within each model, including the reversed-order Q8 repeat. Q4 generation was 2.62 -> 2.61 t/s. Q8 averaged 2.59 -> 2.485 t/s over two runs per version; prefill averaged 2.315 -> 2.28 t/s. The observed approximately 4% Q8 decode slowdown remains unexplained. Reverting only the generic Q8 single-barrier matvec diagnostically did not recover it. Q8 has 378 cached experts versus Q4's 633, but that does not prove an I/O cause. Native resident Q8 microbenchmarks also passed numerical checks and showed shape-dependent timing variation. Do not claim general performance parity or introduce a policy gate on the basis of these short measurements. CUDA/ROCm hardware runs, actual distributed inference and full-model V4 or fully resident inference were unavailable; V4 and resident Metal coverage uses synthetic numerical fixtures. The 447 imported official V4.1 fixture files retain their exact upstream bytes, including intentional whitespace. Models, executables, local logs and intermediate notes are not committed.
The single-token shared expert previously dispatched gate, up, SwiGLU, and output BF16 conversion separately. Add a paired Q8 specialization that preserves BF16 rounding of both projections before clamping and SwiGLU, then rounds the activation output in the same kernel. Admit the V4.1 K5120/M2304 shape with the established NSG4 dispatch. Validate model ranges, tensor sizes, alignment and overlapping views before encoding. Distinguish unsupported inputs from GPU failures so the caller can retain the existing fallback without replaying failed GPU work. Preserve quality, imatrix, tensor-parallel and other Q8 paths. Add the shared-expert oracle, its build target and binary exclusions. Tests compare against the actual four-dispatch sequence byte-for-byte, including BF16 ties, cancellation, clamp boundaries, view offsets, input immutability, canaries and rejected layouts/dispatch settings. Validation: - Metal application, CPU object and dedicated test builds pass. - Shared oracle passes with Metal API Validation on M1 Max, 32 GiB. - Batched microbenchmark: 100.266 -> 76.328 us, median of 11 ABBA samples with 32 invocations per batch. This is a kernel-sequence measurement, not an end-to-end generation throughput claim.
Repeated Q4 dequantization across Q-B output tiles adds compute work during prefill. For eligible K1280/M32768 batches, expand the weights once into the dead attention-head workspace and run the FP16 matrix kernel with the reused half-precision RHS. Admit pre-M5, single-device, non-quality batches of 256 through 2048 rows when the supplied workspace holds the RHS and 80 MiB of expanded weights. Keep RHS-only reuse or native Q4 for smaller views and other shapes. Respect real view offsets and used ranges, including aliases with registered model buffers; allocate no persistent weight cache. Extend the existing dequantization encoder with a scratch offset while keeping zero-offset behavior at its other call sites. Add an independent Q4/F32 oracle before the BF16/RoPE boundary. Cover 127/128/255/256/257/437/512/513/1024/2048 rows, full and RHS-only scratch, an almost-large-enough view, fractional scales, FP16 ties, canaries, queued scratch reuse and freeing views before command completion. Validation: - Metal application, CPU object and Q-B test builds pass. - Numerical and lifetime tests pass with Metal API Validation and unretained command buffers, including the SSD fixture at 437 rows. - Production API benchmark at 512 rows: 9.978 -> 8.954 ms versus the previous RHS-reuse path, with conversion/dequantization included. Nine samples use rotated order; 2048-row results remain variable.
Layers with the same causal geometry rebuilt identical attention masks and block maps. Add separate V4.1 cache slots for raw SWA, ratio-2 compression and ratio-1 compression, preserving the existing V4/M3 cache policy and avoiding collisions between architectures. Reuse only static zero-prefix geometries admitted by the V4.1 batch wrappers. Rebuild on geometry changes and retain the existing paths for append/ring contexts, visual or dynamic masks and unsupported windows/ratios. Keep producer/consumer encoder boundaries explicit and release the cached resources after the prefill sweep has drained. Add cache-on/cache-off bitwise tests for cold and warm entries, changed Q/KV inputs, compressed F32/F16 caches, boundary row counts and fallback geometries. Exercise interleaved cache entries, intermediate flushes, release with an active command buffer, canaries and input hashes. Validation: - Metal application, CPU object and mask test builds pass. - Oracles pass with Metal API Validation and unretained command buffers. - M1 Max benchmark: 1757.778 -> 1725.879 ms for 40 attention calls at 437 rows, using 2 SWA, 18 ratio-2 and 20 ratio-1 calls; seven ABBA samples. This includes attention execution, not just mask creation.
Short expert groups previously used N32 storage and accumulators even when only a few token rows were occupied. Partition the compact route work on the GPU into <=8, <=16 and remaining-row buckets. Dispatch real N8/N16 IQ2_XXS gate/up/SwiGLU and Q2_K down kernels, retaining the established N32 kernels for larger occupied tiles. Preserve expert/tile ownership, dequantization, ascending K reduction, half intermediate values, route weights and fixed output scatter. Generate indirect dispatch arguments without CPU readback. Explicit encoder boundaries order the partition and its consumers; transient retention covers flushed and unretained command buffers. Empty buckets dispatch one group that exits before accessing work items. Enable automatically for the existing pre-M5 V4.1 384-expert/top-6 K5120/M2304/output5120 path through 1024 rows, including SSD prefill with a fully staged layer. Preserve other quantizations and execution modes. A diagnostic force flag makes reduced-shape tests exercise the new path and fails instead of silently testing the fallback. Validation: - Metal application, CPU object and MoE test builds pass. - Bitwise oracles cover N8/N16 boundaries, mixed tails, full tiles, resident/SSD bindings, canaries and caller-owned command buffers. - Release-to-release M1 Max ABBA microbenchmarks reduce elapsed time by 30-32% at 437 rows and 12-13% at 1024 rows. Synthetic routing and warm pipelines do not establish equivalent SSD model throughput.
Batch shared-expert activations and hyper-connections made separate passes for elementwise math and BF16 boundaries. Add fused kernels for SwiGLU, HC expansion with an optional add, and HC weighted sums using either packed pre-weights or the existing split-weight layout. Keep the same gate/up GEMMs, rounding both projections before clamp and SwiGLU and rounding the output afterward. Round the block or block-plus-shared sum before the established ordered HC calculation, then round each output stream. Update prefill and batched-session call sites together so moving the quantization boundary is atomic. Check used tensor ranges, alignment, aliases, row bounds and buffers. Preserve the TP, quality and imatrix fallbacks and the single-row shared-expert path. The diagnostic disable switch compares the same release computation; it does not introduce an opt-in output variant. Validation: - Metal application, CPU object and epilogue test builds pass. - Bitwise tests cover 150 shape/pattern/operation cases, clamp edges, BF16 ties, cancellation, real Q8 GEMMs at 2-8 rows, disjoint views, canaries, immutable inputs and rejected layouts. - Tests pass with Metal API Validation and unretained command buffers. - M1 Max 437-row ABBA sequence timings include 82.375 -> 32.325 us for SwiGLU and 1025.200 -> 207.375 us for HC expansion. These are isolated epilogue timings, not full-model speedups.
The heads8 prefill kernel synchronized the threadgroup around each KV row. Add a V4.1 specialization that stages sixteen chronological rows at once and consumes them in the original order, amortizing barriers without changing the per-row reduction or online-softmax updates. Keep negative and future selections unread, preserve raw-ring indexing, support compressed F16/F32 caches and apply the attention sink last. Use the new path automatically on pre-M5 devices for batched ratio-1/2 attention with 32/64 heads, top-k 512, window 128 and head width 512. Retain decode, M5, quality and unsupported-geometry behavior. Add a diagnostic reference flag and a test getter that reports the actually encoded row block. Expand the indexed/TP oracle through 2048 rows, including short tails, ring/prefix geometry, invalid selections, compressed cache formats and the existing fallback paths. No runtime RB4/RB8/RB16 selection variable is added. Validation: - Metal application, CPU object and indexed attention test builds pass. - Old/new output and full/split-head comparisons are bitwise; the independent CPU oracle remains within its numerical error bound. - Metal API Validation and unretained-command-buffer tests pass. - Final M1 Max ABBA medians, including sorting: 28.756 -> 26.998 ms at 512x64, 50.598 -> 47.670 ms at 1024x64, and 57.067 -> 53.301 ms at 2048x32. These are attention-call timings, not model throughput.
Layer 20 still published its ratio-1 compressor outputs row by row although the batch helper already preserves scalar-order projections. Admit every KV-source layer to batch publication, preserving BF16, normalization, RoPE positions, native FP4 cache rounding and causal visibility. Reuse layers continue consuming the established owners. Later narrow prefill layers already have activations in place after the previous drain and cache seed. Skip their empty begin/end command buffer pair, retaining the initial embedding and wide-chunk carry submissions and their resource-ordering boundaries. Add a publication oracle comparing whole scalar/batch KV and index caches byte-for-byte. Cover 1-2048 rows, irregular tails, long starting positions and suffix replay, quality mode, unchanged cache guards and immutable inputs. Link this internal graph fixture without ds4.o. Validation: - All seven commit states build the Metal application, CPU object and their associated tests independently. - Publication, epilogue, MoE, indexed attention, SSD expert-stream, batch-admission and dispatch tests pass with Metal API Validation and unretained command buffers on M1 Max, 32 GiB. - The combined MoE/epilogue/indexed/scheduler changes were compared with a build already containing the shared/Q-B/mask optimizations: SSD Q4 at 839 tokens improved 29.30 -> 30.28 t/s in one paired run. At 437 tokens, two runs per arm averaged Q4 16.05 -> 15.37 t/s and Q8 15.645 -> 15.735 t/s. A diagnostic Q4 repeat measured 16.05 -> 16.29 t/s; SSD throughput is variable, with no proven overall 10% gain. - Generated outputs match each baseline. Complete prefill logits are byte-identical in both profiled Q4 comparisons (437 and 839 tokens). The memory plan and SSD cache budget remain unchanged.
|
@trueimage Would you like to test this implementation of DeepSeek v4.1? |
The short routed tiles repeatedly loaded the same packed Q2 bytes across four K32 steps and converted each token's gate/up input for every route. Cache the N8 down tile's packed bytes and scales across K128, and share one token-major FP16 input copy between all six IQ2_XXS gate/up routes. Use scalar uint loads for the 84-byte Q2 block stride. Preserve the original dequantization arithmetic, half rounding, K32 traversal and K8 MMA accumulation sequence. Add half-RHS N8/N16 specializations and reuse the established paired N32 half-RHS kernel. N16/N32 down kernels remain unchanged because median-five measurements did not favor packed reuse. Keep the existing pre-M5, TP1, non-quality, V4.1 shape/type and <=1024-row admission. Enable the half copy automatically from 256 rows. Validate all optional pipelines, memory limits, strides and allocations before replacing the baseline plan; unsupported cases retain that plan. Retain the transient RHS through borrowed and flushed command buffers. Add two diagnostic test flags without introducing environment opt-ins. Expand the MoE oracle to nine variants, checking intermediate and final outputs byte-for-byte, guards, input immutability and half-rounding edge cases. Cover resident and SSD full-layer bindings, borrowed command buffers, N8/N16/N32 tails and balanced/skewed 128/437/1024-row batches. Benchmarks use one warmup and five alternating measured sweeps. Validation on M1 Max, 32 GiB, with the complete series: - Metal application, CPU object and MoE test build passed. - Small and full-shape MoE oracles passed; the small fixture also passed Metal API Validation with unretained command buffers. - Representative median call times versus forced baseline small tiles: 128 rows, skew2, Q2-only: 97.629 -> 93.200 ms; 437 rows, skew1, combined: 88.141 -> 85.494 ms; 1024 rows, skew2, combined: 130.049 -> 128.597 ms. These are isolated MoE call measurements, not end-to-end SSD gains.
When all compressed keys fit in 2048 eight-key candidate blocks, ranking those blocks cannot exclude anything. At decoder source layer 20, publish only the valid all-visible mask prefix with an ordered compute kernel. Reindex layers 24/28/32/36 then skip the redundant candidate filter. Keep the existing ranking and filtering path above 16384 compressed keys. Leave the physical mask tail untouched and preserve queued carry-restore ordering. Validate mask size/alignment and bounds before encoding. Using a compute encoder preserves ordering without CPU writes racing pending GPU copies or a blit-induced encoder switch for each token row. The boundary oracle also exposed a pre-existing lifetime bug: candidate block top-k followed by a larger key top-k can grow the scratch slab in one unretained command buffer. Retain the old slab until current and pending work complete. Route both compact and legacy top-k allocation sites through the retention helper. Add an independent legacy-sequence oracle and a dedicated make target. Start cold at 16385 keys to exercise scratch growth without a warmup workaround. Cover both sides of the 16384-key boundary, partial blocks, Full/Reuse/Reindex, ratio-2 layers, reset-to-shorter state, ties, signed zero, infinities, queued carry restoration and single-operation buffers. Compare masks, scores and IDs exactly; check guards and untouched tails. Validation with the complete series on M1 Max: - Metal application, CPU object and candidate oracle build passed. - Candidate oracle passed normally and with Metal API Validation plus unretained command buffers after the scratch lifetime fix. No isolated end-to-end throughput gain is attributed to this commit.
Sequential decode visits each index source before its reuse layers, so the dedicated selected-KV scratch already contains their gathered rows. Factor the gather into a graph helper and skip it only for unprojected reuse layers in the explicit DECODE execution phase. Once compressed KV is present, this reduces gathers from 38 to eight per generated token. Keep the gather for source layers, projected calls, AUTO, PREFILL, VERIFY, BATCH_DECODE and MIXED phases. Layer-major prefill and session batches may share this scratch across token positions and cannot assume that the preceding layer left the required rows there. The optimization adds no new cache state or invalidation protocol and preserves the same owner mapping, selected IDs and top-k bounds. Add an oracle comparing the real graph helper against unconditional gather over all 40 layers and nine positions in normal, quality and SSD modes. Cover ratio-2 and top-k boundaries, reset, phase/projected fallbacks, source immutability and buffer guards. Queue source20/reuse21 and source24/reuse25 together to check ordered scratch reuse in one command buffer. Include the test's build, clean and ignore entries. Validation with the complete series on M1 Max: - Metal application, CPU object and gather oracle build passed. - Exact gather oracle passed with Metal API Validation and unretained command buffers, including absent-resource checks of fallback dispatch. No isolated full-model speedup is claimed for gather elimination alone.
Allow explicit full-layer expert sweeps through 2048 rows instead of 1024, preserving the narrow single-chunk, batch-HC/core, device and memory-capacity guards. The two-layer weight reserve depends on layer bytes, so the larger admitted batch does not enlarge that reserve. Wide, encoder-only and resumed-encoder sweeps retain their existing path. Open a separate read-only, close-on-exec descriptor for uncached prefill reads on macOS. Check its device, inode and size against the model before setting F_NOCACHE. A new open, rather than dup, leaves decode's descriptor and caching policy unchanged. Fall back to the original descriptor when the path cannot be reopened or its identity changed. Close only owned descriptors after every reader has joined, including cancellation/EOF. Ordinary release allowed the Metal allocation cache to retain the dead expert slabs after prefill. Join readers, drain GPU work, detach streamed bindings and munlock pages before marking complete owning Shared buffers purgeable-empty and freeing their handles. Reject views, bound/model buffers, heap/private storage, non-SSD mode and active/pending commands; skip discard if GPU drain fails. Interpret Metal's purgeability return as the prior state, not a success flag. Keep ordinary tensor free intact. Expand graph tests for admission boundaries and exclusions, independent file descriptors and unavailable/replaced-path fallback, reader lifetime, EOF/cancellation, discard rejection, queued copies and safe reallocation. Validation of the complete four-commit series on M1 Max, 32 GiB: - make all, CPU object and all affected test builds passed. - Host admission/fd tests passed. Expert-stream and discard GPU tests, candidate/gather oracles and small MoE fixtures passed with Metal API Validation and unretained command buffers; full-shape MoE tests passed. - SSD runs used DeepSeek-V4.1-Flash-Q2-AProjQ4-requant.gguf, ctx4096, CLI prefill-chunk128 and temperature zero, against base 8ec5f79. - 437 prompt / 32 output tokens, one A/B pair: prefill 14.70 -> 18.25 t/s, decode 2.26 -> 2.65 t/s; post-cleanup footprint 13.10 -> 5.98 GiB. - 1241 prompt / 64 output tokens: fresh baseline 11.39 prefill / 2.81 decode t/s; two final runs averaged 49.925 / 2.81 t/s. Individual final decode results were 2.73 and 2.89 t/s. Final cleanup footprint: 6.04 GiB. - Complete prefill logits and generated text were byte-identical within each prompt family across all 11 baseline/intermediate/final runs. These are combined-series SSD measurements on one device, dominated at 1241 rows by sweep admission, not by the <=1024-row MoE specialization. They are not independent per-commit A/B results or a general speedup claim. Resident kernels have synthetic coverage; the roughly 341 GiB model cannot run fully resident in 32 GiB. CUDA/ROCm were not changed.
Native Q4 output-B GEMM repeats weight dequantization across token tiles. For eligible pre-M5 Apple Silicon batches of 256..2048 rows, expand the 8192x5120 Q4_K matrix once and convert its input once, then use the existing FP16-RHS GEMM. Preserve FP32 accumulation, the native K32 MMA order and the BF16 boundary between the two attention output projections. Borrow the graph's Q storage after attention has consumed it. The optional workspace requires 80 MiB for weights plus 16 KiB per input row and creates no persistent weight cache or additional graph allocation. Close encoders before each dependency and before FFN aliases reuse that storage. Keep the original API as the native fallback. Limit admission to TP1, non-quality, non-concurrent configurations with sufficient aligned storage and supported pipelines. Reject overlapping tensors or model mappings before encoding output-A, including distinct no-copy Metal wrappers over the same pages. Q8 output-B and unsupported shapes retain their old path. Add a dedicated numerical oracle and benchmark target covering dispatch boundaries, partial rows, mixed Q4/Q8 projections, quality/TP fallbacks, insufficient capacity, alias rejection and queued workspace reuse. Validation on Apple M1 Max (32 GiB): - Build and CPU compilation passed for the combined change set. - Output-B oracle passed with Metal validation and unretained command buffers; both the low BF16 boundary and final F32 output match bitwise. - Complete output-A/BF16/output-B microbenchmarks, including conversion and dequantization, improved throughput by 7.3..10.2% at 256/437/1241/2048 rows. - Combined SSD model comparisons retained identical prefill logits and text; these microbenchmarks do not demonstrate a stable whole-model speedup. - CUDA/ROCm hardware and a full resident model were not tested locally.
Scalar decode previously dispatched HC operations and their BF16 rounding separately, and materialized routed-plus-shared MoE output before HC expansion. Reuse the existing fused epilogue kernels to perform these operations together while preserving every original rounding boundary. Admit the fused graph path only for DECODE, TP1, non-quality, non-imatrix execution under the existing epilogue diagnostic control. Retain the old MoE wrapper for callers that consume the rounded block, including prefill views where block aliases routed. Pass deferred routed/shared outputs directly to the scalar HC expansion and retain the FFN pre-mixer copy for the next layer and vocabulary head. This removes six dispatches per layer and one at the output head: 241 fewer dispatches per token for the 40-layer V4.1 graph. This is a static launch count, not a claim of equivalent throughput improvement. Prefill, verification, batched/mixed decode, quality, imatrix and TP paths retain their established behavior. The existing SwiGLU fusion is unchanged. Add a scalar graph oracle exercising actual HC graph helpers across rounding ties, cancellation, recurrence, block/routed aliases, immutable inputs, guarded tensor views and all phase/quality/imatrix/TP exclusions. Add --bench-scalar to the existing epilogue benchmark with alternating baseline/fused samples while retaining its prefill benchmark. Validation on Apple M1 Max (32 GiB): - Graph and epilogue bitwise oracles passed with Metal validation and unretained command buffers; build and CPU compilation passed. - Integrated SSD comparisons at 1241 input tokens/64 generated tokens retained identical prefill logits and generated text. - An isolated short-prompt SSD comparison at temperature 0 generated 256 identical tokens at 2.81 -> 2.86 t/s. Output-B workspace was excluded by the 42-token input size. This single pair does not establish a stable decode speedup; individual per-step decode logits were not dumped.
Add --v41-tail-cull-large with selectable 1025..2048 rows and default 1241/1280/2048 cases. Exercise skewed and balanced expert routing and print N8/N16/N32 tile counts so occupancy can explain differences between forced kernel variants instead of attributing them only to prompt length. Make every timed variant borrow a caller-owned command buffer, matching layer prefill and avoiding an ownership bias between benchmark arms. Retain warmup, five alternating measurement sweeps, exact intermediate and output comparisons, tensor guards and the numerical reference path. Extend the small-shape oracle across the 1024/1025 admission boundary, 1241 rows and the 2048/2049 boundary, including automatic and synthetic SSD paths. Keep production kernels and the automatic 1024-row admission cap unchanged: larger batches did not show a consistent benefit across routing distributions, so the benchmark extension does not enable a new default. Validation on Apple M1 Max (32 GiB): - The expanded occupancy/boundary oracle passed all 44 cases with Metal validation and unretained command buffers. - Large-shape runs at 1241, 1280 and 2048 rows passed exact comparisons for the tested forced variants and both routing distributions. - Build passed; benchmark logs and experimental K64 kernels remain outside the repository and are not part of this commit.
|
The new DeepSeek V4.1 Flash Q2 AProjQ4 GGUF is now available for download for testing with the V4.1 implementation in this branch. Model: Hugging Face PR #4 — download availability, provenance and validation details The upload is complete. Download from the Hugging Face PR revision explicitly: hf download antirez/deepseek-v4.1-flash-gguf \
DeepSeek-V4.1-Flash-Q2-AProjQ4-requant.gguf \
--revision refs/pr/4 \
--local-dir ggufThis variant requantizes the 200 dense attention projections across all 40 layers from Q8_0 to Q4_K, preserving the existing expert quantization and the other tensors. It has been exercised with Metal SSD streaming on an M1 Max with 32 GiB memory; the implementation reference is The attention requantization is experimental and uncalibrated, derived from the existing Q2 GGUF's Q8_0 attention weights. The file retains the original offsets, so its logical file/download size remains 340.60 GiB. Quality and performance results for this V4.1 artifact still need independent validation; the earlier V4 results in this thread do not establish its gains. |
Integrate upstream a04f46f (DeepSeek v4.1 Flash support for CUDA) while preserving this branch's Q4 attention, Metal BF16 epilogues, prefill workspace reuse and explicit SSD expert streaming. Upstream brings CUDA V4.1 activation/attention kernels, bounded SSD caches, batched prefill/session execution, shared-expert overlap, resident tensor parallel decode graphs, aligned shards, split vocabulary projection, and associated host/device fixtures and operating documentation. Resolve shared-graph and backend integration: - Split V4.1 and TP declarations into the upstream headers while retaining Metal-specific typed Q4/workspace and fused BF16 APIs under Apple guards. - Keep CUDA's optimized Q8/Q8 attention-output batch dispatch. For Q4 and mixed pairs, use typed per-row projections with compact TP head/low views and preserve the required A -> BF16 -> B boundary before rank reduction. - Retain the generic Q4 low-projection fallback when the CUDA grouped kernel declines a device/shape; propagate submitted-work errors without retry. - Combine upstream CUDA projection/decode islands and MoE partial/finalize scheduling with Metal deferred epilogues and imatrix capture boundaries. - Preserve Metal's explicit two-buffer SSD prefill storage and selected expert prefetch without exposing Metal-only APIs to CUDA or ROCm builds. - Reconcile CUDA IQ2 helper signatures with the sign-table-free kernels, preserve Q4/F16 cleanup, and apply the prefill phase policy to the new SSD direct-SoA attempt while retaining upstream ownership filtering. - Preserve both test inventories and track the split GPU headers in Make. - Adapt the imported router fixture to Metal's established standalone softplus/sqrt result near zero, retaining CPU top-k/weight checks and existing tolerances rather than changing production router arithmetic. Validation on Apple M1 Max, 32 GiB: - Full Metal and CPU builds pass; final executables restored to Metal. - Full V4.1 Metal suite passes, including router, quantization, Engram, pooling, sparse gather/indexing, Q8 output and local TP attention slices. - Q4 attention, output-B, BF16 epilogues, Engram, prefill dispatch, graph batch admission/scalar epilogues and SSD expert admission/fd/discard/ streaming fixtures pass. - CUDA/ROCm shared-host syntax and non-Apple C fixtures pass. - IQ2 CUDA/ROCm host oracles pass under ASan/UBSan in normal and fast-math builds. A temporary extracted-helper oracle passes 234 CUDA attention dispatch/order/TP-stride/error-cleanup cases under ASan/UBSan. - Sequential pre-merge/merged SSD runs with Q2-AProjQ4-requant, ctx 4096, chunk 128, temp 0 and 50 generated tokens produce identical stdout. Single-pair rates: prefill 2.31 -> 2.41 t/s; decode 2.67 -> 2.63 t/s. These runs are a smoke/parity check, not a statistical speed claim. CUDA/ROCm device compilation, execution and multi-host distributed tests remain unverified because those GPUs are not available locally. No model, intermediate reports or temporary benchmark logs are included.
V4.1 CUDA prefill previously handled any Q4 attention-output pair by creating tensor views and dispatching O-A, BF16 rounding and O-B once per token. Move this work into a typed backend batch API and process up to 64 tokens per chunk to reduce host iteration and GPU launch overhead. Implementation: - Add an MMVQ sample-batch bridge for grouped O-A and full-width O-B. Keep ncols_dst=1 and repeat the scalar calculation over the sample axis, preserving its activation quantization, reduction configuration and finite-value epilogue instead of selecting a different matrix kernel. - Add a native Q4 batch kernel using the existing Q8_K quantizer, quarter-warp dot products and block traversal. Retain full weight-row strides when computing a tensor-parallel output-B K slice. - Keep the BF16 boundary between O-A and O-B inside the backend. Leave output in F32 so the graph retains ownership of the TP sum and final BF16 rounding. Support independent Q4_K/Q8_0 projection types and both local TP ranks without changing the model format or weights. - Preserve the existing Q8/Q8 batch path. For mixed Q4 A / Q8 B on one GPU, retain the scalar Q8 B dispatcher so resident aligned artifacts continue to select their existing arithmetic. - Bound activation scratch by the chunk size, with no persistent weight cache or new opt-in flag. Refresh shared scratch after nested Q8 calls, restore the physical device after lazy MMQ initialization, and reject Q4 batches during capture before resolving weights or growing scratch. - Validate shapes, ranks, tensor devices, ranges, alignment and overlap. Keep the Metal workspace dispatch and CUDA scalar decode API intact. Regression coverage: - Add a synthetic CUDA fixture with 60 type/row/local-TP cases, complete bitwise comparisons against the previous GPU output chain, sampled CPU Q4/Q8 references, BF16 checks, chunk boundaries and poisoned guards. - Add alternating old/new median benchmarks after correctness checks. The test target runs separate MMQ and DS4_CUDA_MMQ=0 processes. - Add ASan/UBSan host tests that extract the production bridge, native kernel indexing and public dispatcher. Cover strides, scratch sizing and relocation, launch failures, capture rejection and device restore. Validation on macOS / Apple M1 Max / 32 GiB, using synthetic tensors: - make -j4 all ds4_cpu.o tests/test_deepseek41_q4_attention tests/test_deepseek41_graph: PASS. - make -j4 cpu: PASS; restored Metal binaries with make -j4 -W ds4.o all. - tests/test_deepseek41_q4_attention: PASS. - tests/test_deepseek41_graph --scalar-epilogues: PASS. - tests/test_deepseek41_graph --prefill-expert-stream: PASS. - make test-cuda-v41-q4-output-host: PASS with ASan/UBSan. - make test-q4-epilogue-host: PASS. - Non-Apple and ROCm-preprocessed ds4.c syntax, strict C99 CUDA test fixture syntax, Linux test-target dry run and git diff --check: PASS. No NVIDIA/ROCm GPU or nvcc was available. Native CUDA compilation, numerical parity and throughput still require tester validation with make test-cuda-v41-q4-output and make bench-cuda-v41-q4-output. Local host checks do not establish GPU arithmetic parity. No whole-model GGUF run or end-to-end speedup is claimed by this commit.
|
@Checksum did you test it? |
Integrate upstream main through 6e4c285: - c2c3ce3 overlaps V4.1 CUDA SSD expert reads with prefill using a private reader, pinned staging ring and upload stream within the expert cache. Reserved slots are published after completion; cancellation and teardown join the reader before releasing storage or changing the model file. - e9e1baa keeps eligible medium SSD appends in one layer sweep while retaining the existing 2048-row arithmetic partitions and tiny-tail path. - 6e4c285 records upstream throughput measurements and regression checks in the QA record and DGX Spark guide. Resolve the ds4_gpu.h declaration conflict by retaining the Apple-only prefill discard/bind APIs and adding the separately guarded CUDA expert prefetch APIs. Preserve the branch's Metal explicit expert streaming, Q4 attention optimizations and CUDA typed O-A/B batch implementation. The remaining six files merge automatically. No intermediate logs or local notes are included. Validation on macOS, Apple M1 Max with 32 GiB RAM: - make -j4 all ds4_cpu.o tests/test_deepseek41_prefill tests/test_deepseek41_q4_attention tests/test_deepseek41_graph: PASS. - make -j4 cpu: PASS; restored the production Metal binaries afterward with make -j4 -W ds4.o all. - tests/test_deepseek41_prefill --dispatch: PASS. - tests/test_deepseek41_q4_attention: PASS. - tests/test_deepseek41_graph --scalar-epilogues: PASS. - tests/test_deepseek41_graph --prefill-expert-stream: PASS. - make test-cuda-v41-q4-output-host: PASS with ASan/UBSan. - A temporary host harness extracting the actual CUDA prefill dispatch functions: PASS for medium-sweep boundaries, flags, capacity limits, TP/resident exclusions and unchanged 2048-row encoder partitions. - Non-Apple and ROCm-preprocessed ds4.c syntax and strict C99 syntax for the CUDA SSD cache and Q4 output fixtures: PASS. - git diff --cached --check: PASS; no unresolved conflict entries. The tests above use synthetic tensors and dispatch checks, not a full GGUF inference run. No NVIDIA/ROCm device or CUDA compiler is available locally, so native GPU compilation, distributed inference and CUDA SSD numerical/throughput regressions remain for hardware testers. Upstream benchmark results in the imported documentation were not rerun here.
Q4 attention output previously launched a separate BF16 pass between O-A and O-B. Fold that rounding into the MMVQ finite-value epilogue or the native Q4 reduction store to avoid another launch and low-buffer pass. - Extend the sample-batch MMQ bridge with an explicit output_bf16 mode and reject unsupported mode values before launching work. Update its sole production caller, declaration and host dispatch fixtures together. - Preserve sanitize-then-round ordering on MMVQ: NaN and infinity inputs become zero, while finite values that round to infinity remain infinite. Retain round-to-nearest-even and signed-zero behavior. - Round native Q4 results only after the full quarter-warp reduction. Do not introduce sanitization into that path or change its Q8_K activation quantizer, block traversal or accumulation order. - Keep output-B in F32 for the graph's TP sum and final BF16 boundary. Preserve the standalone BF16 pass for mixed Q8-A paths, the existing Q8/Q8 delegation and scalar Q8-B dispatch. - Extend host regression coverage for both bridge modes, invalid modes, launch failures and mixed-type dispatch. Compare the actual fused helper against the previous sanitize/BF16 functions over 655,360 bit patterns, including NaNs, infinities, ties and finite overflow. Validation on macOS / Apple M1 Max / 32 GiB, using synthetic tensors: - make test-cuda-v41-q4-output-host: PASS with ASan/UBSan. - CPU and Metal production builds: PASS as part of this change series. - Strict C99 syntax for the native CUDA Q4 fixture and Linux test-target dry run: PASS. - git diff --cached --check: PASS. No NVIDIA GPU or nvcc is available locally. Host extraction checks do not establish native CUDA compilation, GPU arithmetic parity or performance. Hardware testers should run make test-cuda-v41-q4-output and make bench-cuda-v41-q4-output. No whole-model speedup is claimed.
V4.1 prefill previously dispatched the optional block sum, block BF16 rounding, HC expansion and output BF16 rounding separately. Add a CUDA kernel that performs the same ordered arithmetic while reusing each block value and four residual loads across all four output streams. - Specialize the existing 5120-wide, four-stream HC operation. Perform the optional block addition and BF16 boundary before the initial product, accumulate residual streams in their original order, then round output. Retain the build's FMA contraction policy, including --fmad=false. - Select the fused operation only for multirow CUDA prefill on an initialized single-GPU backend with TP world size one. Preserve decode, quality, imatrix, ROCm and multi-GPU dispatch behavior. - Share the existing Metal HC API declaration with CUDA. Validate sizes, alignment, pointer bounds, output/input overlap, tensor devices and current physical device before launching. Inputs remain immutable; the wrapper does not allocate, switch devices or synchronize the stream. - Add a native CUDA fixture comparing the complete legacy GPU chain with fused results, plus independent CPU checks, guards, aliases, invalid views and quality-mode rejection. Benchmark the actual old/fused work without counting fixture-only copies. - Add extracted production-kernel host tests under ASan/UBSan for both contracted and noncontracted arithmetic. Verify prefill-only graph selection, shared-expert ownership and failure propagation. - Add native test/benchmark and host-test Make targets; ignore and clean the generated executable. Validation on macOS / Apple M1 Max / 32 GiB: - make test-cuda-v41-hc-host: PASS for both arithmetic modes and graph dispatch, with ASan/UBSan. - CPU and Metal production builds: PASS for the complete change series. - Non-Apple and ROCm-preprocessed ds4.c syntax; strict C99 CUDA fixture syntax; Linux CUDA test-target dry run: PASS. - Metal scalar-epilogue, prefill-dispatch and SSD expert-stream tests: PASS. - git diff --cached --check: PASS. No native CUDA build, GPU numerical run or throughput measurement was possible on this machine. Hardware validation remains available through make test-cuda-v41-hc and make bench-cuda-v41-hc. Host tests and reduced launch counts do not establish an end-to-end prefill speedup.
The Q4 output-B workspace path rounded the low activation buffer to BF16 and then read it again to produce its FP16 RHS. Replace those two passes with one kernel that writes both representations while preserving the F32 -> BF16 -> F16 rounding sequence. - Process four values per thread with a scalar tail. Keep the BF16 value in F32 storage and convert that rounded value to half, preserving the existing double-rounding, NaN, infinity and signed-zero behavior. - Add a narrow helper with capacity, alignment, offset and overlap checks, including distinct no-copy Metal objects backed by the same CPU pages. - Resolve the fused pipeline and validate its buffers before output-A. Integrate it into the existing eligible transient Q4 output-B path: pre-M5 Apple Silicon, 256..2048 rows, TP1 and sufficient dead workspace. Retain the existing two-pass fallback when the fused pipeline is absent. - Reuse the current graph workspace and weight expansion without adding scratch allocations, tensor views, persistent caches or opt-in flags. - Add bitwise old/new GPU comparisons and independent integer references for BF16/F16 conversion, rounding edges, tails, guards and invalid views. Include a synchronized resident benchmark using the same physical buffers, warmup, alternating ABBA/BAAB order and 11 samples of 32 repeats. - Add test/benchmark Make targets and executable ignore/clean entries. Validation on Apple M1 Max / 32 GiB, using synthetic tensors: - CPU and Metal production builds: PASS for the complete change series. - tests/test_deepseek41_bf16_rhs --bench: correctness and guards PASS. - tests/test_deepseek41_outb: complete low-BF16/output parity PASS, including transient workspace use, queued reuse, alias rejection and SSD streaming. - tests/test_deepseek41_q4_attention: PASS. - tests/test_deepseek41_graph --scalar-epilogues and --prefill-expert-stream; tests/test_deepseek41_prefill --dispatch: PASS. - git diff --cached --check: PASS. Resident conversion-only median latency in microseconds, old -> fused: 256 rows: 56.734 -> 50.875 (1.115x) 512 rows: 99.750 -> 77.406 (1.289x) 1024 rows: 228.609 -> 150.172 (1.522x) 2048 rows: 667.781 -> 468.266 (1.426x) These timings cover warm, idempotent conversion on resident buffers and include dispatch/completion overhead. Smaller cases are noisier, and the public baseline wrapper has validation overhead. No full GGUF run or end-to-end prefill/decode throughput gain is established by these results.
6 participants
Updated 2026-09-11. PR head:
7a5002de.The latest full-model CUDA/ROCm reports below tested
d0bc6f07; the Metal Q4/Q8 report tested6a20b131. Subsequent runtime changes require validation on the final merge candidate.Summary
This PR supersedes #621 and adds support for the DeepSeek V4 Flash AProjQ4 layout: imatrix-guided requantization of 215 dense attention-projection tensors from
Q8_0toQ4_K.The matching model is 2.14 GiB smaller. Tests report faster generation on Metal, CUDA and ROCm, with results tied to the revisions and workloads below. The latest CUDA incremental-prefill retest shows that most of the earlier regression has been recovered. ROCm retains a measurable large-chunk prefill cost.
The conversion preserves the original
-0731checkpoint, expert and output-head quantization, tokenizer, GGUF metadata, and all unselected tensors. The selected families areattn_q_a,attn_q_b,attn_kv,attn_output_aandattn_output_b, across 43 layers.Beyond layout support and conversion, the current diff includes CPU/GPU execution changes, F16 compressor and aligned-Q8 paths, MoE and SSD-cache changes, tests and benchmark tooling. Later commits add execution-phase handling, HC normalization reuse and Metal indexer/top-k optimizations. The ROCm Q4 lookahead and prepared CUDA/ROCm indexer candidates remain outside automatic inference dispatch. #621 remains the earlier development archive.
CUDA implementation comparison — tested at
d0bc6f07On DGX Spark / GB10, @adamlawi tested
d0bc6f07against a production build based onad8149df+ PR #704 + two local CUDA stubs. Both used the same AProjQ4 GGUF; the baseline already contains the earlier #621 work. This measures the difference between those builds, not Q4 versus Q8 or this PR versus upstream main.The 4.5-hour server campaign used eight counterbalanced blocks, matching prompts and 400 generated tokens per request.
d0bc6f07decodeReported overall decode gain: +7.4% across 52 pairs. Returned content matched by SHA-256 in 52/52 pairs. The 300K prefill estimate is noisy. Average power rose about 1–2 W in this campaign, without thermal throttling. Full report
Incremental-prefill follow-up
An 80-run retest used 2,048-token increments, rotated arm order and 15 retained measurement blocks. The earlier approximately 12% Q4 slowdown has largely recovered: versus
8c22d667Q4,d0bc6f07Q4 is 1.0–1.1% slower at 4K–8K, and 2.2% slower at the first 2K frontier. The residual slowdown remains measurable.The same sweep now favors Q4 over Q8 for both prefill and decode, as shown below. It closes the requested retest for
d0bc6f07; it does not validate later revisions. Telemetry also found software power-cap events on baseline/production arms, correcting the earlier branch-specific interpretation. Retest and correctionModel-layout comparison: AProjQ4 versus AProjQ8
These comparisons use matching model layouts on the same engine revision. They are separate from the implementation comparison above; the percentages should not be added together.
6a20b131— historicald0bc6f07gfx1151, ROCm 7.1.1d0bc6f07Metal: four repeated campaigns used ratios calculated within each repetition and alternating model order. The corrected decode ratios were 1.158, 1.139, 1.152 and 1.159. These replace the earlier statistical interpretation; they do not establish performance on the current head. Corrected report
CUDA workload scope: the new row measures an incremental sweep. Earlier CLI tests at
f0541235, using code/prose prompts at 8K and 100K, measured +11.3% to +14.3% decode and −0.7% to +1.0% prefill. Keep these workload-specific results separate. Earlier CLI reportROCm: the latest reported retest, at
d0bc6f07, gives prefill −8.4% to −9.2% with chunk 4096 and −8.5% to −9.2% with chunk 8192; decode gains range from +10.9% to +11.6%. ROCm reportEnabling 8K batches improved absolute prefill throughput for both layouts by approximately 1.5–2.2%, without closing the relative Q4 deficit. Batch-path comparison
These performance reports use resident execution. They do not quantify expert SSD-streaming performance on a memory-constrained Mac.
Correctness and remaining validation
Historical quality checks on 100 cases / 2,313 target tokens found no measured regression on that fixture. The CUDA server test at
d0bc6f07additionally establishes returned-text identity for the tested Q4/Q4 pairs; neither result establishes universal quality equivalence or bit-identical logits.The later
0edc5534–7a5002decommits change production execution phases, HC paths and Metal indexer dispatch. Their documented host/native-Metal checks and isolated kernel timings are separate from the full-model results above; no full-model CUDA/ROCm report for7a5002dehas been posted in this discussion. Implementation and validation scopeBefore marking validation complete:
d0bc6f07, with the residual slowdown documented above.8c22d667, including two substantivedowndisagreements. The contributor now reports no remaining GB10 issues, but the latest benchmark report does not include this specific test's result. Original MMQ reportFor prefill results, record
ctx_tokens, actualprefill_tokens,--prefill-chunk, KV reuse and execution path. Counterbalance arm order. Prompt-file length and generation flags did not explain the earlier discrepancy; the actual prefill increment mattered. Methodology correctionThe five outdated inline review threads still need their disposition recorded; their unresolved status alone does not establish five current defects. Prepare the final series for squash as requested in review. Review
Reproduce the model and identify the artifact
Convert directly from the matching AProjQ8 GGUF; the original safetensors are not required:
For a dry run, omit
--outand add--dry-run. Strict mode rejects missing or invalid importance data for selected tensors. Unselected tensors are copied byte for byte.Artifact:
Exact sizes: AProjQ8 86,720,111,488 bytes; AProjQ4 84,420,584,288 bytes.
Thanks to @adamlawi for CUDA measurements and investigation, @iammac2 for repeated ROCm testing, @evandhoffman for Metal validation and methodology corrections, and @OPS-NeoRetro, @trueimage and @kyuz0 for review and contributions.