Add Q4-dense variant: DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ4-OutQ8-chat-v2-imatrix.gguf

[GiorgioOppo]

Context / Motivation

On memory-constrained Macs (16 GB) with SSD streaming, decoding the Flash IQ2XXS model (81 GB) is ~2× slower than it needs to be. Root cause: the dense / non-routed weights are Q8 and take ~8.2 GiB resident, which crushes the dynamic expert cache down to 1.27 GiB — below a single token's routed working set (1.70 GiB) → thrashing (experts are re-streamed from SSD every token).

Moving the dense tensors from Q8 → Q4 frees ~2.6 GiB: the expert cache now fits under the RAM ceiling without swapping → higher hit rate → faster decode. This mirrors what the Swift app does (Q4 dense).

What changes vs. the current GGUF

Only the type of the dense tensors; routed experts and everything else are identical:

Component | Current (Q8) | New (Q4) -- | -- | -- attention projections (attn_q_a/q_b/kv/output_a/output_b) | Q8_0 | Q4_K shared experts (ffn_{gate,up,down}_shexp) | Q8_0 | Q4_K routed experts (ffn_{gate,up,down}_exps) | IQ2_XXS / Q2_K | unchanged output / embedding / norm / indexer | unchanged | unchanged

Coherence: native output is coherent (verified with greedy generation).

Expected gain

• ~2.1× decode throughput in SSD streaming on memory-constrained (16 GB) machines.
• Modest quality cost: ~+3.4% perplexity, with greedy next-token choice essentially unchanged (86.1% → 85.9%). The model picks the same tokens, just slightly less confident.
• No engine changes required to run it (Q4_K dense is already supported via matvec_q4_k).

Notes

• The requant was done with a NULL imatrix; using --imatrix <file.dat> (the same one used for the base model) would further shrink the quality gap — recommended for an official release.
• Q4 dense is lossy: an intentional speed/quality trade-off.

[GiorgioOppo]

Update: recommendation revised → AProjQ4-SExpQ8 (keep shared experts at Q8)

Follow-up with full data. Three findings since the original write-up — ending with a better variant than the one proposed above, so I'd retitle the proposal to ...AProjQ4-SExpQ8-OutQ8....

1. Dense imatrix (collector extended, gap halved)

The stock --imatrix-out collector only covers routed MoE experts, so the model's existing imatrix has no vectors for the dense tensors this requant touches. I extended the collector to also accumulate the dense-matmul inputs (attention projections + shared experts) and wired --requant-gguf to consume it. Imatrix-weighted Q4 cut the quality gap roughly in half: +3.38% → +1.38% perplexity vs Q8.

Side finding: the imatrix benefit saturates fast at 4-bit — going from 32k calibration tokens to 220k diverse tokens (Italian prose + English docs + C code) changed nothing (0.387 vs 0.389 avg_nll, within noise). A small calibration set is enough.

2. Behavioral drift found: Q4 shared experts flip the reasoning language

Controlled A/B (same Italian prompt, greedy, deterministic): the Q8 model thinks in Italian, the all-Q4 model thinks in Chinese (the final answer is still Italian and correct). Root cause: shared experts fire on every token of every layer, and the reasoning-language choice is a near-tie fork at the first thinking token — the Q4 error on the shared experts is enough to flip it, and greedy decoding cascades the flip. Invisible in aggregate metrics (top-1 stays ~86%), but very visible to users.

3. Fix: mixed variant AProjQ4-SExpQ8-OutQ8

Quantize only the 5 attention projections (215 tensors); keep the shared experts at Q8 (copied byte-for-byte). Costs ~0.55 GiB extra resident — the expert cache still fits.

# 1. collect a dense-aware imatrix (any ~32k-token text)
./ds4 --metal --ssd-streaming --ssd-streaming-cache-experts 512 -m MODEL-Q8.gguf \
  --imatrix-dataset calibration.txt --imatrix-out dense-imatrix.dat

# 2. requant (GGUF -> GGUF, no HF source needed)
./gguf-tools/deepseek4-quantize --requant-gguf \
  --template MODEL-Q8.gguf \
  --out DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
  --attention-proj q4_k --imatrix dense-imatrix.dat --threads 8

# 3. run (no env flags; explicit expert cache)
./ds4 --metal --ssd-streaming --ssd-streaming-cache-experts 448 -m MODEL-AProjQ4-SExpQ8.gguf

Quality — 100 official Flash continuations (2289 target tokens)

variant	avg_nll ↓	ppl vs Q8	top-1	thinks in user's language
Q8 (baseline)	0.3732	—	86.1%	✅
all-Q4, no imatrix	0.4065	+3.38%	85.9%	❌ (Chinese)
all-Q4, imatrix	0.3870	+1.38%	86.2%	❌ (Chinese)
mixed AProjQ4-SExpQ8	0.3831	+0.99%	86.4%	✅

The mixed variant also wins 56/100 cases head-to-head against the imatrix all-Q4.

Performance — clean 3-way, same session, no expert bundle (M1 Pro 16 GB)

ctx	metric	Q8	all-Q4	mixed	mixed vs Q8
512	decode t/s	0.73	2.06	2.06	2.8×
512	first-token	1.90 s	1.22 s	1.16 s	−39%
2048	decode t/s	0.56	2.02	1.79	3.2×
—	prefill t/s	11–19	14–19	15–19	≥ parity

Identical to all-Q4 at ctx 512; within variance at 2048 (it runs a slightly smaller expert cache to pay for the +0.55 GiB resident shared experts).

Bottom line

Ship AProjQ4-SExpQ8-OutQ8: 2.8–3.2× decode, +0.99% perplexity, top-1 unchanged, and no reasoning-language drift. The all-Q4 variant remains an option only where the extra 0.55 GiB of resident memory matters more than the drift.

[GiorgioOppo]

python3 gguf-tools/quality-testing/compare_scores.py /tmp/old_full.tsv /tmp/mixed_q.tsv

cases 100
tokens 2289
old_avg_nll 0.373209999
new_avg_nll 0.383090306
delta_new_minus_old 0.009880307
relative_nll_change 2.647%
case_wins_new_old_ties 44 56 0
first_token_matches_old_new 67 66
avg_greedy_lcp_old_new 6.450 6.020
old_api_target_tokens 2289
old_api_target_mae 0.373209999
old_api_target_mean_delta -0.373209999
old_api_top_coverage 0.828658803
old_api_top1_rate 0.860637833
old_api_topn_recall 0.320856179
old_api_top_mae 7566.523196211
old_api_top_mean_delta 7566.343044864
old_api_pair_rate 0.988622173
new_api_target_tokens 2289
new_api_target_mae 0.383090306
new_api_target_mean_delta -0.383090306
new_api_top_coverage 0.828658803
new_api_top1_rate 0.864132809
new_api_topn_recall 0.319485449
new_api_top_mae 7566.806415708
new_api_top_mean_delta 7566.621495060
new_api_pair_rate 0.988899681

new best cases:
case_068 delta_nll=-5.682944 tokens=24 old=0.794976 new=0.558186
case_055 delta_nll=-4.779023 tokens=24 old=1.092227 new=0.893101
case_039 delta_nll=-3.543982 tokens=24 old=0.736604 new=0.588939
case_061 delta_nll=-2.416818 tokens=24 old=0.522727 new=0.422026
case_088 delta_nll=-2.262405 tokens=24 old=0.529073 new=0.434806
case_069 delta_nll=-1.976598 tokens=24 old=0.416514 new=0.334156
case_040 delta_nll=-1.965485 tokens=24 old=0.291215 new=0.209320
case_090 delta_nll=-1.949768 tokens=24 old=0.557764 new=0.476524

old best cases:
case_007 delta_nll=4.806138 tokens=24 old=0.432655 new=0.632910
case_082 delta_nll=3.734868 tokens=24 old=0.164310 new=0.319929
case_057 delta_nll=3.149504 tokens=24 old=0.462620 new=0.593850
case_080 delta_nll=3.097285 tokens=24 old=0.867047 new=0.996101
case_071 delta_nll=3.070840 tokens=24 old=0.430651 new=0.558603
case_087 delta_nll=2.557579 tokens=24 old=0.253981 new=0.360547
case_066 delta_nll=2.472013 tokens=24 old=0.507912 new=0.610912

[GiorgioOppo]

https://huggingface.co/antirez/deepseek-v4-gguf/discussions/22

[GiorgioOppo]

Call for testers

I'd love some help from the community validating this PR before it moves forward, both for correctness and speed. I've only been able to test on a single machine (M1 Pro, 16 GB, SSD streaming), so coverage on other configurations would be extremely valuable.

Metal, high-RAM Macs (≥96/256 GB, full residency — no SSD streaming). This is the project's primary target and the config I could not test at all. The decode-graph changes apply there too, so I'd like confirmation that:

the AProjQ4 GGUF (DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix.gguf) loads and produces coherent output with --temp 0, and --decode-consistency 3 reports max_abs=0 rms=0;
the existing AProjQ8 GGUFs produce bit-identical tokens and unchanged tok/s compared to a main build — the Q8 paths were deliberately left on the exact kernels they always used, and I'd like that verified where it matters most.
Metal, SSD streaming (any RAM size). Same correctness checks as above, plus speed A/B against main with and without the new opt-in flags (DS4_METAL_STREAMING_EXPERT_NOCACHE=1, DS4_METAL_STREAMING_EXPERT_PREAD_SPLIT=4). On my 16 GB machine they take decode from ~1.9 to ~2.2 tok/s and make it stable under page-cache pressure; both are off by default, and the defaults should be perfectly neutral.

CUDA (single and multi-GPU, DGX Spark, TP setups). Heads-up: this PR does not implement dense Q4_K attention projections on the CUDA backend. What I'd like verified there is:

loading the AProjQ4 GGUF fails with a clear "unsupported type" error instead of producing garbage (the decode-TP attention split now also refuses non-Q8 output projections explicitly);
the AProjQ8 GGUFs still work bit-identically and at unchanged speed vs main;
if anyone wants to take a stab at the CUDA Q4_K dense kernels, the Metal side can serve as a reference for the dispatch structure.
ROCm / Strix Halo. Untested by me entirely — any run at all (Q8 regression check + AProjQ4 behavior) would be useful.

If you can spare a run, please post your hardware, backend, model file, the exact command, and the prefill/generation numbers ds4 reports. Interleaved runs (baseline, PR, baseline, PR) help a lot to filter out page-cache noise. Thanks!

[GiorgioOppo]

Update: dense Q4_K support now implemented for CUDA and ROCm as well — untested, testers wanted

Since the call for testers above, the branch has grown support for the remaining backends (commit a4a4e5d).

Backend	Implementation	Testing status
Metal	complete	✅ tested (M1 Pro 16 GB)
CPU	complete	⚠️ untested
CUDA	complete (single-GPU)	⚠️ untested, not compile-tested
ROCm	same CUDA source via HIP	⚠️ untested

Details:

• Metal: coherent greedy output on the AProjQ4 GGUF, --decode-consistency reports max_abs=0 rms=0, and the AProjQ8 GGUFs stay bit-identical to main.
• CPU: new matvec_q4_K family — activations prequantized to Q8_K, rows reduced with the same dot as the routed-MoE path.
• CUDA/ROCm: one new dense kernel that quantizes activations to Q8_K with the existing routed-MoE quantizer and reduces each weight row with the already-validated Q4_K×Q8_K superblock dot, wired into ds4_gpu_matmul_quant_tensor as type 12. Where a backend lacks a specialized Q4_K attention-output kernel, the shared graph falls back to the generic per-group dense path instead of failing the layer. The decode-TP attention split and the fused kslice paths still refuse non-Q8 projections with a clear error — never garbage. CUDA prefill reuses the same matvec kernel: correct but unoptimized (a dequant+GEMM path can follow if profiles justify it). Authored on a Mac with no NVIDIA toolchain, hence not even compile-tested.

This supersedes the CUDA paragraph in my earlier call for testers. What I'd like verified now:

• CUDA / ROCm build: does it even compile? (make / the ROCm target) — this is the first thing that can fail, please report compiler errors verbatim.
• CUDA / ROCm runtime: AProjQ4 GGUF → coherent output with --temp 0, and --decode-consistency 3 → max_abs=0 rms=0; AProjQ8 GGUF → bit-identical tokens and unchanged tok/s vs a main build.
• Multi-GPU TP setups: AProjQ4 should refuse with a clear error message, not produce garbage.
• CPU: --cpu (or --first-token-test on a Metal build) with the AProjQ4 GGUF exercises the new CPU reference path — it's slow, a handful of coherent tokens is enough to confirm.

As before: hardware, backend, exact command, and the prefill/generation numbers ds4 reports in the replies, please. Thanks!

[GiorgioOppo]

#621