Skip to content

CUDA: use selected expert cache for IQ2 SSD prefill - #1031

Open
Matthley wants to merge 4 commits into
antirez:mainfrom
Matthley:fix/cuda-ssd-expert-cache
Open

CUDA: use selected expert cache for IQ2 SSD prefill#1031
Matthley wants to merge 4 commits into
antirez:mainfrom
Matthley:fix/cuda-ssd-expert-cache

Conversation

@Matthley

@Matthley Matthley commented Sep 12, 2026

Copy link
Copy Markdown

Fixes CUDA SSD streaming for routed MoE models.

  1. IQ2/Q2 batch prefill now uses the compact selected-expert table rather than resolving full expert tables.
  2. --ssd-streaming-cache-experts is a bounded CUDA LRU, released on model/lifecycle changes.
  3. Resident experts are stored in contiguous gate/up/down slabs. When the selected set is resident, kernels receive the slab bases plus remapped stable slot IDs, avoiding per-layer GPU-to-GPU copies into compact staging.
  4. CUDA now supports --expert-profile / DS4_EXPERT_HOTLIST to collect routed-expert locality data.

AProjQ4 matched prefill (GB10, e9528de, ctx 1024, forced batch prefill):

  • Before: 1.09 t/s; CUDA file cache 78.62 GiB.
  • Selected-expert path: 2.59 t/s; CUDA file cache 6.06 GiB.
  • Result: 2.38x prefill (+138%) and 92.3% less file cache.

AProjQ8 cache-size tuning smoke (GB10, ctx 1024, deterministic prompt; 22 emitted tokens):

Expert slots Expert cache Prefill Generation Hits Misses
256 1.69 GiB 2.85 t/s 1.62 t/s 755 18,056
512 3.38 GiB 2.87 t/s 2.05 t/s 5,027 13,784
1024 6.76 GiB 2.84 t/s 4.02 t/s 11,066 7,745
2048 13.52 GiB 2.65 t/s 6.24 t/s 13,809 5,002

For this model/host, 1024 slots is the measured throughput/memory compromise; 2048 favors generation throughput where the extra memory is available. These are single-run, host- and model-specific tuning observations, not a general performance claim.

Validation:

  • make -j4 cuda-spark: PASS
  • make test-cuda-q8-scratch: PASS
  • Deterministic SSD-streaming smoke: PASS, identical output.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant