engram: parallelize DeepSeek V4.1 Flash decode reads on macOS

[Dango233]

This reads DeepSeek V4.1 Flash Engram decode rows in parallel on macOS. On this M2 Ultra, reading both tables takes 1.359 ms per token instead of 4.787 ms. Full 512-token decode throughput increases from 10.295 to 10.800 tokens/s on average, a 4.9% improvement.

Cause and change

Each token reads 24 rows from each of two Engram tables. The existing reader performs all 48 uncached reads serially. Sorting and deduplicating rows alone did not improve the measured workload; using four readers reduced the time spent waiting for reads.

DS4_ENGRAM_PARALLEL_DECODE=1 splits each 24-row call into four six-row calls to the existing reader. It is off by default. Inputs are validated before dispatch, each worker writes its own output and error slots, and all workers join before the call returns. Errors propagate to the caller; IO or data errors can leave partial output, as with the serial reader. The batched prefill reader and calls with other row counts keep their existing behavior.

Performance

M2 Ultra, 192 GiB, macOS 15.7.4, Metal; calibrated DeepSeek V4.1 Flash IQ2_XXS/Q2_K. Base: bd66c402070042bf0a79ad6ece8242de4c93680c. Each run uses the same binary, 2,048-token prompt, 8,257 allocated context, automatic expert cache, and 512 generated tokens.

Both variants use DS4_METAL_DISABLE_STREAMING_EXPERT_SLABS=1 to avoid the large-slab submission delay on this host. Only the Engram flag changes. Each run starts a fresh engine; the OS/file cache is not flushed.

Run	Parallel reads	Prefill t/s	Decode t/s
1	off	68.60	10.27
2	on	68.68	10.80
3	on	68.69	10.80
4	off	68.80	10.32

All four 512-token outputs match. The row-reading benchmark uses 128 deterministic row sets per table and eight alternating passes, checking every output bit. The 4.787/1.359 ms figures exclude the first pass.

Tests

• Serial and parallel Engram tests: row order, duplicates, signed zero, output guards, invalid input rejected before writing, EDOM in each worker partition, and EIO on short reads.
• Real-weight V4.1 sessions, snapshot restoration and cancellation with the flag enabled under Metal API validation.
• Clean Metal build, CPU compilation, and frontend, GGUF and quality-tool unit tests.

The build retains two existing SDK 15 warnings about unused Metal 4 symbols. The real-model tests use matching V4.1 fixtures; the full legacy make test suite was not run. Hardware validation covers this M2 Ultra.

Commands, model hash, raw CSVs and detailed results.