asr: allocate inference graphs with ggml_gallocr, not alloc_ctx_tensors

[derekja]

Summary

Several ASR inference graphs are built with no_alloc = true and then allocated with ggml_backend_alloc_ctx_tensors, which allocates every tensor in the context simultaneously. Intermediate activations for all layers stay resident for the whole graph instead of being recycled, so peak memory scales with sequence length at roughly 50× the legitimate cost.

This switches the affected graphs to ggml_gallocr, which this repository already uses elsewhere — including in qwen3_asr/audio_encoder.cpp and hviske_asr/encoder.cpp, one file away from two of the sites changed here.

Output is unchanged: 22 transcript comparisons between the two builds are byte-identical.

The problem

ggml_init_params params{arena, nullptr, true};   // no_alloc
...
graph_ = ggml_new_graph_custom(ctx_.get(), N, false);
ggml_build_forward_expand(graph_, output_);
buffer_ = ggml_backend_alloc_ctx_tensors(ctx_.get(), backend);   // every tensor, no reuse

Measured on a 12 GB H100 slice with Qwen3-ASR 0.6B, 32 s of audio and a 3000-character recognition prompt, this asks for a single cudaMalloc of 9739 MiB for 886 prompt steps:

alloc_tensor_range: failed to allocate CUDA0 buffer of size 10212568704

That is 11.0 MiB per token, where the KV cache for a 0.6 B model across 28 layers is about 224 KB per token. It fails as one monolithic allocation rather than as gradual pressure, so it is not a fragmentation issue and more free VRAM does not help proportionally.

Results

Peak GPU, sampled at ~11 ms intervals, fresh process per measurement, three repetitions per point (deterministic to the MiB).

citrinet_asr — encoder, single output, no KV cache

audio	before	after
4 s	492 MiB	392 MiB	−20%
8 s	588 MiB	396 MiB	−33%
16 s	780 MiB	398 MiB	−49%
32 s	1163 MiB	404 MiB	−65%
slope	23.96 MiB/s	0.43 MiB/s	56× flatter

qwen3_asr — audio length, no prompt

audio	0.6B before	0.6B after	1.7B before	1.7B after
4 s	2197	1867	3632	3158
8 s	2617	1991	4138	3252
16 s	3594	2263	5344	3526
32 s	5660	2741	7813	4006
slope	123.7 MiB/s	31.2	149.3 MiB/s	30.3

qwen3_asr 0.6B — recognition-prompt length, fixed 8 s clip

prompt	before	after
0 chars	2617 MiB	1991 MiB
898	3767 MiB	2127 MiB
1795	5218 MiB	2271 MiB
2996	7582 MiB	2463 MiB
slope	1.657 MiB/char	0.158 MiB/char (10.5×)

Previously failing, now working

case	before	after
0.6B, 32 s + 2996-char prompt	OOM (9739 MiB request)	3227 MiB
1.7B, 32 s + 1795-char prompt	OOM (8246 MiB request)	4294 MiB

Two details that are not just swapping the allocator

K/V readback. PrefillGraph hands run() the decoder's K/V state, which is an intermediate the allocator would recycle. Marking those tensors as outputs is not sufficient — they may be views, and GGML_TENSOR_FLAG_OUTPUT on a view does not protect its view_src; ggml_gallocr_free_node frees view_src on its own flag. Each is copied into a tensor of its own with ggml_cpy over ggml_dup_tensor, and the copy is marked as an output. This matches the existing pattern in framework/modules/transformers/qwen_causal_decode_runtime.cpp.

Constant leaves. Both qwen3_asr graphs uploaded position ids once at build time. That is safe when every tensor is pinned, but the graph allocator may reuse a leaf once its last consumer has run, so a second run() on a cached graph would read stale positions. The upload moves into run(). Anyone converting the remaining sites should audit for this specifically.

Deliberately not changed

• DecodeGraph in qwen3_asr/thinker.cpp keeps alloc_ctx_tensors. Its context holds a persistent step cache that must retain stable storage across successive decode steps, and its per-step activations are one token wide, so it does not contribute to length scaling. Converting it would mean splitting the cache into its own buffer.
• hviske_asr/decoder.cpp, for a similar reason: PrefillGraph builds views onto prefill K/V that separate DecodeGraph/BeamDecodeGraph contexts consume, needing stable addresses across graph boundaries. Its memory is also flat with audio length (fixed 30 s padded window), so it does not show the symptom.

Correctness

22 comparisons of the CLI text_output line, baseline binary vs patched, same clip and flags — all byte-identical:

• citrinet at 4/8/16/32 s
• qwen3 0.6B and 1.7B at 4/8/16/32 s
• qwen3 0.6B at 8 s × prompts of 499/898/1795/2996 chars
• hviske at 4/32 s (unchanged code, both builds)
• CPU backend (--backend cpu), exercising the uses_host_graph_plan path
• 3-request batch runs (--batch-audio-dir), exercising graph-cache reuse via matches() — the path the positions_ change protects

Possibly affected elsewhere

51 .cpp files build graphs and also call ggml_backend_alloc_ctx_tensors. Many of those calls are legitimate — weight contexts and persistent caches — so this is a triage list rather than a bug list. Files that build graphs with no gallocr anywhere include vibevoice/decoder.cpp, higgs_audio_tts/ar.cpp, qwen3_tts/talker.cpp, ace_step/condition_encoder.cpp, vevo2/ar.cpp, miotts/causal_lm.cpp, and several single-site runtimes. I have not measured those and make no claim about them here.

[0xShug0]

@derekja Thank you for the PR! Using ggml_backend_alloc_ctx_tensors for large graphs is a legacy pattern from the early days of audio.cpp. I've only changed a few models so far, mainly in response to user requests to reduce VRAM usage. Good to see PRs taking care of the older models!

Could you share a few RTF comparisons from before and after the PR?