From 7032f514bbd036058f018bfb9f3287cc0ef110ae Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Sat, 3 Oct 2026 19:40:04 -0700 Subject: [PATCH 01/46] dasllama-server: an ASR model whose decoder is an LLM decodes on the device, an ASR model serves alone, and served_bench sends every transcription a clip no server has heard The ASR worker's context carried no Metal mode, so a two-file model (Qwen3-ASR, Canary, the gemma audio route) prefilled and decoded on the CPU. The worker takes the engine's mode; a one-file model keeps its planar CPU decode. /v1/stats names each ASR model's decoder under asr.models. A boot with --asr and no LLM serves instead of entering setup mode, and its stats count its workers and clips. served_bench uploads a copy of the clip with one sample moved: the reference server answers a repeated clip from its prompt cache. Co-Authored-By: Claude Fable 5.1 --- .../reference/utils/dasllama_server.rst | 11 +++-- .../ARCHITECTURE_MEASUREMENT_SERVED.md | 5 +- modules/dasLLAMA/PERF_LEDGER.md | 18 +++++++ modules/dasLLAMA/followup_metal.md | 8 --- modules/dasLLAMA/harness/served_bench.das | 29 ++++++++++- modules/dasLLAMA/tests/CLAUDE.md | 4 +- modules/dasLLAMA/tests/test_served_bench.das | 38 ++++++++++++++ utils/dasllama-server/README.md | 21 +++++--- utils/dasllama-server/main.das | 23 ++++----- utils/dasllama-server/openai_server.das | 49 ++++++++++++++----- utils/dasllama-server/test_openai_server.das | 2 +- .../test_openai_server_speech_multi.das | 49 ++++++++++++++++++- 12 files changed, 209 insertions(+), 48 deletions(-) diff --git a/doc/source/reference/utils/dasllama_server.rst b/doc/source/reference/utils/dasllama_server.rst index 6ae6b53017..8ce66cfd42 100644 --- a/doc/source/reference/utils/dasllama_server.rst +++ b/doc/source/reference/utils/dasllama_server.rst @@ -432,7 +432,7 @@ Setup mode and the model catalog ================================ A start with no LLM model at all - no ``--model``, no config, or every -configured path missing - and no servable ``--tts`` boots into setup mode: the +configured path missing - and no servable ``--tts`` or ``--asr`` boots into setup mode: the port opens, the control page serves, every inference route answers with a clean error, and the page leads with the model catalog - a curated, sha-pinned list of current models (``model_catalog.das``, a view over the module's one @@ -452,8 +452,13 @@ wears a fit badge (fits gpu / fits / tight / too big) from the box facts the ``/catalog`` document carries; the advertised working set is a hint, not a load gate. -``--tts`` alone is a serving start, not a setup start: a speech-only server has -no LLM slot, so ``/v1/stats`` answers the slotless shape with ``setup`` false. +``--tts`` or ``--asr`` alone is a serving start, not a setup start: a speech-only +server has no LLM slot, so ``/v1/stats`` answers the slotless shape with ``setup`` +false and its speech counters live. An ASR model whose decoder is an LLM - a +two-file model, ``--asr`` with ``--mmproj`` - decodes where the server's LLMs do: +its worker takes the boot's Metal mode, and ``/v1/stats`` names the outcome under +``asr.models[]`` as ``decoder`` (``metal`` or ``cpu``). A one-file model (whisper, +parakeet) decodes on the CPU. A ``--tts`` the server cannot serve degrades, it does not die: a missing file or a failed load is logged, the speech route is dropped, and the boot serves whatever is left - the LLM slots if any loaded, else setup mode. diff --git a/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT_SERVED.md b/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT_SERVED.md index 5a3bbe910b..b8eabd3240 100644 --- a/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT_SERVED.md +++ b/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT_SERVED.md @@ -33,7 +33,10 @@ cached token count differs between reps. Its speech rows time a whole transcription request - the clip uploaded as a multipart form to `/v1/audio/transcriptions` - one row a clip, over all reps behind one untimed request, every rep -reading the same text. A second row a clip is timed while a chat turn decodes at the chat +reading the same text. Every request uploads a copy of the clip with one sample moved, the sample +picked off the clock: a server that keeps the prompts it has evaluated answers a clip it has heard +from that cache - audio rows included - and a transcription service is not sent the same audio +twice. A clip that is no PCM WAV goes up as it is. A second row a clip is timed while a chat turn decodes at the chat endpoint: the reps start at that turn's first reply text, and the row is refused when the turn ends before they do or the text differs from the idle row's. Where the speech encoder runs - beside the chat model on the GPU or on the CPU team - is the server's launch, not the diff --git a/modules/dasLLAMA/PERF_LEDGER.md b/modules/dasLLAMA/PERF_LEDGER.md index db526bb147..bfb3521b46 100644 --- a/modules/dasLLAMA/PERF_LEDGER.md +++ b/modules/dasLLAMA/PERF_LEDGER.md @@ -11,6 +11,24 @@ what it costs today and what the fix would change. ## Entries +- **MEASURED (2026-10-03, `direction-grade`) - a served Qwen3-ASR transcription decodes on the device, and a + repeated clip measures the other server's prompt cache.** M5 Max, Metal, `dasllama-server` with the + ASR model alone (`main.das -- --asr --mmproj `), the pinned llama-server as + `-m --mmproj -ngl 99 -np 1`, `harness/served_bench.das --no-chat --asr-url + --clip --reps 5` (tagged `served`, `out-of-process`), a 60 s rest before each life; the tune + sidecar predates the binary, so the CPU kernels ran their fallback stamps. + - The ASR worker's context carried no Metal mode, so the decoder - a `Model` session - prefilled + and decoded on the CPU: Qwen3-ASR-0.6B Q8_0 read a 4 s clip in 127.5 ms. The worker takes the + engine's mode for a two-file model: 50.5 ms. Inside it (the worker's own stage clock): encode + 10.7 ms, the 75-row prefill 6.7, twelve decode steps 29.0 - 2.4 ms a token, where the reference's + own timing line reads 2.65. + - The reference keeps the prompts it has evaluated, audio rows included: the same clip sent again + costs it one prompt token (`cached_tokens` 79 of 80) and reads 37.4 ms, and a clip one sample + off costs the whole prompt. The instrument's rows before this entry repeated one clip and so + read that cache; every request now uploads a clip no server has heard. + - Unseen clips, ours / the reference, ms a transcription: Qwen3-ASR-0.6B 4 s 50.5 / 60.7, 8 s + 76.6 / 98.1 (the reference's cv 6.1%, void); Qwen3-ASR-1.7B 4 s 85.5 / 95.1, 8 s 130.8 / 164.4. + - **MEASURED (2026-10-02) - the K/V mirror's block codecs on the whole-model driver (`ARCHITECTURE_GPU_VULKAN_ATTN.md#vk-kv-block-codecs`, `set_gpu_kv_dtype` / the server's `--kv-dtype`): a q8_0 mirror is 17/32 of the f16 mirror's bytes and tq4 9/32, and on a model whose context was asked whole diff --git a/modules/dasLLAMA/followup_metal.md b/modules/dasLLAMA/followup_metal.md index 9824de02df..4507705985 100644 --- a/modules/dasLLAMA/followup_metal.md +++ b/modules/dasLLAMA/followup_metal.md @@ -878,14 +878,6 @@ Llama-3.2-3B, M5 Max), and leaves the decode driver to upload them again. Three releases the slice and takes a larger one). The work: the old slice's rows copied device-side into the new one. -## 37. A served Qwen3-ASR transcription runs its decoder on the CPU - -The server's ASR worker context carries no Metal mode, so a family whose decoder is an LLM -(Qwen3-ASR) prefills and decodes it on the CPU arm while the reference server runs the same file -offloaded. The row is `harness/served_bench.das --asr-url --clip ` on both servers. -The work: the worker serving the decoder on the device where one is armed, and that row beside -the reference's. - ## 38. Two ASR models transcribing at once slow each other Two ASR models asked for the same clip at the same instant each take several times their idle diff --git a/modules/dasLLAMA/harness/served_bench.das b/modules/dasLLAMA/harness/served_bench.das index 80e60e6573..f17844f90e 100644 --- a/modules/dasLLAMA/harness/served_bench.das +++ b/modules/dasLLAMA/harness/served_bench.das @@ -483,12 +483,37 @@ def private chat_rows(plan : Plan; corpus, sizes : string; settle : int; fixed_c return tally } -//! one whole transcription request under the client clock, the clip uploaded as a multipart form +//! A copy of `clip` no server has heard - one sample, picked off the clock, moved - as a 16-bit temp file; +//! "" when `clip` is no PCM WAV or the copy cannot be written. +def unseen_clip(clip : string) : string { + var samples : array + var rate = 0 + var channels = 0 + var path = "" + if (read_wav(clip, samples, rate, channels) && !empty(samples)) { + let ticks = ref_time_ticks() + let at = ticks % long_length(samples) + samples[at] += samples[at] < 0.5 ? 2.0 / 32767.0 : -2.0 / 32767.0 // two steps: write_wav truncates, and one would round back + var err = "" + path = path_join(temp_directory(err), "served_bench_{ticks}.wav") + if (!write_wav(path, samples, uint(rate), channels)) { + path = "" + } + } + delete samples + return path +} + +//! one whole transcription request under the client clock, an unseen copy of the clip uploaded as a multipart form def transcribe(speech : Speech; clip : string) : Heard { var heard = Heard() + let unseen = unseen_clip(clip) let t0 = ref_time_ticks() - let res = openai_audio::transcribe(client_of(speech.url), speech.model, clip) + let res = openai_audio::transcribe(client_of(speech.url), speech.model, empty(unseen) ? clip : unseen) heard.wall_us = int64(get_time_usec(t0)) + if (!empty(unseen)) { + remove(unseen) + } if (!res.ok) { heard.why = "HTTP {res.error.status}" return heard diff --git a/modules/dasLLAMA/tests/CLAUDE.md b/modules/dasLLAMA/tests/CLAUDE.md index 997613465a..a258fde81f 100644 --- a/modules/dasLLAMA/tests/CLAUDE.md +++ b/modules/dasLLAMA/tests/CLAUDE.md @@ -1773,7 +1773,9 @@ stream (counts, both spans past the server's 30 ms pauses, the server's own TTFT refusals - no usage chunk, no reply text, a reply of one token (no decode span to time), HTTP 500 - plus no server at the address, the follow-up row standing over complete turns and refused by a failed one, and the image cells - a file as a data URI and a missing one as none, an image question's wall and prompt tokens off the fake's buffered completion, HTTP 500 saying why, an image row standing over a file and refused without one. The speech rows run against a fake speech server on a second thread: a transcription's wall holds the server's -300 ms pause, a failed and a blank answer say why, a row answers the text its reps read and refuses +300 ms pause, a failed and a blank answer say why, every request of a row uploads a clip the fake +has not heard (a WAV's four reps read `new` four times; a clip that is no WAV goes up as it is, the +fake hears it twice and the row is refused), a row answers the text its reps read and refuses reps that read different texts or a text other than the one asked, and the loaded row stands while the fake chat turn outlasts two fast reps and is refused when the reps outlive the turn, when the turn carries no reply text, and when the loaded text differs from the idle one. The synthesis and diff --git a/modules/dasLLAMA/tests/test_served_bench.das b/modules/dasLLAMA/tests/test_served_bench.das index 24ecb2db18..1039f5d3ab 100644 --- a/modules/dasLLAMA/tests/test_served_bench.das +++ b/modules/dasLLAMA/tests/test_served_bench.das @@ -57,6 +57,28 @@ class FakeChatServer : HvWebServer { } var g_takes = 0 //! the speech fake's request count, in its own thread's context +var g_heard : table //! every upload the speech fake's `seen` model was sent, in its own thread's context + +//! what tells one uploaded clip from another: the indices of its samples that are not silence, "" for an upload that is no WAV +def private upload_key(var req : HttpRequest?) : string { + var err = "" + let path = create_temp_file("served_bench_upload", ".wav", err) + save_form_file(req, "file", path) + var samples : array + var rate = 0 + var channels = 0 + read_wav(path, samples, rate, channels) + remove(path) + let key = build_string() $(writer) { + for (i, s in count(), samples) { + if (s != 0.0) { + writer |> write("{i} ") + } + } + } + delete samples + return key +} class FakeSpeechServer : HvWebServer { def override onInit { @@ -75,6 +97,12 @@ class FakeSpeechServer : HvWebServer { if (model == "slow") { sleep(300u) } + if (model == "seen") { + let upload = upload_key(req) + let again = g_heard |> key_exists(upload) + g_heard |> insert(upload) + return resp |> JSON("\{\"text\":\"{again ? "heard before" : "new"}\"}") + } return resp |> JSON("\{\"text\":\"hello world\"}") } POST("/v1/audio/speech") <| @(var req : HttpRequest?; var resp : HttpResponse?) : http_status { @@ -135,6 +163,16 @@ def test_speech_rows_against_fakes(t : T?) { t |> equal(clip_row(Speech(url = speech_url, model = "drift"), clip, "asr clip", "", 3), "", "three reps, three texts") t |> equal(clip_row(Speech(url = speech_url, model = "fail"), clip, "asr clip", "", 3), "", "a failed rep refuses the row") } + t |> run("every request uploads a clip the server has not heard; a clip that is no WAV goes up as it is") @(t : T?) { + let wav = silent_wav(160000) // ten seconds: the copies differ in one sample picked off the clock + t |> equal(clip_row(Speech(url = speech_url, model = "seen"), wav, "asr clip", "", 4), "new", "four reps, four uploads the fake never saw") + let copy = unseen_clip(wav) + t |> success(!empty(copy) && copy != wav && stat(copy).size == stat(wav).size, "the copy is a file of its own, the clip's size") + remove(copy) + remove(wav) + t |> equal(unseen_clip(clip), "", "no WAV, no copy") + t |> equal(clip_row(Speech(url = speech_url, model = "seen"), clip, "asr clip", "", 2), "", "the same bytes twice: the fake's second answer differs and the row is refused") + } with_test_server(type) $(chat_url) { t |> run("a loaded row stands while the chat turn outlasts its reps") @(t : T?) { let load = LoadJob(url = chat_url, model = "ok", question = "ask", budget = 8) diff --git a/utils/dasllama-server/README.md b/utils/dasllama-server/README.md index d77b99ffb7..0f2ca51669 100644 --- a/utils/dasllama-server/README.md +++ b/utils/dasllama-server/README.md @@ -285,7 +285,7 @@ table's value, as a flag overrides any key of a config that is not authoritative ## Setup mode and the model catalog A start with **no LLM model at all** (no `--model`, no config, or every configured path -missing) **and no servable `--tts`** does not exit - it boots into **setup mode**: the port +missing) **and no servable `--tts` or `--asr`** does not exit - it boots into **setup mode**: the port opens, the control page serves, every inference route answers with a clean error, and the page leads with the **model catalog** (sec. 03) - a curated, sha-pinned list of current models (`model_catalog.das`; commit-pinned @@ -297,9 +297,14 @@ model** - the page writes the config and restarts (exit 4, the watchdog contract into serving it. The same catalog stays available on a serving server for pulling more models. -**`--tts` alone is a serving start, not a setup start.** A speech-only server has no LLM slot, -so `/v1/stats` answers the slotless shape - but its `setup` reads `false`, and the page keeps -its serving surface instead of leading with the catalog. +**`--tts` or `--asr` alone is a serving start, not a setup start.** A speech-only server has no +LLM slot, so `/v1/stats` answers the slotless shape - but its `setup` reads `false`, its ASR and +TTS counters are live, and the page keeps its serving surface instead of leading with the catalog. + +**An ASR model whose decoder is an LLM decodes where the server's LLMs do.** A two-file model +(`--asr` with `--mmproj`: Qwen3-ASR, Canary, the gemma audio route) takes the boot's Metal mode in +its worker; a one-file model (whisper, parakeet) decodes on the CPU. `/v1/stats` says which under +`asr.models[]` - `id`, `workers`, `ready`, and `decoder` (`metal` or `cpu`). **A `--tts` the server cannot serve degrades, it does not die.** A missing file (the GGUF, or `tts_g2p.bin` / `tts_postag.bin` beside it) is logged as an error before the worker starts, and @@ -475,7 +480,7 @@ DLLs; the config and the bundle's tune state live in `~/.dasllama` and survive t | `POST` | `/bake` | `{"model"?: name}` loopback-only: bake the slot's prepared `.dlim` image by spawning `dasllama-convert` (empty body bakes the default slot; 409 while a bake or bench runs or streams are active; the dlim GC of never-loadable images runs on completion) | | `GET` | `/bake` | Bake state (`idle | running | done | failed`), the slot it runs for, log lines, the result JSON | | `GET` | `/v1/images` | Per-slot prepared-image inventory: source GGUF path, the flavor THIS process mapped (planar/vulkan/metal, or raw gguf), the trimmed flag, and each on-disk `.dlim`'s info - plus the slot name a bake is currently running for | -| `GET` | `/v1/stats` | Scheduler counters (`gen_tokens`, `prefill_tokens`, TTFT last/avg, ...) plus `model`/`active_model`/`ctx`/`uptime_s`/`draining` identity fields, memory footprint (`weights_bytes`, `kv_bytes`, das heaps, `gpu_vram_bytes`/`gpu_budget_bytes`), `gpu_cpu_passes` (the calls the armed GPU path handed back to the CPU since the model armed, `{reason, words, count}` per reason that fired - `words` is the reason as the control page prints it; empty means the device served every call), a `hardware` line (CPU * lanes * GPU), `asr_workers`, `asr_ready`, `asr_active`, `asr_pending`, speech counters (`tts_done_jobs` - syntheses served since boot, `tts_audio_s` - the speech seconds they carried), a `tts` block present ONLY while a speech model is configured - and still there when its worker could not load it; it describes the default TTS model (the first that loaded, else the first that failed), and carries `models` - the ids of every loaded TTS model - only when several serve (`id`, `ready`, `pending`, `done_jobs`, `audio_s`, `voices[]` and `sample_rate` as the loaded model declares them, `cloning` (the model takes a voice from a clip), `speed` (a `speed` other than 1.0 is honoured; false for a Pocket model, which refuses one) and `lang` (the first language it declares), the `lane` its worker pinned or, before there is an answer, the one the boot asked for, and `error` - the loader's reason, present only when the configured model failed to load, which the control page's offer card reads instead of claiming the model is still on its way) - the control page's speech studio gates on `ready`, media counters (`media_pending`, `media_rows`, `mrope_streams` - streams whose media rode the qwen mrope grid walk), and `models[]` - one entry per slot: `file` (source GGUF base name - the page's serve-live gate), `is_active`, `holds_gpu`, requested `backend` vs `backend_effective` (`cpu`/`gpu:rails`/`gpu:resident`), `served` (how the slot is served, in plain words: where the weights sit and where the streams' caches do - the control page prints it on the model card), `served_note` (why it is not served better: the whole-model driver's decline with its remedy, the tower or self-speculation that keeps the caches on the host, fewer K/V regions than streams; empty when nothing holds the slot back), `device_kv` (every stream's K/V cache lives on the GPU, a region each), `vision` and `audio` (the towers that slot actually loaded - the catalog row's chip reads them, and `audio` is what tells the chat mic to attach a clip instead of transcribing it), per-slot cache counters, `last_used_s`, switch count/avg ms | +| `GET` | `/v1/stats` | Scheduler counters (`gen_tokens`, `prefill_tokens`, TTFT last/avg, ...) plus `model`/`active_model`/`ctx`/`uptime_s`/`draining` identity fields, memory footprint (`weights_bytes`, `kv_bytes`, das heaps, `gpu_vram_bytes`/`gpu_budget_bytes`), `gpu_cpu_passes` (the calls the armed GPU path handed back to the CPU since the model armed, `{reason, words, count}` per reason that fired - `words` is the reason as the control page prints it; empty means the device served every call), a `hardware` line (CPU * lanes * GPU), `asr_workers`, `asr_ready`, `asr_active`, `asr_pending`, an `asr` block present while an ASR model is configured (`models[]`: each model's `id`, `workers`, `ready` and `decoder` - `metal` where its workers decode on the device, `cpu` elsewhere, empty until a worker answered), speech counters (`tts_done_jobs` - syntheses served since boot, `tts_audio_s` - the speech seconds they carried), a `tts` block present ONLY while a speech model is configured - and still there when its worker could not load it; it describes the default TTS model (the first that loaded, else the first that failed), and carries `models` - the ids of every loaded TTS model - only when several serve (`id`, `ready`, `pending`, `done_jobs`, `audio_s`, `voices[]` and `sample_rate` as the loaded model declares them, `cloning` (the model takes a voice from a clip), `speed` (a `speed` other than 1.0 is honoured; false for a Pocket model, which refuses one) and `lang` (the first language it declares), the `lane` its worker pinned or, before there is an answer, the one the boot asked for, and `error` - the loader's reason, present only when the configured model failed to load, which the control page's offer card reads instead of claiming the model is still on its way) - the control page's speech studio gates on `ready`, media counters (`media_pending`, `media_rows`, `mrope_streams` - streams whose media rode the qwen mrope grid walk), and `models[]` - one entry per slot: `file` (source GGUF base name - the page's serve-live gate), `is_active`, `holds_gpu`, requested `backend` vs `backend_effective` (`cpu`/`gpu:rails`/`gpu:resident`), `served` (how the slot is served, in plain words: where the weights sit and where the streams' caches do - the control page prints it on the model card), `served_note` (why it is not served better: the whole-model driver's decline with its remedy, the tower or self-speculation that keeps the caches on the host, fewer K/V regions than streams; empty when nothing holds the slot back), `device_kv` (every stream's K/V cache lives on the GPU, a region each), `vision` and `audio` (the towers that slot actually loaded - the catalog row's chip reads them, and `audio` is what tells the chat mic to attach a clip instead of transcribing it), per-slot cache counters, `last_used_s`, switch count/avg ms | | `GET` | `/v1/streams` | Per-stream poll surface: `model` (the slot it runs on), state (`queued`/`prefilling`/`decoding`/`finished`), token counts, TTFT, and capped text tails (prompt head + generated tail); finished streams linger ~10 s flagged `finished`. Plus `cache`: the prefix-cache donation chains (tokens, live pages, hits, age, preview) and `asr`: recent ASR jobs (`model` - the id of the ASR model that took the job, empty once its client has gone - state, audio s, wall ms, RTF) | | `GET` | `/config` | Effective config with per-key source (`default`/`cli`/`toml`) - one entry per row of the flags table above, `tts` (the speech model path), `tts_lane` (`q8` | `f32`) and `tts_voices_dir` (the clip directory a cloning model reads) included - plus the `[[models]]` roster, the speech rosters `asr_models` (`id`, `path`, `mmproj`, `workers` an entry) and `tts_models` (`id`, `path`, `lane`, `voices_dir`) - one entry for a one-path `asr` / `tts` key, one a repeated `--asr` / `--tts` flag, one a `[[asr]]` / `[[tts]]` table - model files beside the served one, active rail (gguf vs prepared `.dlim`), GPU tier status (`supported` + `reason` when the loaded model can't ride it). `chunk_idle` answers raw: 0 when unset, meaning the serving backend's default (2048 on Metal, 512 elsewhere) | | `POST` | `/config` | Validate a `{key: value}` JSON body (`asr` and `tts` each a path, or an array of entries with the `[[asr]]` / `[[tts]]` table keys) and write it as an **authoritative** TOML (`authoritative = true`) to the config path (or `~/.dasllama/dasllama-server.toml` on a config-less start). Applies on the next restart | @@ -726,8 +731,10 @@ absent; set `DASLLAMA_MODELS_DIR`): two ASR models and two TTS models, every id on `/v1/models`, each answering under its own id (the job's model read off `/v1/streams`) and the first of a kind answering a request that names none, the `404` for an id nothing serves with its upload removed, a model added twice served - once, the four answering four requests sent at once, one ASR model taking any name, and an ASR - model that cannot load failing the boot. Needs both Qwen3-ASR GGUFs with + once, the four answering four requests sent at once, one ASR model taking any name, an ASR + model serving alone - its decoder on the CPU with the Metal mode off and on the device with it + on, read off `/v1/stats`'s `asr.models[].decoder`, the stats counting its worker and its clip - + and an ASR model that cannot load failing the boot. Needs both Qwen3-ASR GGUFs with their mmprojs, `kitten-nano.gguf` with the front-end packs, and `pocket-tts-en-q8.gguf`. - `test_cli_args.das` - dasllama-cli's model-free half, run everywhere: the plan a command line parses to, the bare-token walk, the help surfaces, the config file's keys and roster, the diff --git a/utils/dasllama-server/main.das b/utils/dasllama-server/main.das index ce8ff4f0e4..1b5120496a 100644 --- a/utils/dasllama-server/main.das +++ b/utils/dasllama-server/main.das @@ -379,11 +379,14 @@ def speech_roster_ids(tts : bool) : array { return <- [for (m in tts ? g_tts_models : g_asr_models); empty(m.id) ? m.path : m.id] } -//! whether any configured TTS model has its whole file set: such a boot serves the speech route with no LLM -def private any_tts_present : bool { +//! whether any configured speech model has its whole file set: such a boot serves its speech route with no LLM +def private any_speech_present : bool { for (m in g_tts_models) { return true if (tts_files_present(m.path)) } + for (a in g_asr_models) { + return true if (stat(a.path).is_valid && (empty(a.mmproj) || stat(a.mmproj).is_valid)) + } return false } @@ -904,11 +907,10 @@ def parse_args() { request_exit() return } - // a [[models]] roster needs no flat model key; a --tts start with its file set complete is a - // SERVING server (the speech route), so it never enters setup mode - an incomplete set is one - // configure_tts drops, which leaves this boot nothing to serve + // a [[models]] roster needs no flat model key; a --tts or --asr start with its file set complete + // is a SERVING server (the speech routes), so it never enters setup mode speech_roster_from_flags(g_cfg) - if (empty(g_cfg.model) && empty(g_models) && !any_tts_present()) { + if (empty(g_cfg.model) && empty(g_models) && !any_speech_present()) { g_setup_mode = true print("no model configured - starting in setup mode (open the control page to download one from the catalog)\n") } @@ -1083,12 +1085,12 @@ def init() { // nolint:STYLE037,STYLE038 — boot sequence: the knobs are orde } delete g_models g_models <- roster - if (empty(g_models) && !g_setup_mode && !any_tts_present()) { + if (empty(g_models) && !g_setup_mode && !any_speech_present()) { // not an exit: a crash-looping boot under the watchdog never gets a page to fix the config from g_setup_mode = true - to_log(LOG_ERROR, "dasllama-server: no loadable model - no configured path survived its file check (roster entries, and the --tts file set); starting in setup mode\n") + to_log(LOG_ERROR, "dasllama-server: no loadable model - no configured path survived its file check (roster entries, and the --tts and --asr file sets); starting in setup mode\n") } elif (empty(g_models) && !g_setup_mode) { - to_log(LOG_WARNING, "dasllama-server: no loadable LLM model - serving the speech route only (the TTS model still loads)\n") + to_log(LOG_WARNING, "dasllama-server: no loadable LLM model - serving the speech routes only (the TTS and ASR models still load)\n") } if (!empty(g_cfg.image_mmproj) && !stat(g_cfg.image_mmproj).is_valid) { to_log(LOG_ERROR, "dasllama-server: dropping --image-mmproj - file not found: {g_cfg.image_mmproj}\n") @@ -1206,8 +1208,7 @@ def init() { // nolint:STYLE037,STYLE038 — boot sequence: the knobs are orde add_asr_model(a.path, a.mmproj, max(a.workers, 1), a.id) } } - // outside the serving gate: a --tts start with no loadable LLM is a speech-only SERVER, and - // its route has to arm on that boot too + // a --tts start with no loadable LLM is a speech-only SERVER, and its route arms on that boot too configure_tts("") for (m in g_tts_models) { add_tts_model(m.path, m.lane, m.voices_dir, m.id) diff --git a/utils/dasllama-server/openai_server.das b/utils/dasllama-server/openai_server.das index 52838ee65c..fb8a8d4b5b 100644 --- a/utils/dasllama-server/openai_server.das +++ b/utils/dasllama-server/openai_server.das @@ -325,6 +325,7 @@ struct AsrSlot { mmproj : string workers : int ready : int //! workers that loaded and answered + decoder : string //! where the workers decode, off a ready event: `metal` for a decoder loaded as the Metal flavor, else `cpu` } //! One served TTS model: one worker loads and owns it. The answer-side fields are the worker's own, off its ready event. @@ -350,6 +351,7 @@ var private g_asr_alias = "" //! the default ASR model's other na var private g_asr_jobs : array //! a job channel a running ASR model, by its row; the thread that started the workers holds them var private g_tts_jobs : array //! the same a TTS model, null for one that is not armed var g_asr_team_dispatch = false +var g_speech_metal = MetalMode.off //! the engine's Metal mode, handed to the ASR workers whose decoder is a Model session var g_tts_team_dispatch = false let TTS_VOICE_CLIP_EXTENSIONS : array <- [".wav", ".flac", ".mp3", ".ogg"] // what the decode rail reads var g_shutdown_requested = false @@ -831,6 +833,10 @@ def server_has_tts : bool { return tts_default() >= 0 } +//! True when a speech model serves with no LLM slot beside it: an armed TTS model or a configured ASR model. +def server_has_speech : bool + => server_has_tts() || !empty(g_asr) + //! Stop admitting new inference/ASR work and exit after all accepted work finishes. Also wired to //! ``POST /shutdown``. def request_stop() { @@ -3817,6 +3823,15 @@ def private tts_stats_member() : string { return ",\"tts\":\{\"id\":{json_str(m.id)},\"ready\":{m.ready > 0},\"pending\":{g_tts_pending},\"done_jobs\":{g_tts_done_jobs},\"audio_s\":{g_tts_audio_s},\"voices\":[{voices}],\"sample_rate\":{m.sample_rate},\"lane\":{json_str(lane)},\"cloning\":{m.cloning},\"speed\":{m.speed},\"lang\":{json_str(m.lang)}{models}{failed}}" } +//! the stats answer's `asr` member: each served ASR model with its workers and where they decode; "" with none +def private asr_stats_member() : string { + return "" if (empty(g_asr)) + var models <- [for (a in g_asr); "\{\"id\":{json_str(a.id)},\"workers\":{a.workers},\"ready\":{a.ready},\"decoder\":{json_str(a.decoder)}}"] + let member = ",\"asr\":\{\"models\":[{join(models, ",")}]}" + delete models + return member +} + // splice a member into a compact JSON object body: sprint_json/write_json_compact both close on // the final `}`, and the stats shapes are objects with at least one member def private with_json_member(body : string; member : string) : string { @@ -3832,7 +3847,7 @@ def private handle_stats(var resp : Reply) : http_status { // `setup` is the MODE, not the slot count — a speech-only server serves the speech route // and must not draw the page's pick-a-model surface. return resp |> JSON( - "\{\"setup\":{g_setup_mode},\"model\":\"(no model)\",\"active_model\":\"\",\"ctx\":0,\"uptime_s\":{g_started_at > 0l ? get_time_nsec(g_started_at) / 1_000_000_000l : 0l},\"draining\":{g_shutdown_requested},\"active\":0,\"queued\":0,\"peak_active\":0,\"admitted\":0,\"finished\":0,\"evicted\":0,\"decode_steps\":0,\"gen_tokens\":0,\"avg_batch\":0,\"cached_tokens\":0,\"prefix_pages\":0,\"max_streams\":0,\"chunk_tokens\":0,\"asr_workers\":0,\"asr_ready\":0,\"asr_active\":0,\"asr_pending\":0,\"mtp_drafted\":0,\"mtp_accepted\":0,\"prefill_tokens\":0,\"ttft_avg_ms\":0,\"ttft_last_ms\":0,\"weights_bytes\":0,\"kv_bytes\":0,\"gpu_vram_bytes\":0,\"gpu_budget_bytes\":0,\"hardware\":{json_str(g_hardware_line)},\"asr_done_jobs\":0,\"asr_audio_s\":0,\"tts_done_jobs\":{g_tts_done_jobs},\"tts_audio_s\":{g_tts_audio_s},\"vision\":false,\"audio\":false,\"models\":[]{tts_stats_member()}}", + "\{\"setup\":{g_setup_mode},\"model\":\"(no model)\",\"active_model\":\"\",\"ctx\":0,\"uptime_s\":{g_started_at > 0l ? get_time_nsec(g_started_at) / 1_000_000_000l : 0l},\"draining\":{g_shutdown_requested},\"active\":0,\"queued\":0,\"peak_active\":0,\"admitted\":0,\"finished\":0,\"evicted\":0,\"decode_steps\":0,\"gen_tokens\":0,\"avg_batch\":0,\"cached_tokens\":0,\"prefix_pages\":0,\"max_streams\":0,\"chunk_tokens\":0,\"asr_workers\":{[for (a in g_asr); a.workers] |> sum()},\"asr_ready\":{[for (a in g_asr); a.ready] |> sum()},\"asr_active\":{g_asr_active},\"asr_pending\":{g_asr_pending},\"mtp_drafted\":0,\"mtp_accepted\":0,\"prefill_tokens\":0,\"ttft_avg_ms\":0,\"ttft_last_ms\":0,\"weights_bytes\":0,\"kv_bytes\":0,\"gpu_vram_bytes\":0,\"gpu_budget_bytes\":0,\"hardware\":{json_str(g_hardware_line)},\"asr_done_jobs\":{g_asr_done_jobs},\"asr_audio_s\":{g_asr_audio_s},\"tts_done_jobs\":{g_tts_done_jobs},\"tts_audio_s\":{g_tts_audio_s},\"vision\":false,\"audio\":false,\"models\":[]{asr_stats_member()}{tts_stats_member()}}", http_status.OK) } let tier = gpu_tier_status() @@ -3901,7 +3916,7 @@ def private handle_stats(var resp : Reply) : http_status { } let body = sprint_json(out, false) delete out - return resp |> JSON(with_json_member(body, tts_stats_member()), http_status.OK) + return resp |> JSON(with_json_member(body, "{asr_stats_member()}{tts_stats_member()}"), http_status.OK) } // /v1/streams: the poll surface behind the page's swimlane + stream cards — per-stream state, @@ -4641,14 +4656,18 @@ def private asr_worker_main( worker_index : int; model_path, mmproj_path : string; team_dispatch : bool; + metal : MetalMode; var jobs : Channel?&; var events : Channel?& ) { this_context().name := "dasllama_asr_{worker_index}" configure_media_worker_dispatch(team_dispatch) - // the engine's Metal mode is the engine context's: this one starts with none, and a family whose decoder is an - // LLM (Qwen3-ASR, Canary) prefills it here on the CPU arm - a legitimate serving configuration, not a profiling slip - allow_cpu_prefill() + // a two-file model's decoder is a Model session and takes the engine's Metal mode; a one-file model (whisper, + // parakeet) decodes on hand-written CPU loops over planar weights, which the mode's portable pin would slow + if (!empty(mmproj_path)) { + set_metal_mode(metal) + } + allow_cpu_prefill() // a decline lands the decoder's prefill on the CPU arm: served, not a profiling slip var model = AsrModel() var load_failure = "" let load_failed = catch_worker_panic(load_failure) { @@ -4661,7 +4680,7 @@ def private asr_worker_main( events |> release return } - emit_asr_event(events, AsrEvent(kind = AsrEventKind.ready)) + emit_asr_event(events, AsrEvent(kind = AsrEventKind.ready, body = model.dec.metal_blob ? "metal" : "cpu")) var session = AsrSession() var session_language = "" var has_session = false @@ -4765,12 +4784,13 @@ def private start_asr_worker_threads() { let model_path = a.path let mmproj_path = a.mmproj let team_dispatch = g_asr_team_dispatch + let metal = g_speech_metal var jobs_capture = jobs var events_capture = g_asr_events - new_thread() <| @capture(= worker_index, = model_path, = mmproj_path, = team_dispatch, + new_thread() <| @capture(= worker_index, = model_path, = mmproj_path, = team_dispatch, = metal, = jobs_capture, = events_capture) { asr_worker_main( - worker_index, model_path, mmproj_path, team_dispatch, jobs_capture, events_capture) + worker_index, model_path, mmproj_path, team_dispatch, metal, jobs_capture, events_capture) } index++ } @@ -4779,7 +4799,8 @@ def private start_asr_worker_threads() { g_asr_events |> pop_with_timeout_clone(1000) $(event : AsrEvent#) { if (event.kind == AsrEventKind.ready) { a.ready++ - to_log(LOG_INFO, "dasllama-server: ASR worker ready ({a.id}: {a.ready}/{a.workers})\n") + a.decoder = clone_string(event.body) + to_log(LOG_INFO, "dasllama-server: ASR worker ready ({a.id}: {a.ready}/{a.workers}, decoder on {a.decoder})\n") } elif (event.kind == AsrEventKind.failed) { load_failure = clone_string(event.body) } @@ -6176,6 +6197,7 @@ struct private SpeechBoot { asr_team : bool tts_team : bool asr_alias : string + metal : MetalMode } //! the speech state the engine's status routes and its drain read @@ -6265,6 +6287,7 @@ def private speech_main(var boot : Channel?&; var inbox : Channel?&; var to_engi g_tts := b.tts g_asr_team_dispatch = b.asr_team g_tts_team_dispatch = b.tts_team + g_speech_metal = b.metal g_asr_alias = clone_string(b.asr_alias) } boot |> release @@ -6371,7 +6394,7 @@ def private start_speech_thread { g_speech_out = worker_channel() var boot = worker_channel() g_asr_alias = empty(g_slots) ? "" : "{g_slots[g_default_slot].name}-asr" - var first = SpeechBoot(asr_team = g_asr_team_dispatch, tts_team = g_tts_team_dispatch, asr_alias = g_asr_alias) + var first = SpeechBoot(asr_team = g_asr_team_dispatch, tts_team = g_tts_team_dispatch, asr_alias = g_asr_alias, metal = get_metal_mode()) first.asr := g_asr first.tts := g_tts boot |> push_clone(first) @@ -6605,8 +6628,8 @@ def init_server(port : int; max_streams : int64 = 4l; max_queue : int64 = 32l; c to_log(LOG_ERROR, "dasllama-server: init_server called while already initialized\n") return false } - if (empty(g_slots) && !g_setup_mode && !server_has_tts()) { - to_log(LOG_ERROR, "dasllama-server: init_server with no model slots (call set_model/add_model first, or configure_tts for a speech-only server)\n") + if (empty(g_slots) && !g_setup_mode && !server_has_speech()) { + to_log(LOG_ERROR, "dasllama-server: init_server with no model slots (call set_model/add_model first, or add_tts_model / add_asr_model for a speech-only server)\n") return false } g_shutdown_requested = false // reset so a restart in the same context after /shutdown doesn't drain at once @@ -6629,7 +6652,7 @@ def init_server(port : int; max_streams : int64 = 4l; max_queue : int64 = 32l; c g_media_events = worker_channel() init_worker_rails() if (empty(g_slots)) { - if (!g_setup_mode && !server_has_tts()) { + if (!g_setup_mode && !server_has_speech()) { // the speech-only boot's worker dropped the route as it loaded: the catalog/config // page is all that is left to serve, and it is where the config gets fixed to_log(LOG_WARNING, "dasllama-server: the speech route did not arm and no model is loaded - falling back to setup mode\n") diff --git a/utils/dasllama-server/test_openai_server.das b/utils/dasllama-server/test_openai_server.das index 7da92c8a35..1db5741131 100644 --- a/utils/dasllama-server/test_openai_server.das +++ b/utils/dasllama-server/test_openai_server.das @@ -731,7 +731,7 @@ def test_bench_inprocess(t : T?) { // nolint:STYLE038 — one live server sess } } -let TEST_PORT_HELD = 18161 // 18150-18156 are test_openai_server_mtp.das's, 18170-18174 test_openai_server_speech_multi.das's +let TEST_PORT_HELD = 18161 // 18150-18156 are test_openai_server_mtp.das's, 18170-18176 test_openai_server_speech_multi.das's //! The HTTP front answers a status route from the engine's last answer while a handler holds the engine: an embeddings //! forward over a long input runs in-handler for hundreds of milliseconds, and /v1/stats still answers in a fraction of diff --git a/utils/dasllama-server/test_openai_server_speech_multi.das b/utils/dasllama-server/test_openai_server_speech_multi.das index 9c805dd00e..4dc654ace8 100644 --- a/utils/dasllama-server/test_openai_server_speech_multi.das +++ b/utils/dasllama-server/test_openai_server_speech_multi.das @@ -5,6 +5,7 @@ options stack = 524288 // every dasLLAMA program root takes this budget (optio require dastest/testing_boost public require openai_server // add_asr_model / add_tts_model / run_server (sibling) require _server_rig // models_dir (sibling) +require dasllama/dasllama // set_metal_mode: the mode the ASR workers' decoder takes require openai/openai_audio // transcribe / speech: the two speech requests require dashv/dashv_boost // GET / POST: the readiness poll, /v1/models, the shutdown require daslib/jobque_boost // with_job_status / new_thread: the server thread, the four clients at once @@ -21,7 +22,7 @@ require daslib/strings_boost // count // Model-gated (Qwen3-ASR 0.6B and 1.7B with their mmprojs, kitten-nano with the front-end packs, // pocket-tts-en-q8) and JIT-only; skips cleanly otherwise. -let TEST_PORT = 18170 // 18170, 18173, 18174: this file's three boots; 18097-18163 belong to the sibling suites +let TEST_PORT = 18170 // 18170, 18173 .. 18176: this file's five boots; 18097-18163 belong to the sibling suites let ASR_A = "Qwen3-ASR-0.6B-Q8_0" let ASR_B = "Qwen3-ASR-1.7B-Q8_0" @@ -202,6 +203,52 @@ def test_one_asr_model_takes_any_name(t : T?) { } } +//! where /v1/stats says the one ASR model of an ASR-only boot decodes, "" when the boot never served or answered no clip +def private asr_alone_decoder(t : T?; port : int; mode : MetalMode) : string { + var decoder = "" + with_job_status(1) $(finished) { + new_thread() <| @capture(= finished, = port, = mode) { + set_metal_mode(mode) // the engine's mode: the workers take it for a decoder that is a Model session + add_asr_model(path_join(models_dir(), "{ASR_A}.gguf"), asr_mmproj(ASR_A)) + run_server(port) // blocks until POST /shutdown + finished |> notify_and_release + } + let ready = serve_clients(port, 1200, "/v1/models") $(base_url) { + let res = transcribe(openai_client("{base_url}/v1"), ASR_A, JFK_PATH) + t |> success(res.ok && heard_jfk(res.text), "{mode}: {res.text}{res.error.message}") + GET("{base_url}/v1/stats") $(resp) { + var err = "" + var js = read_json(resp != null ? string(resp.body) : "", err) + t |> equal(js?["asr_ready"] ?? 0, 1, "the stats of a boot with no LLM count its ASR worker") + t |> equal(js?["asr_done_jobs"] ?? 0, 1, "and the clip it read") + if (res.ok) { + decoder = js?["asr"]?["models"]?[0]?["decoder"] ?? "" + } + unsafe { + delete js + } + } + } + finished |> join + t |> success(ready, "a boot with an ASR model and nothing else serves") + } + return decoder +} + +//! An ASR model serves with no LLM and no TTS beside it, and its decoder runs where the engine's Metal mode puts it: +//! on the device under the mode a Metal build serves with, on the CPU with the mode off. +[test] +def test_asr_alone_serves_on_the_engine_backend(t : T?) { + var inscope files <- [path_join(models_dir(), "{ASR_A}.gguf"), asr_mmproj(ASR_A)] + return if (model_missing(t, files)) + t |> equal(asr_alone_decoder(t, TEST_PORT + 5, MetalMode.off), "cpu") + if (!has_prefill_override("metal")) { + t |> skip("no Metal driver in this build: the device half needs one") + return + } + t |> equal(asr_alone_decoder(t, TEST_PORT + 6, MetalMode.when_available), "metal") +} + //! An ASR model that cannot load fails the boot with the loader's reason: the speech thread reports it to the engine, //! which was waiting on that thread's first word. The control is the return itself - a boot that hung never gets here. [test] From 90ce36671a7f1c384e2d63f248daeaea27199442 Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Sat, 3 Oct 2026 19:43:32 -0700 Subject: [PATCH 02/46] dasLLAMA: the vision towers' refusal of a projector past the staged-mint line prints its size in decimal - a uint64 in a string reads as hex Co-Authored-By: Claude Fable 5.1 --- modules/dasLLAMA/dasllama/dasllama_gemma3v.das | 2 +- modules/dasLLAMA/dasllama/dasllama_gemma4v.das | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/dasLLAMA/dasllama/dasllama_gemma3v.das b/modules/dasLLAMA/dasllama/dasllama_gemma3v.das index 309232b7bf..64dcccf48f 100644 --- a/modules/dasLLAMA/dasllama/dasllama_gemma3v.das +++ b/modules/dasLLAMA/dasllama/dasllama_gemma3v.das @@ -440,7 +440,7 @@ def stage_gemma3v_tower(path : string) : Gemma3vStaging { panic("dasLLAMA: cannot open mmproj gguf '{path}'") } if (stat(path).size >= GEMMA3V_STAGE_CAP) { - panic("dasLLAMA gemma3v: '{path}' is {stat(path).size / 1048576ul} MB - at or past the 1 GiB staged-mint line; serve the f16 mmproj (an f32-widened twin is an oracle artifact, not a serving file)") + panic("dasLLAMA gemma3v: '{path}' is {int64(stat(path).size / 1048576ul)} MB - at or past the 1 GiB staged-mint line; serve the f16 mmproj (an f32-widened twin is an oracle artifact, not a serving file)") } fmap(f) $(var bytes : array#) { var inscope m <- parse_gguf_meta(bytes) diff --git a/modules/dasLLAMA/dasllama/dasllama_gemma4v.das b/modules/dasLLAMA/dasllama/dasllama_gemma4v.das index f154014ecc..39e0482927 100644 --- a/modules/dasLLAMA/dasllama/dasllama_gemma4v.das +++ b/modules/dasLLAMA/dasllama/dasllama_gemma4v.das @@ -409,7 +409,7 @@ def stage_gemma4v_tower(path : string) : Gemma4vStaging { // the staged mint holds source + image at once; past the line a carrier streams like a decoder, // and this tower has no streamed mint — the shipped vision mmprojs sit under it (E2B bf16 0.99 GB) if (stat(path).size >= GEMMA4V_STAGE_CAP) { - panic("dasLLAMA gemma4v: '{path}' is {stat(path).size / 1048576ul} MB - at or past the 1 GiB staged-mint line; serve the bf16 mmproj (an f32-widened twin is an oracle artifact, not a serving file)") + panic("dasLLAMA gemma4v: '{path}' is {int64(stat(path).size / 1048576ul)} MB - at or past the 1 GiB staged-mint line; serve the bf16 mmproj (an f32-widened twin is an oracle artifact, not a serving file)") } fmap(f) $(var bytes : array#) { var inscope m <- parse_gguf_meta(bytes) From f44bb5ab4bd061e1e3a46e20466a7e1e4918843f Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Sat, 3 Oct 2026 21:01:18 -0700 Subject: [PATCH 03/46] dasLLAMA: the Metal tower reads a whisper encoder's q8 planes - the block GEMMs and the stem's second conv off a device q8 blob on the prefill ladder Whisper serves q8 and the tower declined q8, so a served whisper encoder ran on the CPU: large-v3-turbo read an 8 s clip in 912 ms where whisper.cpp's server reads it in 127. The whisper-class chain uploads the q8 planes once a tower through the transform the ASR-decoder driver uses and runs every site on the prefill driver's q8 GEMM: 137 ms on the same 1.2 GB image. Co-Authored-By: Claude Fable 5.1 --- modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md | 10 ++ modules/dasLLAMA/ARCHITECTURE_MEDIA.md | 12 +- modules/dasLLAMA/PERF_LEDGER.md | 19 +++ .../dasllama/dasllama_metal_prefill.das | 10 ++ .../dasllama/dasllama_metal_tower.das | 115 +++++++++++++++--- modules/dasLLAMA/tests/CLAUDE.md | 13 +- modules/dasLLAMA/tests/test_model_image.das | 72 ++++++----- modules/dasLLAMA/tests/test_whisper.das | 10 +- 8 files changed, 201 insertions(+), 60 deletions(-) diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md b/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md index bee310d2d2..24add36743 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md @@ -52,6 +52,16 @@ best-effort: it answers false (or -1) on any shape, knob, quant-mode or device d CPU chain serves that encode. Engage is read from counter deltas (`metal_tower_stats`, `metal_tower_f16_encodes`), never from "the model ran". +The whisper-class chain serves both weight lanes. An f32 tower's block GEMMs read the f32 blob or +its halfword twin. A q8 tower's read the q8 planes themselves: `tw_q8_attach` uploads the stem's +second conv and each block's six GEMM regions once a tower, out of the CPU backend's repack layout +into the 34-byte q8 blocks the prefill driver's q8 GEMM reads (`q8_region_to_metal_blob`, the +transform the ASR-decoder driver uploads its planes through), and every site runs on the prefill +ladder (`pf_enc_q8_mm`) with the activations converted to one half panel where that ladder reads +one. The device blob is 34/32 of the q8 planes and no image carries it; it drops with the weights +epoch and the tower shutdown. The stem's first conv reads the f32 blob on both lanes - its +3 x n_mel columns are not quantized. + The FastConformer chain (canary and parakeet share it: one context, one block body over the canary offsets record, parakeet's offsets mapped onto it with no GEMM biases and its tap-major depthwise stamp) runs the rel-pos (Transformer-XL) attention one head at a time on the f32 GEMM diff --git a/modules/dasLLAMA/ARCHITECTURE_MEDIA.md b/modules/dasLLAMA/ARCHITECTURE_MEDIA.md index fd1040a9eb..5c0ff47a3f 100644 --- a/modules/dasLLAMA/ARCHITECTURE_MEDIA.md +++ b/modules/dasLLAMA/ARCHITECTURE_MEDIA.md @@ -206,9 +206,10 @@ the driver's seats survive the test. Every hook answers "declined" in its own return - `false` for the block hooks and qwen3a's mel and front hooks, `-1` for the hooks that return a row count (gemma4a's whole chunk, canary's front) - so a decline is a fallback, never an outage, and the CPU form stays the reference. A family calls -its hook on either lane: the Metal driver declines the q8 encoder and the Vulkan driver the exact -one, each as a counted non-policy `quant_mode` decline, which under the driver's required mode -panics - a caller that pins a lane pins the one its driver serves. While the stage-diff witness is +its hook on either lane: the Metal driver declines the q8 encoder of every family but the +whisper-class tower, whose chain reads both lanes (`ARCHITECTURE_GPU_TOWER.md#tower-encode-chains`), +and the Vulkan driver the exact one, each as a counted non-policy `quant_mode` decline, which under +the driver's required mode panics - a caller that pins a lane pins the one its driver serves. While the stage-diff witness is armed (`set_audio_encode_ref_dir`), the whisper-class tower skips its conv and block hooks and runs the CPU forms: the witness diffs the CPU stages' rows against reference dumps, and a stage the device served leaves nothing to diff. @@ -227,8 +228,9 @@ column buffer (`s.x0`, `st.xw`) and never the residual stream the CPU half would A tower serves its GEMMs on one of two lanes: q8 planes (the CPU serving format) or the file's exact f32 planes. Un-pinned, the lane follows the fastest GEMM path on the box - a serving Metal -driver reads the f32 blob and declines q8, so `*_gpu_serves` answering true flips the default to -exact, and every other box takes q8. Each family exposes the same trio over one `GemmLane` pin +driver that reads the f32 blob and declines q8 makes `*_gpu_serves` answering true flip the default +to exact, and every other box takes q8. Whisper carries no lane policy and serves q8 everywhere: +the Metal tower reads its q8 planes. Each family exposes the same trio over one `GemmLane` pin (`lane_serves_q8`, `dasllama_common.das`): `set_*_q8` pins a lane, `reset_*_q8` returns to the policy, `*_serves_q8` reports the lane the next load would take. The lane picks the image tag, so the two lanes are separate images that coexist. diff --git a/modules/dasLLAMA/PERF_LEDGER.md b/modules/dasLLAMA/PERF_LEDGER.md index bfb3521b46..00eb170925 100644 --- a/modules/dasLLAMA/PERF_LEDGER.md +++ b/modules/dasLLAMA/PERF_LEDGER.md @@ -11,6 +11,25 @@ what it costs today and what the fix would change. ## Entries +- **MEASURED (2026-10-03, `direction-grade`) - the Metal tower reads a whisper encoder's q8 planes.** + M5 Max, Metal, whisper large-v3-turbo, `dasllama-server` with the ASR model alone + (`main.das -- --asr ggml-large-v3-turbo.bin`), the reference whisper.cpp `6fc7c33` as + `whisper-server -m --inference-path /v1/audio/transcriptions`, `harness/served_bench.das + --no-chat --asr-url --clip --reps 5` (tagged `served`, `out-of-process`), a 60 s + rest before each life, every request a clip no server has heard; the tune sidecar predates the + binary, so the CPU kernels ran their fallback stamps. Clips of 8 s, 11 s and 64 s (one 30 s + window, one, three), ms a transcription. + - The tower declined a q8 encoder, and whisper serves q8: the encoder ran on the CPU and read + 911.7 / 951.4 / 3092.8. The reference reads 131.3 / 147.1 / 601.5 on the f16 file and + 127.1 / 139.8 / 554.8 on its q8_0 file (0.87 GB). + - The block GEMMs off the q8 blob on the plain q8 tile: 240.5 / 253.1 / 876.7. On the prefill + ladder: 153.5 / 166.1 / 608.9. With the stem's second conv off the same blob: 137.4 / 149.0 / + 541.0, on the 1.2 GB q8 image. The f32 encoder lane on the same tower reads 132.3 / 140.0 / + 517.9 on a 4.3 GB image. + - A window's stages on the q8 lane (the encoder's own stage clock, in process): blocks 102.5 ms + (94.0 on the f32 lane), the stem 2.2 (15.6 on the CPU), the decode 21.2, the cross K/V 6.4, + the mel 1.4. + - **MEASURED (2026-10-03, `direction-grade`) - a served Qwen3-ASR transcription decodes on the device, and a repeated clip measures the other server's prompt cache.** M5 Max, Metal, `dasllama-server` with the ASR model alone (`main.das -- --asr --mmproj `), the pinned llama-server as diff --git a/modules/dasLLAMA/dasllama/dasllama_metal_prefill.das b/modules/dasLLAMA/dasllama/dasllama_metal_prefill.das index d4a767f645..5fadc5140e 100644 --- a/modules/dasLLAMA/dasllama/dasllama_metal_prefill.das +++ b/modules/dasLLAMA/dasllama/dasllama_metal_prefill.das @@ -5824,6 +5824,16 @@ def private enc_gemm_mm(enc : MetalComputeEncoder?; bwblob : MetalBuffer?; wboff } } +//! whether a GEMM through `pf_enc_q8_mm` reads a half activation panel on this box: the caller converts one a panel +def pf_q8_mm_half() : bool => (g_pf_mm_tensor && g_pf_pso_mm_th != null) || (g_pf_tensor_ok && g_pf_pso_hmm_th != null) + +//! One q8 GEMM over a 34-byte-block blob on the prefill ladder, for a driver outside this file: `mp` rows padded to 32, +//! no GEMV tail; `bxh` is the caller's half twin of `bx`, null where `pf_q8_mm_half` says none is read. +[arch(at="../ARCHITECTURE_GPU_PREFILL.md#prefill-gemm-ladder")] +def pf_enc_q8_mm(enc : MetalComputeEncoder?; bwblob : MetalBuffer?; wboff : uint64; bx, bxh, by, bk, bn : MetalBuffer?; mp, d, kdim : int64) { + enc_gemm_mm(enc, bwblob, wboff, bx, by, bk, bn, mp, d, 0ul, kdim, 0l, bxh) +} + //! kdim feeds the tall/remainder split's X stride; kdim-less callers stay whole-dispatch 32-tile [arch(at="../ARCHITECTURE_GPU_PREFILL.md#prefill-gemm-ladder")] def pf_enc_bf16_mm(enc : MetalComputeEncoder?; bw : MetalBuffer?; wboff : uint64; diff --git a/modules/dasLLAMA/dasllama/dasllama_metal_tower.das b/modules/dasLLAMA/dasllama/dasllama_metal_tower.das index 1df697cf93..1f140fb8f1 100644 --- a/modules/dasLLAMA/dasllama/dasllama_metal_tower.das +++ b/modules/dasLLAMA/dasllama/dasllama_metal_tower.das @@ -108,6 +108,7 @@ def metal_tower_shutdown { release_handle(g_tw_ones) st2_slab_free(g_tw_q3a_wpad) st2_slab_free(g_tw_fc_twin) + st2_slab_free(g_tw_q8) st2_slab_drop() pool_drain(g_tw_pool) g_tw_failed = false @@ -212,23 +213,87 @@ def private metal_gemma4uv_encode(t : Gemma4uvEmbedder; var s : Gemma4uvState; n return true } +var private g_tw_q8 : St2SlabBase //!< the served whisper-class tower's q8 GEMM planes as one device blob, resident until the weights drop +var private g_tw_q8_boffs : array //!< a layer's regions in that blob, TW_Q8_MATS a layer: q, k, v, o, fc1, fc2 +var private g_tw_q8_conv2 = 0ul //!< the stem's second conv in that blob +let private TW_Q8_MATS = 6l +let private TW_Q8_BLOCK_BYTES = Q8_QPB + Q8_SPB + +//! one GEMM region out of the CPU backend's layout into the q8 blob's blocks at the 16-byte-aligned cursor; answers its byte offset +def private tw_q8_region(t : AudioTower; off, n, rows : int64; lay : Q8RepackLayout; var staging : array; var cur : uint64&) : uint64 { + cur = (cur + 15ul) & ~15ul + let at = cur + q8_region_to_metal_blob(plane_at(t.qblob, off), plane_at(t.qscales, off / Q8_BLOCK_ELEMS), + n, rows, lay.repacked, lay.mr, lay.kgroup, lay.wbias, unsafe(addr(staging[int64(cur)]))) + cur += uint64((rows * n / Q8_BLOCK_ELEMS) * TW_Q8_BLOCK_BYTES) + return at +} + +//! The whisper-class tower's q8 block GEMM planes on the device, uploaded once a tower. False where the buffer cannot be made. +[cold_path, arch(at="../ARCHITECTURE_GPU_TOWER.md#tower-encode-chains")] +def private tw_q8_attach(t : AudioTower) : bool { + let key = hash_combine64(intptr(plane_at(t.qblob, 0l)), uint64(length(t.qblob))) + return true if (st2_slab_current(g_tw_q8, key)) + st2_slab_free(g_tw_q8) + let d = t.d_model + let ff = t.n_ff + let layer_elems = 4l * d * d + 2l * d * ff + var staging : array + staging |> reserve_resize(((t.n_layer * layer_elems + 3l * d * d) / Q8_BLOCK_ELEMS) * TW_Q8_BLOCK_BYTES + 16l * (t.n_layer * TW_Q8_MATS + 1l)) + // a pure read of the layout the load repacked under: selecting again here could name another + let lay = active_q8_repack_layout(kernel_backend_needs_repack(active_kernel_backend())) + g_tw_q8_boffs |> resize(t.n_layer * TW_Q8_MATS) + var cur = 0ul + g_tw_q8_conv2 = tw_q8_region(t, t.conv2_w_off, 3l * d, d, lay, staging, cur) + for (l in range64(t.n_layer)) { + let lo = enc_layer_offsets(t, l) + var mi = 0l + for (woff in [lo.q_w, lo.k_w, lo.v_w, lo.o_w]) { + g_tw_q8_boffs[l * TW_Q8_MATS + mi] = tw_q8_region(t, woff, d, d, lay, staging, cur) + mi++ + } + g_tw_q8_boffs[l * TW_Q8_MATS + 4l] = tw_q8_region(t, lo.fc1_w, d, ff, lay, staging, cur) + g_tw_q8_boffs[l * TW_Q8_MATS + 5l] = tw_q8_region(t, lo.fc2_w, ff, d, lay, staging, cur) + } + var blob = metal_new_buffer(g_dev, cur) + if (blob != null) { + unsafe { + memcpy(metal_buffer_contents(blob), addr(staging[0]), cur) + } + g_tw_q8.buf = blob + g_tw_q8.key = key + } + delete staging + return blob != null +} + +//! one block GEMM of the whisper-class chain: region `site` of the device q8 blob on a q8 tower, tw_mm's forms on an f32 one +def private wt_mm(enc : MetalComputeEncoder?; q8 : bool; site : int64; f16w, bf16w : bool; bwh, bblob : MetalBuffer?; welem : int64; + bx, bxh, by : MetalBuffer?; bk, bn : MetalBuffer?; rows, nd, kdim : int64; cvt : bool) { + if (q8) { + if (bxh != null && cvt) { + enc_cvt_half(enc, bx, bxh, rows * kdim) + } + pf_enc_q8_mm(enc, g_tw_q8.buf, g_tw_q8_boffs[site], bx, bxh, by, bk, bn, rows, nd, kdim) + } else { + tw_mm(enc, f16w, bf16w, bwh, bblob, welem, bx, bxh, by, 0ul, bk, bn, rows, nd, kdim, cvt) + } +} + // The whisper-class block loop as ONE command buffer; the attention trio rides uend = npos (the -// non-causal uniform span). Frontend and tail stay CPU; f32 towers only. Pad-row garbage is +// non-causal uniform span). Frontend and tail stay CPU. Pad-row garbage is // CONTAINED: QK/AV zero k/v rows >= npos, softmax zeroes [uend, np32), pad outputs go unread. [hot_path, arch(at="../ARCHITECTURE_IMAGE.md#image-tower-twin-plane")] def private metal_tower_blocks(t : AudioTower; var s : EncoderState; npos : int64) : bool { // nolint:STYLE038 — the dispatch chain is the shape if (!g_tw_env_tower) { return tw_decline(MetalTowerDecline.knob) } - if (t.q8) { - return tw_decline(MetalTowerDecline.quant_mode) - } let d = t.d_model let hs = d / t.n_head if (d % 64l != 0l || t.n_ff % 64l != 0l || hs % 64l != 0l || t.layers_off % 4l != 0l || t.layer_stride % 4l != 0l) { return tw_decline(MetalTowerDecline.shape) } - if (!metal_tower_init()) { // per-encode: re-arms after any shutdown seat + if (!metal_tower_init() || (t.q8 && !tw_q8_attach(t))) { // per-encode: re-arms after any shutdown seat return tw_decline(MetalTowerDecline.device) } residency_flush() // pin/commit before the first submission (the post-CPU-window slack) @@ -242,11 +307,11 @@ def private metal_tower_blocks(t : AudioTower; var s : EncoderState; npos : int6 let bytes_att = uint64(t.n_head * mp * nk64 * 2l) let bytes_rowk = uint64(nk64 * d * 4l) //! K panel: the QK grid walks nk64 key rows, past mp let bytes_stat = uint64(t.n_head * mp * 2l * 4l) - let twin = g_tw_env_tower_f16 && length(t.wblob) > 0l + let twin = !t.q8 && g_tw_env_tower_f16 && length(t.wblob) > 0l let f16_route = twin && !t.blk_bf16 && pf_hmm_ready() let bf16_route = twin && t.blk_bf16 let bytes_xh = uint64(mp * ff * 2l) - var bxh = (f16_route || (bf16_route && pf_bf16_half_ready())) ? pool_acquire_pinned(g_tw_pool, g_dev, bytes_xh) : null + var bxh = (f16_route || (bf16_route && pf_bf16_half_ready()) || (t.q8 && pf_q8_mm_half())) ? pool_acquire_pinned(g_tw_pool, g_dev, bytes_xh) : null var bwh = (f16_route || bf16_route) ? plane_buffer(g_dev, unsafe(reinterpret(plane_at(t.wblob, 0l))), uint64(length(t.wblob) * 2l), mapped) : null let f16w = f16_route && bwh != null && bxh != null let bf16w = bf16_route && bwh != null @@ -274,29 +339,29 @@ def private metal_tower_blocks(t : AudioTower; var s : EncoderState; npos : int6 for (l in range64(t.n_layer)) { let lo = enc_layer_offsets(t, l) enc_ln(enc, bx, bblob, uint64(lo.ln1_w * 4l), bblob, uint64(lo.ln1_b * 4l), bxb, 0ul, u_d, u_eps, npos) - tw_mm(enc, f16w, bf16w, bwh, bblob, lo.q_w, bxb, bxh, bq, 0ul, u_d, u_d, mp, d, d, true) + wt_mm(enc, t.q8, l * TW_Q8_MATS, f16w, bf16w, bwh, bblob, lo.q_w, bxb, bxh, bq, u_d, u_d, mp, d, d, true) enc_add_bias_rows(enc, bq, 0ul, bblob, uint64(lo.q_b * 4l), u_d, u_totd, npos * d) - tw_mm(enc, f16w, bf16w, bwh, bblob, lo.k_w, bxb, bxh, bk, 0ul, u_d, u_d, mp, d, d, false) + wt_mm(enc, t.q8, l * TW_Q8_MATS + 1l, f16w, bf16w, bwh, bblob, lo.k_w, bxb, bxh, bk, u_d, u_d, mp, d, d, false) if (t.has_k_bias) { enc_add_bias_rows(enc, bk, 0ul, bblob, uint64(lo.k_b * 4l), u_d, u_totd, npos * d) } - tw_mm(enc, f16w, bf16w, bwh, bblob, lo.v_w, bxb, bxh, bv, 0ul, u_d, u_d, mp, d, d, false) + wt_mm(enc, t.q8, l * TW_Q8_MATS + 2l, f16w, bf16w, bwh, bblob, lo.v_w, bxb, bxh, bv, u_d, u_d, mp, d, d, false) enc_add_bias_rows(enc, bv, 0ul, bblob, uint64(lo.v_b * 4l), u_d, u_totd, npos * d) enc_qk_mm(enc, bq, bk, batt, ka, mp, nk64, t.n_head) enc_rowstat(enc, batt, ka, bstat, null, npos, t.n_head) enc_av_mm(enc, batt, bv, bxb2, ka, bstat, mp, t.n_head, hs) - tw_mm(enc, f16w, bf16w, bwh, bblob, lo.o_w, bxb2, bxh, bxb, 0ul, u_d, u_d, mp, d, d, true) + wt_mm(enc, t.q8, l * TW_Q8_MATS + 3l, f16w, bf16w, bwh, bblob, lo.o_w, bxb2, bxh, bxb, u_d, u_d, mp, d, d, true) enc_add_bias_rows(enc, bxb, 0ul, bblob, uint64(lo.o_b * 4l), u_d, u_totd, npos * d) enc_add(enc, bx, 0ul, bxb, 0ul, u_totd, npos * d) enc_ln(enc, bx, bblob, uint64(lo.ln2_w * 4l), bblob, uint64(lo.ln2_b * 4l), bxb, 0ul, u_d, u_eps, npos) - tw_mm(enc, f16w, bf16w, bwh, bblob, lo.fc1_w, bxb, bxh, bh, 0ul, u_d, u_ff, mp, ff, d, true) + wt_mm(enc, t.q8, l * TW_Q8_MATS + 4l, f16w, bf16w, bwh, bblob, lo.fc1_w, bxb, bxh, bh, u_d, u_ff, mp, ff, d, true) enc_add_bias_rows(enc, bh, 0ul, bblob, uint64(lo.fc1_b * 4l), u_ff, u_totff, npos * ff) if (t.gelu_tanh) { enc_gelu_lut(enc, bh, 0ul, u_totff, npos * ff) } else { enc_gelu_erf(enc, bh, 0ul, u_totff, npos * ff) } - tw_mm(enc, f16w, bf16w, bwh, bblob, lo.fc2_w, bh, bxh, bxb2, 0ul, u_ff, u_d, mp, d, ff, true) + wt_mm(enc, t.q8, l * TW_Q8_MATS + 5l, f16w, bf16w, bwh, bblob, lo.fc2_w, bh, bxh, bxb2, u_ff, u_d, mp, d, ff, true) enc_add_bias_rows(enc, bxb2, 0ul, bblob, uint64(lo.fc2_b * 4l), u_d, u_totd, npos * d) enc_add(enc, bx, 0ul, bxb2, 0ul, u_totd, npos * d) } @@ -2008,12 +2073,25 @@ def private metal_qwen25v_blocks(t : Qwen25vTower; var s : Qwen25vState; npos : return true } +//! the stem's second conv as a GEMM over its im2col rows: the device q8 blob's region on a q8 tower, the f32 blob otherwise +def private conv2_mm(enc : MetalComputeEncoder?; t : AudioTower; bblob, bx2, bx2h, bxo, u_3d, u_d : MetalBuffer?; mp2 : int64) { + let d = t.d_model + if (!t.q8) { + enc_f32_mm(enc, bblob, uint64(t.conv2_w_off * 4l), bx2, bxo, 0ul, u_3d, u_d, mp2, d) + return + } + if (bx2h != null) { + enc_cvt_half(enc, bx2, bx2h, mp2 * 3l * d) + } + pf_enc_q8_mm(enc, g_tw_q8.buf, g_tw_q8_conv2, bx2, bx2h, bxo, u_3d, u_d, mp2, d, 3l * d) +} + // The whisper conv frontend for one mel chunk: im2col + conv GEMMs + bias + GELU + pos-add, // one command buffer, s.x readback. BEST-EFFORT inside the tower knob: any decline is a // silent false (no counter noise — the CPU conv serves and the blocks still ride the GPU). [hot_path] def private metal_conv_frontend(t : AudioTower; var s : EncoderState; mel : MelChunk) : bool { - if (!g_tw_env_tower || t.q8) { + if (!g_tw_env_tower) { return false } let tin = AUDIO_CHUNK_FRAMES @@ -2021,7 +2099,7 @@ def private metal_conv_frontend(t : AudioTower; var s : EncoderState; mel : MelC let d = t.d_model let nm = t.n_mel // tiny's 3x80 im2col width stays CPU; init re-arms after any shutdown seat - if ((3l * nm) % 32l != 0l || d % 64l != 0l || (3l * d) % 32l != 0l || !metal_tower_init()) { + if ((3l * nm) % 32l != 0l || d % 64l != 0l || (3l * d) % 32l != 0l || !metal_tower_init() || (t.q8 && !tw_q8_attach(t))) { return false } let mp1 = round_up(tin, 32l) @@ -2038,6 +2116,7 @@ def private metal_conv_frontend(t : AudioTower; var s : EncoderState; mel : MelC var by1 = pool_acquire_pinned(g_tw_pool, g_dev, bytes_y1) var bx2 = pool_acquire_pinned(g_tw_pool, g_dev, bytes_x2) var bxo = pool_acquire_pinned(g_tw_pool, g_dev, bytes_xo) + var bx2h = (t.q8 && pf_q8_mm_half()) ? pool_acquire_pinned(g_tw_pool, g_dev, bytes_x2 / 2ul) : null unsafe { memcpy(metal_buffer_contents(bmel), addr(mel.data[0]), bytes_mel) } @@ -2059,7 +2138,7 @@ def private metal_conv_frontend(t : AudioTower; var s : EncoderState; mel : MelC enc_gelu_erf(enc, by1, 0ul, u_totd1, tin * d) } enc_st2_im2col(enc, by1, bx2, k2, mp2 * 3l * d) - enc_f32_mm(enc, bblob, uint64(t.conv2_w_off * 4l), bx2, bxo, 0ul, u_3d, u_d, mp2, d) + conv2_mm(enc, t, bblob, bx2, bx2h, bxo, u_3d, u_d, mp2) enc_add_bias_rows(enc, bxo, 0ul, bblob, uint64(t.conv2_b_off * 4l), u_d, u_totd2, tt * d) if (t.gelu_tanh) { enc_gelu_lut(enc, bxo, 0ul, u_totd2, tt * d) @@ -2081,6 +2160,9 @@ def private metal_conv_frontend(t : AudioTower; var s : EncoderState; mel : MelC pool_release(g_tw_pool, by1, bytes_y1) pool_release(g_tw_pool, bx2, bytes_x2) pool_release(g_tw_pool, bxo, bytes_xo) + if (bx2h != null) { + pool_release(g_tw_pool, bx2h, bytes_x2 / 2ul) + } pool_release(g_upool, u_nm3, 4ul) pool_release(g_upool, u_d, 4ul) pool_release(g_upool, u_3d, 4ul) @@ -4514,6 +4596,7 @@ def private metal_styletts2_driver : St2GpuDriver { def private tw_weights_drop { st2_slab_free(g_tw_q3a_wpad) st2_slab_free(g_tw_fc_twin) + st2_slab_free(g_tw_q8) st2_slab_drop() } diff --git a/modules/dasLLAMA/tests/CLAUDE.md b/modules/dasLLAMA/tests/CLAUDE.md index a258fde81f..5e5beb8307 100644 --- a/modules/dasLLAMA/tests/CLAUDE.md +++ b/modules/dasLLAMA/tests/CLAUDE.md @@ -254,11 +254,14 @@ forms under the `mulmm_q8,attn_dev` crown - every head at once, its engage count (`metal_tower_fc_dev_encodes`) and its transcript the CPU's, and the per-head loop with the lever off (`set_metal_fc_attn_dev`), the counter unmoved, the two forms' distances from the CPU rail asserted to differ; the crowns and the three levers are put back as the cell found them), plus -the tower q8-decline - a q8 whisper encoder never dispatches and records the `quant_mode` -decline, and the whisper serving default IS q8 unless `set_asr_fp32` / `set_asr_tower_fp32` -asks for f32 (whisper carries no lane policy). Canary, parakeet and gemma4a do: un-pinned, -their lane follows whether the Metal tower would serve. -Then the required-mode panic cell; the arm's DECODER half is the `test_whisper_metal_cross_kv` +the whisper q8 lane on the tower - a q8 whisper encoder serves on the device off its q8 planes: +the transcript the CPU q8 chain's, one encode and n_layer blocks counted, no `quant_mode` decline, +the halfword twin's counter unmoved, and a q8 tower with one layer's planes zeroed reading another +text through the device chain (the control); the whisper serving default IS q8 unless +`set_asr_fp32` / `set_asr_tower_fp32` asks for f32 (whisper carries no lane policy). Canary, +parakeet and gemma4a do: un-pinned, their lane follows whether the Metal tower would serve, and +their q8 lane declines `quant_mode`. The required-mode panic rides the parakeet q8 cell: that +decline under `MetalMode.required` panics; the arm's DECODER half is the `test_whisper_metal_cross_kv` cell in `test_model_image.das` - GPU cross-KV on the q8 serving default, transcript-exact against the CPU chain with window/step counter deltas and the knob and quant_mode declines, required-mode, step-floor and shutdown-re-arm contract; the voxtral arm re-saves a diff --git a/modules/dasLLAMA/tests/test_model_image.das b/modules/dasLLAMA/tests/test_model_image.das index 2baa5d7614..f2a33fbd7d 100644 --- a/modules/dasLLAMA/tests/test_model_image.das +++ b/modules/dasLLAMA/tests/test_model_image.das @@ -1356,42 +1356,38 @@ def test_whisper_metal_tower(tst : T?) { t |> success(z_text != cpu.text, "a zeroed twin layer CHANGES the GPU transcript (the twin is read)") set_metal_tensor_crowns("") metal_prefill_shutdown() - // the q8 SERVING DEFAULT declines (quant_mode) and never dispatches — the shipped path stays CPU + // the q8 SERVING DEFAULT serves on the tower: its block GEMMs read the q8 planes as a device blob var inscope w8 = WhisperModel() var inscope stg8 <- stage_whisper_model(path, true) mint_whisper_model(stg8, w8) t |> success(w8.enc.q8, "the q8 flavor is on the q8 rail") + set_metal_tower(false) + let q8_cpu = wdec_transcribe(w8, samples) + set_metal_tower(true) let sq0 = metal_tower_stats() - var q8_text = "" - with_dasllama_jobque() { - var s8 <- create_session(w8) - transcribe(w8, s8, samples) $(seg) { - q8_text += seg.text - } - delete s8 - } + let tq0 = metal_tower_f16_encodes() + let dq0 = metal_tower_declines()?["quant_mode"] ?? 0l + let q8_gpu = wdec_transcribe(w8, samples) let sq1 = metal_tower_stats() - t |> equal(sq1.encodes, sq0.encodes, "q8 default never dispatches the tower") - t |> equal(sq1.declines - sq0.declines, 1l, "q8 default records the quant_mode decline") - t |> success(key_exists(metal_tower_declines(), "quant_mode"), "the decline reason is quant_mode") - t |> success(!empty(q8_text), "q8 default still transcribes (CPU serves)") - // required-mode contract: a capability decline (quant_mode) under MetalMode.required - // panics instead of silently CPU-serving; the knob decline is POLICY and never does - set_metal_mode(MetalMode.required) - var tripped = false - try { - with_dasllama_jobque() { - var s9 <- create_session(w8) - transcribe(w8, s9, samples) $(seg) { - pass - } - delete s9 - } - } recover { - tripped = true + to_log(LOG_INFO, "whisper tiny q8 mtower CPU eyeball: '{q8_cpu}'\n") + to_log(LOG_INFO, "whisper tiny q8 mtower GPU eyeball: '{q8_gpu}'\n") + t |> equal(sq1.encodes - sq0.encodes, 1l, "q8: one GPU chunk") + t |> equal(sq1.blocks - sq0.blocks, 4l, "q8: blocks delta = n_layer (tiny = 4)") + t |> equal((metal_tower_declines()?["quant_mode"] ?? 0l) - dq0, 0l, "q8: no quant_mode decline") + t |> equal(metal_tower_f16_encodes(), tq0, "q8: the halfword twin route stays out of it") + t |> success(!empty(q8_gpu) && q8_gpu == q8_cpu, "q8: transcript-exact CPU vs GPU") + // the control: a q8 tower with one layer's GEMM planes zeroed, through the device chain, reads another text + var inscope wz8 = WhisperModel() + var inscope stgz8 <- stage_whisper_model(path, true) + let zq = stgz8.e.enc.qlayers_off + 2l * stgz8.e.enc.qlayer_stride + for (i in range64(stgz8.e.enc.qlayer_stride)) { + stgz8.enc.qblob[zq + i] = int8(0) } - set_metal_mode(MetalMode.when_available) - t |> success(tripped, "required mode: the q8 tower decline panics") + mint_whisper_model(stgz8, wz8) + let sz0 = metal_tower_stats() + let zq_text = wdec_transcribe(wz8, samples) + t |> equal(metal_tower_stats().encodes - sz0.encodes, 1l, "q8 control: the zeroed tower served on the device") + t |> success(zq_text != q8_gpu, "q8 control: a zeroed layer CHANGES the GPU transcript (the q8 blob is read)") // the POLICY class: a knob decline never panics, even under required mode set_metal_mode(MetalMode.required) set_metal_tower(false) @@ -1759,6 +1755,24 @@ def test_parakeet_tower_q8_decline(tst : T?) { t |> success(!empty(text), "transcript non-empty") t |> equal(enc_delta, 0l, "no tower encodes on the q8 lane") t |> success(dec_delta >= 1l, "the quant_mode decline is counted ({dec_delta})") + // required-mode contract: a capability decline (quant_mode) under MetalMode.required + // panics instead of silently CPU-serving + var tripped = false + let mode_was = get_metal_mode() + set_metal_mode(MetalMode.required) + try { + with_dasllama_jobque_() { + var inscope m = AsrModel(kind = AsrKind.parakeet) + var inscope stg <- stage_parakeet_model(path, true) + mint_parakeet_model(stg, m.pk, true) + var inscope s <- create_session(m, "en") + transcribe(m, s, samples) + } + } recover { + tripped = true + } + set_metal_mode(mode_was) + t |> success(tripped, "required mode: the q8 tower decline panics") delete samples } else { t |> skip("no Metal build - the parakeet FastConformer driver has no device to decline") diff --git a/modules/dasLLAMA/tests/test_whisper.das b/modules/dasLLAMA/tests/test_whisper.das index da5459aa0e..4687e6ab5c 100644 --- a/modules/dasLLAMA/tests/test_whisper.das +++ b/modules/dasLLAMA/tests/test_whisper.das @@ -1075,11 +1075,11 @@ def test_whisper_crowned_word_level(t : T?) { to_log(LOG_INFO, "whisper-tiny crowned q8 jfk eyeball: '{text_default}'\n") to_log(LOG_INFO, "whisper-tiny crowned f32/q8 jfk eyeball: '{text_mixed}'\n") // engage witness - without it a declining driver would compare two CPU - // transcripts and pass vacuously. The q8-default ENCODER declines the tower - // by design (quant_mode), so the tower delta is the mixed lane's f32 encode - // alone; both DECODERS ride the crowned wdec - t |> equal(metal_tower_stats().encodes - s0.encodes, 1l, - "the mixed lane's f32 encoder GPU-served (tower encode delta)") + // transcripts and pass vacuously. Both ENCODERS serve on the tower - the q8 + // default off its q8 planes, the mixed lane off its f32 blob - and both + // DECODERS ride the crowned wdec + t |> equal(metal_tower_stats().encodes - s0.encodes, 2l, + "both lanes' encoders GPU-served (tower encode delta)") t |> success(metal_wdec_stats().steps - w0.steps > 0l, "the crowned decoder served (wdec step delta)") let wa <- word_forms(text_default) From f69887a6372d23bed85eaaa5755a24349a48a2ba Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Sat, 3 Oct 2026 22:16:59 -0700 Subject: [PATCH 04/46] dasLLAMA: the Metal whisper decoder leaves a window's cross-KV layouts on the device while its step serves, and its cross-KV GEMMs run on the prefill ladder The driver copied both CPU layouts back every window - 61 MB on large-v3-turbo - for a CPU chain that reads them only when the step declines. They wait on the device and land at a decline, as the Vulkan driver's do. The window's eight GEMMs read one half panel on the prefill ladder. The stage reads 2.8 ms a window where it read 6.4. Co-Authored-By: Claude Fable 5.1 --- .../dasllama/dasllama_metal_asr_dec.das | 86 +++++++++++++++++-- modules/dasLLAMA/tests/CLAUDE.md | 6 +- modules/dasLLAMA/tests/test_whisper.das | 63 ++++++++++++++ 3 files changed, 146 insertions(+), 9 deletions(-) diff --git a/modules/dasLLAMA/dasllama/dasllama_metal_asr_dec.das b/modules/dasLLAMA/dasllama/dasllama_metal_asr_dec.das index ecedc4de4c..cf9d3fb0c8 100644 --- a/modules/dasLLAMA/dasllama/dasllama_metal_asr_dec.das +++ b/modules/dasLLAMA/dasllama/dasllama_metal_asr_dec.das @@ -115,7 +115,54 @@ def private wd_decline(why : MetalWdecDecline) : bool { return false } +var private g_wd_pend_uid = 0ul //!< the state whose cross-KV layouts wait on the device, unread +var private g_wd_pend_bytes = 0ul //!< their size a plane; 0 = nothing waits +var private g_wd_pend_kx : MetalBuffer? +var private g_wd_pend_vx : MetalBuffer? +var private g_wd_dropped : table //!< the states whose waiting layouts were lost unread; a state leaves it at its next window +var private g_wd_uid_next = 1ul + +//! the driver's identity for a decoder state, minted at its first window: a state re-created at the same address inherits nothing +def private wd_uid(var ds : DecoderState) : uint64 { + if (ds.gpu_uid == 0ul) { + ds.gpu_uid = g_wd_uid_next++ + } + return ds.gpu_uid +} + +def private wd_pend_release { + pool_release(g_wd_pool, g_wd_pend_kx, g_wd_pend_bytes) + pool_release(g_wd_pool, g_wd_pend_vx, g_wd_pend_bytes) + g_wd_pend_kx = null + g_wd_pend_vx = null + g_wd_pend_bytes = 0ul +} + +//! the waiting cross-KV layouts of `ds`, where they are its own, landed in its `kx` / `vx`: the CPU chain is about to read them +[cold_path] +def private wd_flush_cross_kv(var ds : DecoderState) { + if (g_wd_pend_bytes == 0ul || g_wd_pend_uid != ds.gpu_uid) { + return + } + unsafe { + memcpy(addr(ds.kx[0]), metal_buffer_contents(g_wd_pend_kx), g_wd_pend_bytes) + memcpy(addr(ds.vx[0]), metal_buffer_contents(g_wd_pend_vx), g_wd_pend_bytes) + } + wd_pend_release() +} + +//! the waiting layouts lost unread (another window's about to take their place, or the driver released): the owner is marked, +//! and its next decode step panics until it starts a window of its own +[cold_path] +def private wd_drop_pending { + if (g_wd_pend_bytes > 0ul) { + g_wd_dropped |> insert(g_wd_pend_uid) + wd_pend_release() + } +} + def private wd_release_all { + wd_drop_pending() release_handles(g_wd_mblob, g_wd_bkxr, g_wd_bvxr, g_wd_bkc, g_wd_bvc, g_wd_sx) release_handles(g_wd_sxb, g_wd_sxb2, g_wd_sq, g_wd_sk, g_wd_sv, g_wd_sh) release_handles(g_wd_slog, g_wd_bpms, g_wd_bpy) @@ -230,6 +277,8 @@ def private wdec_attach(w : WhisperModel) : bool { [hot_path] def private metal_whisper_cross_kv(w : WhisperModel; var ds : DecoderState; enc_out : array) : bool { // nolint:STYLE038 — the dispatch chain is the shape g_wd_window_live = false // whatever happens below, the previous window is over + wd_drop_pending() + g_wd_dropped |> erase(wd_uid(ds)) // this state starts a window of its own, on either chain if (!g_wd_env) { return wd_decline(MetalWdecDecline.knob) } @@ -258,6 +307,7 @@ def private metal_whisper_cross_kv(w : WhisperModel; var ds : DecoderState; enc_ var btmp = pool_acquire(g_wd_pool, g_dev, bytes_rows) var bkx = pool_acquire(g_wd_pool, g_dev, bytes_x) var bvx = pool_acquire(g_wd_pool, g_dev, bytes_x) + var bxh = pf_q8_mm_half() ? pool_acquire(g_wd_pool, g_dev, bytes_rows / 2ul) : null upload_rows_padded(bx, unsafe(addr(enc_out[0])), ta, mp, d) // the pad outputs go unread; the zero guards stale-Inf pool reuse var u_d = uniform_u32(uint(d)) var u_hs = uniform_u32(uint(hs)) @@ -271,11 +321,14 @@ def private metal_whisper_cross_kv(w : WhisperModel; var ds : DecoderState; enc_ let ran = with_compute_encoder(g_queue, err) $(enc : MetalComputeEncoder?) { // nolint:PERF026 — the error-text fetch runs only on the failure leg for (l in range64(nl)) { let lo = dec_layer_offsets(w, l) - enc_gemm_mm_b(enc, g_wd_mblob, g_wd_boffs[l * WD_MATS + 5l], bx, btmp, 0ul, u_d, u_d, mp, d) + if (bxh != null && l == 0l) { // one half panel serves every layer's two GEMMs: they all read the encoder rows + enc_cvt_half(enc, bx, bxh, mp * d) + } + pf_enc_q8_mm(enc, g_wd_mblob, g_wd_boffs[l * WD_MATS + 5l], bx, bxh, btmp, u_d, u_d, mp, d, d) // kx: the pre-scaled CPU layout (the ds.kx readback); kxr: the unscaled resident twin enc_cross_kx(enc, btmp, bkx, uint64(l * d * ta * 4l), u_d, u_hs, u_ta, u_kscale, u_tot, d * ta) enc_cross_vx_h(enc, btmp, 0ul, g_wd_bkxr, uint64(l * d * ta * 2l), bblobf, 0ul, u_d, u_hs, u_ta, u_t0, u_ta, u_h0, u_tot, d * ta) - enc_gemm_mm_b(enc, g_wd_mblob, g_wd_boffs[l * WD_MATS + 6l], bx, btmp, 0ul, u_d, u_d, mp, d) + pf_enc_q8_mm(enc, g_wd_mblob, g_wd_boffs[l * WD_MATS + 6l], bx, bxh, btmp, u_d, u_d, mp, d, d) // cv bias folds into both shuffles: the f32 readback (ds.vx) and the f16 resident enc_cross_vx(enc, btmp, 0ul, bvx, uint64(l * d * ta * 4l), bblobf, uint64(lo.cv_b * 4l), u_d, u_hs, u_ta, u_t0, u_ta, u_h1, u_tot, d * ta) enc_cross_vx_h(enc, btmp, 0ul, g_wd_bvxr, uint64(l * d * ta * 2l), bblobf, uint64(lo.cv_b * 4l), u_d, u_hs, u_ta, u_t0, u_ta, u_h1, u_tot, d * ta) @@ -283,10 +336,6 @@ def private metal_whisper_cross_kv(w : WhisperModel; var ds : DecoderState; enc_ } var served = false if (ran) { - unsafe { - memcpy(addr(ds.kx[0]), metal_buffer_contents(bkx), bytes_x) - memcpy(addr(ds.vx[0]), metal_buffer_contents(bvx), bytes_x) - } g_wdec_windows++ g_wd_window_key = intptr(unsafe(addr(ds))) g_wd_window_live = true @@ -299,10 +348,26 @@ def private metal_whisper_cross_kv(w : WhisperModel; var ds : DecoderState; enc_ } served = true } + if (served && g_wd_window_step) { // the device step reads the resident twins: the CPU layouts wait, read back only at a decline + g_wd_pend_uid = ds.gpu_uid + g_wd_pend_bytes = bytes_x + g_wd_pend_kx = bkx + g_wd_pend_vx = bvx + } else { + if (served) { + unsafe { + memcpy(addr(ds.kx[0]), metal_buffer_contents(bkx), bytes_x) + memcpy(addr(ds.vx[0]), metal_buffer_contents(bvx), bytes_x) + } + } + pool_release(g_wd_pool, bkx, bytes_x) + pool_release(g_wd_pool, bvx, bytes_x) + } pool_release(g_wd_pool, bx, bytes_rows) pool_release(g_wd_pool, btmp, bytes_rows) - pool_release(g_wd_pool, bkx, bytes_x) - pool_release(g_wd_pool, bvx, bytes_x) + if (bxh != null) { + pool_release(g_wd_pool, bxh, bytes_rows / 2ul) + } pool_release(g_upool, u_d, 4ul) pool_release(g_upool, u_hs, 4ul) pool_release(g_upool, u_ta, 4ul) @@ -344,7 +409,11 @@ def private wd_mm(enc : MetalComputeEncoder?; boff : uint64; bx, by : MetalBuffe // logits) and the full-vocab logits readback — the CPU filter/sampler stays the parity anchor. [hot_path] def private metal_whisper_decode(w : WhisperModel; var ds : DecoderState; toks : int const?; n : int64) : bool { // nolint:STYLE038 — the dispatch chain is the shape + if (ds.gpu_uid != 0ul && key_exists(g_wd_dropped, ds.gpu_uid)) { + panic("dasLLAMA wdec: this decoder window's cross-attention memory was lost before it was read back - another state's window or a driver release took its device planes; ONE live decoder window per process, and a state decodes only a window it started after the loss") + } if (!g_wd_window_live || !g_wd_window_step || g_wd_window_key != intptr(unsafe(addr(ds)))) { + wd_flush_cross_kv(ds) // an evicted GPU-decoded window: the CPU chain would read kcache rows the GPU owns if (ds.gpu_kv_gen != 0l && ds.n_past > 0l) { panic("dasLLAMA wdec: the GPU decoder window was evicted mid-window - ONE live decoder window per process (a second DecoderState may not interleave whisper_cross_kv/whisper_decode_batch)") @@ -354,6 +423,7 @@ def private metal_whisper_decode(w : WhisperModel; var ds : DecoderState; toks : if (n > WD_ROW_CAP) { if (ds.n_past == 0l) { // first batch: nothing GPU-owned yet — release the claim, the CPU serves the window g_wd_window_live = false + wd_flush_cross_kv(ds) return false } panic("dasLLAMA wdec: decode batch of {n} exceeds the {WD_ROW_CAP}-row step cap mid-window") diff --git a/modules/dasLLAMA/tests/CLAUDE.md b/modules/dasLLAMA/tests/CLAUDE.md index 5e5beb8307..11dd4f614f 100644 --- a/modules/dasLLAMA/tests/CLAUDE.md +++ b/modules/dasLLAMA/tests/CLAUDE.md @@ -1706,7 +1706,11 @@ token and bar, the batch with its last token changed landing outside the bar as CPU blocks over them transcribe the all-CPU chain's text), `test_whisper_vulkan_wdec_lifetime` (tiny, one session reused the way a serving worker reuses one: a model drop between two transcriptions - the second serves again and reads the same; the decoder knob turned off between two - the second reads as a fresh knob-off session; the block hooks pinned off after a served window - no -handoff for the CPU-encoded windows, the text of the CPU-encoder chain), the ASR knob cells (`set_asr_fp32`, `set_asr_tower_fp32` - the mixed +handoff for the CPU-encoded windows, the text of the CPU-encoder chain), `test_whisper_metal_wdec_flush` +(tiny, the step's floor lifted so the Metal step serves and the window's cross-KV layouts wait on +the device: the host layouts zeroed, a 33-row first batch hands the window to the CPU chain, the +layouts land first and its logits match the knob-off chain's token and bar, the batch with its +last token changed the control), the ASR knob cells (`set_asr_fp32`, `set_asr_tower_fp32` - the mixed f32-enc/q8-dec serving mode and its `asr_exec_fmt` stamp; the strict token-identity cell pins the simdgroup lane, and its tolerance-graded twin pins the crowns ON and asserts WORD equality - the tensor twins' quality gate), the q8-gate CPU-vs-CPU claims diff --git a/modules/dasLLAMA/tests/test_whisper.das b/modules/dasLLAMA/tests/test_whisper.das index 4687e6ab5c..a98ac38381 100644 --- a/modules/dasLLAMA/tests/test_whisper.das +++ b/modules/dasLLAMA/tests/test_whisper.das @@ -777,6 +777,69 @@ def test_whisper_vulkan_wdec_flush(t : T?) { } } +//! The Metal decoder leaves a window's cross-KV layouts on the device while its step serves; a first batch past the step's +//! 32-row cap hands the window to the CPU chain, and the layouts land first. +[test] +def test_whisper_metal_wdec_flush(t : T?) { + t |> run("ggml-tiny.bin: the waiting cross-KV readback lands when the step declines; the CPU chain over it matches the knob-off chain") @(tt : T?) { + static_if (typeinfo builtin_module_exists(das_metal)) { + let model_path = path_join(whisper_dir(), "ggml-tiny.bin") + return if (!jit_model_available(tt, model_path)) + let samples <- read_wav_pcm16_mono(path_join(models_dir(), "jfk.wav")) + if (empty(samples)) { + tt |> skip("jfk.wav not present") + return + } + set_asr_fp32(false) + set_asr_tower_fp32(false) + var inscope m <- mint_asr_whisper(model_path) + let batch <- [for (i in range(33)); m.whisper.tok_sot + i] // one past the step's cap, the first batch of a window + var batch_moved := batch + batch_moved[length(batch_moved) - 1]++ + var cpu_logits : array + var cpu_moved : array + var dev_logits : array + with_dasllama_jobque_() { + set_metal_wdec(false) + var inscope sc <- create_session(m) + transcribe(m, sc, samples) $(_seg) {} + sc.w.ds.n_past = 0l + whisper_decode_batch(m.whisper, sc.w.ds, batch) + cpu_logits := sc.w.ds.logits + sc.w.ds.n_past = 0l + whisper_decode_batch(m.whisper, sc.w.ds, batch_moved) + cpu_moved := sc.w.ds.logits + set_metal_wdec(true) + set_metal_wdec_step(true) + set_metal_wdec_step_min_d(0l) // tiny sits under the step's serving floor: with it lifted the step serves and the layouts wait + let d0 = metal_wdec_stats() + var inscope sg <- create_session(m) + transcribe(m, sg, samples) $(_seg) {} + let d1 = metal_wdec_stats() + tt |> success(d1.windows - d0.windows >= 1l && d1.steps > d0.steps, "the device served the window's cross-KV and its steps ({d1.windows - d0.windows}, {d1.steps - d0.steps})") + for (i in range(length(sg.w.ds.kx))) { // the host layouts are stale by construction: only a flush makes the CPU chain right + sg.w.ds.kx[i] = 0.0 + } + sg.w.ds.n_past = 0l + sg.w.ds.gpu_kv_gen = 0l + whisper_decode_batch(m.whisper, sg.w.ds, batch) + tt |> equal(metal_wdec_stats().steps, d1.steps, "the device served no step of the wide batch: the CPU chain decoded it over the flushed memory") + dev_logits := sg.w.ds.logits + wdec_flush_compare(tt, m.whisper, cpu_logits, dev_logits, cpu_moved) // inside the queue: the argmax is the parallel one + set_metal_wdec_step_min_d(1024l) + } + delete cpu_logits + delete cpu_moved + delete dev_logits + delete m + metal_wdec_shutdown() + metal_tower_shutdown() + } else { + tt |> skip("dasMetal not present") + } + } +} + //! a lifetime cell's seat: tiny on jfk, both drivers on, the q8 default load, one session created for `body` the way a serving //! worker reuses one - the decoder's window lifetime across that session's transcriptions is the claim def private wdec_lifetime_cell(tt : T?; body : block<(m : AsrModel; var s : AsrSession; samples : array) : void>) { From cfb5713870b0ce76b1a920246d7a404c95192a74 Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Sat, 3 Oct 2026 23:13:41 -0700 Subject: [PATCH 05/46] dasLLAMA: gemma-4 E4B audio loads - its audio embedder widens the encoder's 1536 to the decoder's 2560 - and the gemma4a projector tail runs on the Metal tower The loader read the embedder as square, so E4B's audio half read 1536 wide, the ASR facade refused the pair, and a server boot with E4B's projector file died on the audio arm it armed beside vision. The two widths are read apart off the file; an image minted under the square reading mints again. The Metal chain runs the projector tail in the blocks' command buffer on both models. REVIEW_TOWER.md: a stage of a Metal chain on the CPU is a defect; followup_metal 39 lists the audio stages that are. Co-Authored-By: Claude Fable 5.1 --- modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md | 7 +++ modules/dasLLAMA/REVIEW_TOWER.md | 8 +++ .../dasLLAMA/dasllama/dasllama_gemma4a.das | 56 +++++++++++-------- .../dasllama/dasllama_metal_tower.das | 55 +++++++++++++++++- .../dasllama/dasllama_vulkan_tower.das | 2 +- modules/dasLLAMA/followup_metal.md | 17 ++++++ modules/dasLLAMA/followup_vulkan.md | 6 ++ modules/dasLLAMA/tests/CLAUDE.md | 6 +- modules/dasLLAMA/tests/test_model_image.das | 20 ++++++- 9 files changed, 148 insertions(+), 29 deletions(-) diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md b/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md index 24add36743..a342fc31c9 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md @@ -62,6 +62,13 @@ one. The device blob is 34/32 of the q8 planes and no image carries it; it drops epoch and the tower shutdown. The stem's first conv reads the f32 blob on both lanes - its 3 x n_mel columns are not quantized. +The gemma4a chain ends in the projector tail, in the blocks' command buffer: the out projection +and its bias at the encoder's own output width (`gemma4a_mid_dim`, the length of the weightless +norm's ones row), that norm, then the audio embedder to the decoder's width (`proj_dim`). The +embedder is square on E2B (1536 by 1536) and widens on E4B (1536 to 2560), so the two widths are +read apart off the file - the out projection's rows, the embedder's rows - and never assumed equal. +The chain lands the soft tokens in `Gemma4aState.out` and says so (`out_ready`). + The FastConformer chain (canary and parakeet share it: one context, one block body over the canary offsets record, parakeet's offsets mapped onto it with no GEMM biases and its tap-major depthwise stamp) runs the rel-pos (Transformer-XL) attention one head at a time on the f32 GEMM diff --git a/modules/dasLLAMA/REVIEW_TOWER.md b/modules/dasLLAMA/REVIEW_TOWER.md index 67bd74a79b..b999e1b81b 100644 --- a/modules/dasLLAMA/REVIEW_TOWER.md +++ b/modules/dasLLAMA/REVIEW_TOWER.md @@ -83,6 +83,14 @@ call site skips when the hook serves. **A diff that makes a tower chain compute anything the CPU code its seat's hook replaces does not compute changes that CPU code the same way, in the same diff.** +**A diff that adds or changes a family's chain in the Metal tower or in the Metal ASR-decoder +driver (`dasllama/dasllama_metal_asr_dec.das`) leaves every stage of that family's served chain - +the mel, the front, the blocks, the tail, the cross-KV and the decode step - on the device, for +every model and every weight lane the family serves: a stage that runs on the CPU, and a decline +other than the knob's, the device's or a failed command buffer's, is a defect - moved to the +device in the same change, or filed in `followup_metal.md` with the stage and the model named.** +A stage on the CPU behind a Metal chain reads as served and sets the whole chain's rate. + **A diff that changes what a TTS seat's chain computes in a tower driver applies `REVIEW_TTS.md` too.** diff --git a/modules/dasLLAMA/dasllama/dasllama_gemma4a.das b/modules/dasLLAMA/dasllama/dasllama_gemma4a.das index 0a944eb57b..7754fbd899 100644 --- a/modules/dasLLAMA/dasllama/dasllama_gemma4a.das +++ b/modules/dasLLAMA/dasllama/dasllama_gemma4a.das @@ -86,7 +86,7 @@ struct Gemma4aEncoder { hann : PlaneF = PlaneF() // [512]: 320 Hann at the front, zeros to 512 twiddles : PlaneF = PlaneF() // DFT twiddle matrix [n_fft × (n_bins·2)] pos_emb : PlaneF = PlaneF() // sinusoidal RPE table [13 × d_model] - ones : PlaneF = PlaneF() // [proj_dim] weight for the final weightless RMSNorm + ones : PlaneF = PlaneF() // [mid] weight for the final weightless RMSNorm; its length IS the encoder's own output width (gemma4a_mid_dim) blob : PlaneF = PlaneF() qblob : PlaneI8 = PlaneI8() qscales : PlaneF = PlaneF() @@ -312,10 +312,12 @@ def load_gemma4a_encoder(path : string) : Gemma4aEncoder { let img = image_path_for(path, tag) let ts_img = ref_time_ticks() var t = Gemma4aEncoder() - if (load_image(img, t, tag)) { + // an image minted off a widening embedder read as a square one carries the encoder's width as the decoder's: mint again + if (load_image(img, t, tag) && t.proj_dim == gemma4a_probe_proj_dim(path)) { to_log(LOG_INFO, "dasLLAMA: prepared gemma4a-encoder image mapped in {get_time_usec(ts_img) / 1000} ms - {img}\n") return <- t } + delete t var inscope st <- stage_gemma4a_encoder(path) var built = Gemma4aEncoder() if (cache_via_image_staged(st, built, img, tag)) { @@ -356,10 +358,12 @@ def gemma4a_probe_proj_dim(path : string) : int64 { } var inscope m <- parse_gguf_meta(bytes) let out_proj = gguf_find_tensor(m, "a.pre_encode.out.weight") - if (out_proj >= 0 && gguf_has(m, "clip.audio.embedding_length")) { + let embedder = gguf_find_tensor(m, "mm.a.input_projection.weight") + if (out_proj >= 0 && embedder >= 0 && gguf_has(m, "clip.audio.embedding_length")) { let d = gguf_int(m, bytes, "clip.audio.embedding_length") - if (d > 0l) { - dim = m.tensors[out_proj].n_elem / d + let mid = d > 0l ? m.tensors[out_proj].n_elem / d : 0l + if (mid > 0l) { + dim = m.tensors[embedder].n_elem / mid // the embedder's rows: the decoder's width } } } @@ -408,8 +412,9 @@ def private g4a_repack_all(var st : Gemma4aStaging) { let ff = st.e.n_ff let feat = st.e.sub_ch1 * (st.e.n_mel / 4l) g4a_repack(st, st.e.inproj_w, feat, d) - g4a_repack(st, st.e.outproj_w, d, st.e.proj_dim) - g4a_repack(st, st.e.mmproj_w, st.e.proj_dim, st.e.proj_dim) + let mid = long_length(st.ones) + g4a_repack(st, st.e.outproj_w, d, mid) + g4a_repack(st, st.e.mmproj_w, mid, st.e.proj_dim) for (l in range64(st.e.n_layer)) { let lo = st.layers[l] g4a_repack(st, lo.ff_up, d, ff) @@ -435,6 +440,7 @@ def stage_gemma4a_encoder(path : string) : Gemma4aStaging { if (f == null) { panic("dasLLAMA gemma4a: cannot open mmproj gguf '{path}'") } + var mid = 0l //! the encoder's own output width: the out projection's rows, the weightless norm's, the embedder's columns fmap(f) $(var bytes : array#) { var inscope m <- parse_gguf_meta(bytes) let proj = (gguf_has(m, "clip.audio.projector_type") @@ -461,7 +467,6 @@ def stage_gemma4a_encoder(path : string) : Gemma4aStaging { } st.e.sub_ch0 = m.tensors[i0].n_elem / 9l // cin = 1 st.e.sub_ch1 = m.tensors[i1].n_elem / (9l * st.e.sub_ch0) - // proj_dim from the audio multimodal embedder mm.a.input_projection [proj_dim × proj_dim] let ip = gguf_find_tensor(m, "mm.a.input_projection.weight") if (ip < 0) { panic("dasLLAMA gemma4a: missing mm.a.input_projection.weight") @@ -470,7 +475,9 @@ def stage_gemma4a_encoder(path : string) : Gemma4aStaging { if (op < 0) { panic("dasLLAMA gemma4a: missing a.pre_encode.out.weight") } - st.e.proj_dim = m.tensors[op].n_elem / d // out proj [d × proj_dim] + // the encoder's own output width, then the decoder's: the audio multimodal embedder is square on E2B and widens on E4B + mid = m.tensors[op].n_elem / d // out proj [d × mid] + st.e.proj_dim = m.tensors[ip].n_elem / mid // mm.a.input_projection [mid × proj_dim] // blob layout: subsample + output first, then per-layer. GEMM fields ride the q cursor // on a q8 load (the field then indexes qblob); convs, norms, biases stay on the blob @@ -483,9 +490,9 @@ def stage_gemma4a_encoder(path : string) : Gemma4aStaging { st.e.norm1_w = o; o += st.e.sub_ch1 let feat = st.e.sub_ch1 * (st.e.n_mel / 4l) // 32 × 32 = 1024 st.e.inproj_w = take_cursor(q8, o, qo, d * feat) - st.e.outproj_w = take_cursor(q8, o, qo, st.e.proj_dim * d) - st.e.outproj_b = o; o += st.e.proj_dim - st.e.mmproj_w = take_cursor(q8, o, qo, st.e.proj_dim * st.e.proj_dim) + st.e.outproj_w = take_cursor(q8, o, qo, mid * d) + st.e.outproj_b = o; o += mid + st.e.mmproj_w = take_cursor(q8, o, qo, st.e.proj_dim * mid) st.layers |> resize(st.e.n_layer) for (l in range64(st.e.n_layer)) { var lo = G4aLayer() @@ -529,9 +536,9 @@ def stage_gemma4a_encoder(path : string) : Gemma4aStaging { gguf_read_tensor_f32(m, bytes, "a.conv1d.1.weight", st.blob, st.e.c1_w, st.e.sub_ch1 * st.e.sub_ch0 * 9l) gguf_read_tensor_f32(m, bytes, "a.conv1d.1.norm.weight", st.blob, st.e.norm1_w, st.e.sub_ch1) g4a_read_gemm(m, bytes, "a.input_projection.weight", st, wscratch, st.e.inproj_w, d * feat) - g4a_read_gemm(m, bytes, "a.pre_encode.out.weight", st, wscratch, st.e.outproj_w, st.e.proj_dim * d) - gguf_read_tensor_f32(m, bytes, "a.pre_encode.out.bias", st.blob, st.e.outproj_b, st.e.proj_dim) - g4a_read_gemm(m, bytes, "mm.a.input_projection.weight", st, wscratch, st.e.mmproj_w, st.e.proj_dim * st.e.proj_dim) + g4a_read_gemm(m, bytes, "a.pre_encode.out.weight", st, wscratch, st.e.outproj_w, mid * d) + gguf_read_tensor_f32(m, bytes, "a.pre_encode.out.bias", st.blob, st.e.outproj_b, mid) + g4a_read_gemm(m, bytes, "mm.a.input_projection.weight", st, wscratch, st.e.mmproj_w, st.e.proj_dim * mid) for (l in range64(st.e.n_layer)) { let lo = st.layers[l] let p = "a.blk.{l}" @@ -596,8 +603,8 @@ def stage_gemma4a_encoder(path : string) : Gemma4aStaging { st.pos_emb[pp * st.e.d_model + i + num_ts] = cos(scaled) } } - st.ones |> resize(st.e.proj_dim) - for (i in range64(st.e.proj_dim)) { + st.ones |> resize(mid) + for (i in range64(mid)) { st.ones[i] = 1.0 } return <- st @@ -1066,21 +1073,26 @@ def gemma4a_encode(enc : Gemma4aEncoder; var st : Gemma4aState) : int64 { def private g4a_tail_after_gpu(enc : Gemma4aEncoder; var st : Gemma4aState; npos : int64) : int64 => st.out_ready ? npos : g4a_projector_tail(enc, st, npos) +//! The encoder's own output width - the out projection's rows and the audio embedder's columns; `proj_dim` is the +//! embedder's rows, the decoder's width. Equal on E2B, 1536 against 2560 on E4B. +def gemma4a_mid_dim(enc : Gemma4aEncoder) : int64 => length(enc.ones) + def private g4a_projector_tail(enc : Gemma4aEncoder; var st : Gemma4aState; npos : int64) : int64 { let d = enc.d_model + let mid = gemma4a_mid_dim(enc) let tprof = ref_time_ticks() - ensure_length(st.tmp, npos * enc.proj_dim) + ensure_length(st.tmp, npos * mid) ensure_length(st.out, npos * enc.proj_dim) - g4a_mm_rq(enc, st, st.tmp, enc.outproj_w, st.x, d, enc.proj_dim, npos) - add_bias_rows(st.tmp, enc.blob, enc.outproj_b, enc.proj_dim, npos) + g4a_mm_rq(enc, st, st.tmp, enc.outproj_w, st.x, d, mid, npos) + add_bias_rows(st.tmp, enc.blob, enc.outproj_b, mid, npos) unsafe { var tp = addr(st.tmp[0]) let wp = plane_at(enc.ones, 0l) for (r in range64(npos)) { - rmsnorm(tp + r * enc.proj_dim, tp + r * enc.proj_dim, wp, enc.proj_dim, enc.eps) + rmsnorm(tp + r * mid, tp + r * mid, wp, mid, enc.eps) } } - g4a_mm_rq(enc, st, st.out, enc.mmproj_w, st.tmp, enc.proj_dim, enc.proj_dim, npos) + g4a_mm_rq(enc, st, st.out, enc.mmproj_w, st.tmp, mid, enc.proj_dim, npos) asr_prof_add("g4a.enc.proj", tprof) return npos } diff --git a/modules/dasLLAMA/dasllama/dasllama_metal_tower.das b/modules/dasLLAMA/dasllama/dasllama_metal_tower.das index 1f140fb8f1..8ac404a557 100644 --- a/modules/dasLLAMA/dasllama/dasllama_metal_tower.das +++ b/modules/dasLLAMA/dasllama/dasllama_metal_tower.das @@ -512,7 +512,7 @@ struct private G4aGpuCtx : TwGpuCtx { def private g4a_shape_ok(t : Gemma4aEncoder) : bool { let d = t.d_model return (!t.q8 && d % 64l == 0l && t.n_ff % 64l == 0l && t.n_head * t.d_head == d - && gemma4a_attn_past() <= 15l) + && gemma4a_attn_past() <= 15l && gemma4a_mid_dim(t) % 64l == 0l && t.proj_dim % 64l == 0l && gemma4a_mid_dim(t) <= int64(TW_ONES_DIM)) } def private g4a_ctx_make(t : Gemma4aEncoder; npos : int64) : G4aGpuCtx { @@ -558,6 +558,53 @@ def private g4a_ctx_release(var c : G4aGpuCtx) { tw_ctx_release(c) } +//! the projector tail's device rows and uniforms: the encoder-width rows the out projection lands, the decoder-width soft tokens +struct private G4aTail { + btmp : MetalBuffer? + bout : MetalBuffer? + u_mid : MetalBuffer? + u_proj : MetalBuffer? + u_totmid : MetalBuffer? + bytes_tmp : uint64 + bytes_out : uint64 +} + +def private g4a_tail_make(t : Gemma4aEncoder; c : G4aGpuCtx) : G4aTail { + let mid = gemma4a_mid_dim(t) + let bytes_tmp = uint64(c.mp * mid * 4l) + let bytes_out = uint64(c.mp * t.proj_dim * 4l) + return G4aTail(bytes_tmp = bytes_tmp, bytes_out = bytes_out, + btmp = pool_acquire_pinned(g_tw_pool, g_dev, bytes_tmp), bout = pool_acquire_pinned(g_tw_pool, g_dev, bytes_out), + u_mid = uniform_u32(uint(mid)), u_proj = uniform_u32(uint(t.proj_dim)), u_totmid = uniform_u32(uint(c.npos * mid))) +} + +//! The projector tail behind the blocks, in the blocks' command buffer: the out projection and its bias, the weightless +//! norm, the audio embedder - the CPU tail's four steps at its widths. +[arch(at="../ARCHITECTURE_GPU_TOWER.md#tower-encode-chains")] +def private g4a_tail_body(enc : MetalComputeEncoder?; t : Gemma4aEncoder; c : G4aGpuCtx; tl : G4aTail) { + let mid = gemma4a_mid_dim(t) + enc_f32_mm(enc, c.bblob, uint64(t.outproj_w * 4l), c.bx, tl.btmp, 0ul, c.u_d, tl.u_mid, c.mp, mid) + enc_add_bias_rows(enc, tl.btmp, 0ul, c.bblob, uint64(t.outproj_b * 4l), tl.u_mid, tl.u_totmid, c.npos * mid) + pf_enc_rms(enc, tl.btmp, 0ul, g_tw_ones, tl.btmp, tl.u_mid, c.u_eps, c.npos) + enc_f32_mm(enc, c.bblob, uint64(t.mmproj_w * 4l), tl.btmp, tl.bout, 0ul, tl.u_mid, tl.u_proj, c.mp, t.proj_dim) +} + +//! the tail's soft tokens into `s.out` where the command buffer ran, the CPU tail told to stand down; the tail's buffers released +def private g4a_tail_land(t : Gemma4aEncoder; c : G4aGpuCtx; var tl : G4aTail; ran : bool; var s : Gemma4aState) { + if (ran) { + ensure_length(s.out, c.npos * t.proj_dim) + unsafe { + memcpy(addr(s.out[0]), metal_buffer_contents(tl.bout), uint64(c.npos * t.proj_dim * 4l)) + } + s.out_ready = true + } + pool_release(g_tw_pool, tl.btmp, tl.bytes_tmp) + pool_release(g_tw_pool, tl.bout, tl.bytes_out) + pool_release(g_upool, tl.u_mid, 4ul) + pool_release(g_upool, tl.u_proj, 4ul) + pool_release(g_upool, tl.u_totmid, 4ul) +} + //! The 12-block dispatch chain on an already-armed context - the blocks-only and whole-chunk drivers share it. [arch(at="../ARCHITECTURE_GPU_TOWER.md#tower-encode-chains")] def private g4a_blocks_body(enc : MetalComputeEncoder?; t : Gemma4aEncoder; c : G4aGpuCtx) { @@ -630,12 +677,15 @@ def private metal_gemma4a_blocks(t : Gemma4aEncoder; var s : Gemma4aState; npos residency_flush() // pin/commit before the first submission (the post-CPU-front slack) let d = t.d_model var c <- g4a_ctx_make(t, npos) + var tl <- g4a_tail_make(t, c) upload_rows_padded(c.bx, unsafe(addr(s.x[0])), npos, c.mp, d) var err = "" let ran = with_compute_encoder(g_queue, err) $(enc : MetalComputeEncoder?) { // nolint:PERF026 — the error-text fetch runs only on the failure leg g4a_blocks_body(enc, t, c) + g4a_tail_body(enc, t, c, tl) } let served = tw_readback(c, ran, s.x, d, t.n_layer) + g4a_tail_land(t, c, tl, ran, s) g4a_ctx_release(c) if (!served) { return tw_decline(MetalTowerDecline.gpu_error) @@ -1161,6 +1211,7 @@ def private metal_gemma4a_chunk(t : Gemma4aEncoder; var s : Gemma4aState; n_len let kd1 = t.sub_ch0 * 9l let conv1_cout_pad = round_up(t.sub_ch1, 64l) var c <- g4a_ctx_make(t, npos) + var tl <- g4a_tail_make(t, c) let bytes_xw = uint64(mpd * n_fft * 4l) let bytes_tw = uint64(dft_out_pad * n_fft * 4l) let bytes_reim = uint64(mpd * dft_out_pad * 4l) @@ -1232,7 +1283,9 @@ def private metal_gemma4a_chunk(t : Gemma4aEncoder; var s : Gemma4aState; n_len //! input projection: [npos x feat] -> [npos x d] enc_f32_mm(enc, c.bblob, uint64(t.inproj_w * 4l), c.bxb, c.bx, 0ul, u_feat, c.u_d, c.mp, d) g4a_blocks_body(enc, t, c) + g4a_tail_body(enc, t, c, tl) } + g4a_tail_land(t, c, tl, ran, s) var served = false if (ran) { ensure_length(s.x, npos * d) diff --git a/modules/dasLLAMA/dasllama/dasllama_vulkan_tower.das b/modules/dasLLAMA/dasllama/dasllama_vulkan_tower.das index 8612fc61c0..fa5a231410 100644 --- a/modules/dasLLAMA/dasllama/dasllama_vulkan_tower.das +++ b/modules/dasLLAMA/dasllama/dasllama_vulkan_tower.das @@ -1156,7 +1156,7 @@ def private vulkan_gemma4a_blocks(t : Gemma4aEncoder; var s : Gemma4aState; npos let rel_cap_rows = past + 1l let tail_dim = t.proj_dim if (d % 64l != 0l || ff % 64l != 0l || ff < 2l * d || t.n_head * hs != d || (hs != 64l && hs != 128l) || past != int64(G4A_ATTN_PAST) || gemma4a_softcap() != G4A_ATTN_CAP - || npos > VT_MAX_ENCODE_ROWS || npos <= 0l || tail_dim % 64l != 0l || tail_dim > max(d, ff)) { + || npos > VT_MAX_ENCODE_ROWS || npos <= 0l || tail_dim % 64l != 0l || tail_dim > max(d, ff) || tail_dim != gemma4a_mid_dim(t)) { return vt_decline(r, VulkanTowerDecline.shape) } let arm = vt_arm(r) diff --git a/modules/dasLLAMA/followup_metal.md b/modules/dasLLAMA/followup_metal.md index 4507705985..211e371129 100644 --- a/modules/dasLLAMA/followup_metal.md +++ b/modules/dasLLAMA/followup_metal.md @@ -886,3 +886,20 @@ idle wall. The rows are `harness/served_bench.das --scene --scene-asr --sce (each lane's median alone beside its median in the scene). The work: a kernel profile of the scene that says where the two transcriptions meet, and the dispatch the speech workers take set from it. + +## 39. Audio stages a Metal chain leaves on the CPU + +A Metal chain serves every stage of its family on the device (`REVIEW_TOWER.md`); these do not. +Each is a stage and the models it holds for, read off the drivers' own gates: + +- **The whisper mel** runs on the CPU on every whisper size, and the whisper-class conv stem on a + model whose 3 x n_mel im2col width is off the 32 lattice (tiny, base, small, medium: 80 mels). +- **The whisper decode step** serves on the device from a text width of 1024 up + (`g_wd_step_min_d`); below it the step runs on the CPU, and the encoder's post-norm on every size. +- **The parakeet decode step** (the TDT predictor and joint) is a CPU team publish. +- **The q8 encoder lane** declines `quant_mode` on gemma4a, canary and parakeet, so those families + serve their f32 lane on the device at the larger image; the whisper-class tower reads q8. +- **Qwen3-ASR, Qwen3-Omni and Canary-Qwen** have had no stage-by-stage check. + +The work: each stage on the device, and a counter per stage a gate can read, so a served model's +row names the stages it ran and where. diff --git a/modules/dasLLAMA/followup_vulkan.md b/modules/dasLLAMA/followup_vulkan.md index d7067ba12d..82adfc69df 100644 --- a/modules/dasLLAMA/followup_vulkan.md +++ b/modules/dasLLAMA/followup_vulkan.md @@ -2184,3 +2184,9 @@ module) is independent and can land any time - it is pure structure. `tq4_rotate_for_store` / `tq4_unrotate_from_store`, which the per-op rails call; the Metal sites need an Apple run. Each fold is behavior-neutral; the proof is `test_vulkan_kv_codec_kernels.das` with the codec stamps' SPIR-V diff for the first and the Metal decode parity arms `arm7b-tq4kv` for the second. +139. **The gemma4a chain declines a widening audio embedder.** The Vulkan chain's projector tail sizes + one tail width for the out projection, the norm and the embedder (`tail_dim`), which holds where + the embedder is square (E2B). On E4B the embedder widens the encoder's 1536 to the decoder's 2560 + (`gemma4a_mid_dim` against `proj_dim`), and the chain declines `shape`, so the CPU chain serves + E4B audio. Done = the tail's regions, tiles and buffers at the two widths (the Metal chain's + `g4a_tail_body` is the form), and `test_gemma4a_vulkan_twin` run on the E4B pair. diff --git a/modules/dasLLAMA/tests/CLAUDE.md b/modules/dasLLAMA/tests/CLAUDE.md index 11dd4f614f..1b17edadae 100644 --- a/modules/dasLLAMA/tests/CLAUDE.md +++ b/modules/dasLLAMA/tests/CLAUDE.md @@ -241,8 +241,10 @@ twin-knob freeze and whisper's own wblob-ONLY poison that must CHANGE the GPU tr (both legs are whisper's alone; qwen3a carries neither; a twin-W route reads its GEMM weights from `wblob` alone, so zeroing that buffer alone poisons it, while a route that also reads the f32 plane, `fblob`, is poisoned only with both zeroed), the gemma4a Metal -Conformer cell (f32-lane transcript equality CPU vs GPU + encode rel-l2 + counter deltas - -the lane pin/reset discipline mirrors qwen3a's), the canary Metal FastConformer cell (the +Conformer cells, E2B and E4B (f32-lane transcript equality CPU vs GPU + the soft tokens' rel-l2 + +counter deltas, the projector tail run on the device - `out_ready` set on the device leg and clear +on the CPU leg - and the two projector widths read off the file: the encoder's 1536 on both, the +embedder's 1536 on E2B and 2560 on E4B; the lane pin/reset discipline mirrors qwen3a's), the canary Metal FastConformer cell (the same discipline over the rel-pos XL block loop; decoder = the q8_0 serving artifact), the parakeet Metal FastConformer cell (the same chain over parakeet's f32 blob, minted in memory; transcript equality CPU vs GPU + the encoder rows' rel-l2 + counter deltas - the subsample front diff --git a/modules/dasLLAMA/tests/test_model_image.das b/modules/dasLLAMA/tests/test_model_image.das index f2a33fbd7d..649096cc50 100644 --- a/modules/dasLLAMA/tests/test_model_image.das +++ b/modules/dasLLAMA/tests/test_model_image.das @@ -2266,11 +2266,21 @@ def test_gemma4a_transcribe_q8_ab(t : T?) { [test] def test_gemma4a_tower_metal(t : T?) { - t |> run("gemma4a tower: the Metal Conformer block loop matches the CPU f32 rail on jfk") @(t : T?) { + g4a_tower_metal_cell(t, "E2B", "gemma-4-E2B-it-Q8_0.gguf", "mmproj-gemma-4-E2B-it-bf16.gguf", 1536l) +} + +//! E4B's audio embedder widens the encoder's 1536 to the decoder's 2560, where E2B's is square +[test] +def test_gemma4a_tower_metal_e4b(t : T?) { + g4a_tower_metal_cell(t, "E4B", "gemma-4-E4B-it-Q8_0.gguf", "mmproj-gemma-4-E4B-it-BF16.gguf", 2560l) +} + +def private g4a_tower_metal_cell(tst : T?; tag, dec_file, mmp_file : string; want_proj : int64) { + tst |> run("gemma4a tower {tag}: the Metal Conformer chain, the projector tail on the device, matches the CPU f32 rail on jfk") @(t : T?) { static_if (typeinfo builtin_module_exists(das_metal)) { return if (!family_on(t, "gemma4a") || !arm_on(t, "mtower") || !jit_only(t)) - let dec_path = path_join(models_dir(), "gemma-4-E2B-it-Q8_0.gguf") - let mmp_path = path_join(models_dir(), "mmproj-gemma-4-E2B-it-bf16.gguf") + let dec_path = path_join(models_dir(), dec_file) + let mmp_path = path_join(models_dir(), mmp_file) if (!model_available(t, dec_path) || !model_available(t, mmp_path)) return var samples <- read_wav_pcm16_mono(path_join(models_dir(), "jfk.wav")) if (empty(samples)) { @@ -2290,6 +2300,9 @@ def test_gemma4a_tower_metal(t : T?) { set_gemma4a_q8(false) var inscope enc <- load_gemma4a_encoder(mmp_path) t |> success(!enc.q8, "tower is on the f32 rail") + t |> equal(enc.proj_dim, want_proj, "the embedder projects to the decoder's width") + t |> equal(enc.proj_dim, dec.config.dim) + t |> equal(gemma4a_mid_dim(enc), 1536l, "the encoder's own output width") nlayer = enc.n_layer set_metal_tower(false) var inscope s <- create_gemma4a_asr_session(dec) @@ -2314,6 +2327,7 @@ def test_gemma4a_tower_metal(t : T?) { gemma4a_log_mel(enc, samples, 0l, long_length(samples), stg) gemma4a_encode(enc, stg) enc_rel_l2 = rel_l2(stg.out, stc.out) + t |> success(stg.out_ready && !stc.out_ready, "the device leg ran the projector tail, the CPU leg its own") reset_gemma4a_q8() delete dec } From a441d1649b5af6f59fff931be1e1ef003b21c65f Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Sat, 3 Oct 2026 23:26:08 -0700 Subject: [PATCH 06/46] dasLLAMA: the Metal tower fills the whisper blocks-with-post-norm seat - the encoder's post-norm runs behind the blocks on the device The seat existed and only the Vulkan driver filled it, so on Metal the CPU normed the rows the device read back. One row pass in the blocks' command buffer lands the normed rows in xb. Co-Authored-By: Claude Fable 5.1 --- modules/dasLLAMA/ARCHITECTURE_MEDIA.md | 4 +- modules/dasLLAMA/dasllama/dasllama_audio.das | 4 +- .../dasllama/dasllama_metal_tower.das | 23 +++++- modules/dasLLAMA/followup_metal.md | 2 +- modules/dasLLAMA/tests/CLAUDE.md | 6 +- modules/dasLLAMA/tests/test_whisper.das | 72 +++++++++++++++++++ 6 files changed, 102 insertions(+), 9 deletions(-) diff --git a/modules/dasLLAMA/ARCHITECTURE_MEDIA.md b/modules/dasLLAMA/ARCHITECTURE_MEDIA.md index 5c0ff47a3f..9a0917aa77 100644 --- a/modules/dasLLAMA/ARCHITECTURE_MEDIA.md +++ b/modules/dasLLAMA/ARCHITECTURE_MEDIA.md @@ -191,8 +191,8 @@ blocks seats it serves and the front seats it fills (qwen3a's mel and conv front whole chunk, canary's front, parakeet's whole encode - the Metal driver's, asked ahead of the CPU front with the blocks seat behind a decline - and the whisper-class blocks-with-post-norm seat `register_tower_blocks_ln_post_gpu`, which the whisper encode asks ahead of its CPU block loop and -post-norm - a decline asks no second seat, its driver's blocks seat declining the same way, while a -driver with no post-norm seat (Metal) serves the blocks at the blocks seat and the CPU norms; the +post-norm - a decline asks no second seat, its driver's blocks seat declining the same way; both +tower drivers fill it, the post-norm one more row pass in the blocks' command buffer; the seat serves the family whose post-norm follows the blocks with nothing between, whisper, while the families that pool before it, and ultravox, keep the CPU post-norm). The direction is forced: the driver requires the family file for its diff --git a/modules/dasLLAMA/dasllama/dasllama_audio.das b/modules/dasLLAMA/dasllama/dasllama_audio.das index fe3e907b4c..ed53efa4e6 100644 --- a/modules/dasLLAMA/dasllama/dasllama_audio.das +++ b/modules/dasLLAMA/dasllama/dasllama_audio.das @@ -1341,8 +1341,8 @@ def audio_encode_blocks(t : AudioTower; var s : EncoderState; mel : MelChunk) { } //! the stem, then the blocks with the tower's post-norm folded where the blocks-with-post-norm seat serves (true: `s.xb` holds the -//! normed rows), else the blocks alone (false: `s.x` holds their output and the caller norms it) - a driver with no post-norm -//! seat (Metal) serves them at the blocks seat; a post-norm seat that declines asks no second seat, its blocks seat declines alike +//! normed rows), else the blocks alone (false: `s.x` holds their output and the caller norms it) - a build with no post-norm +//! seat serves them at the blocks seat; a post-norm seat that declines asks no second seat, its blocks seat declines alike def audio_encode_blocks_ln_post(t : AudioTower; var s : EncoderState; mel : MelChunk) : bool { audio_encode_stem(t, s, mel) let tt = t.n_ctx diff --git a/modules/dasLLAMA/dasllama/dasllama_metal_tower.das b/modules/dasLLAMA/dasllama/dasllama_metal_tower.das index 8ac404a557..7a32cacdc9 100644 --- a/modules/dasLLAMA/dasllama/dasllama_metal_tower.das +++ b/modules/dasLLAMA/dasllama/dasllama_metal_tower.das @@ -284,7 +284,16 @@ def private wt_mm(enc : MetalComputeEncoder?; q8 : bool; site : int64; f16w, bf1 // non-causal uniform span). Frontend and tail stay CPU. Pad-row garbage is // CONTAINED: QK/AV zero k/v rows >= npos, softmax zeroes [uend, np32), pad outputs go unread. [hot_path, arch(at="../ARCHITECTURE_IMAGE.md#image-tower-twin-plane")] -def private metal_tower_blocks(t : AudioTower; var s : EncoderState; npos : int64) : bool { // nolint:STYLE038 — the dispatch chain is the shape +def private metal_tower_blocks(t : AudioTower; var s : EncoderState; npos : int64) : bool + => metal_tower_blocks_form(t, s, npos, false) + +//! the blocks with the tower's post-norm behind them in the same command buffer: the normed rows land in `s.xb` +[hot_path] +def private metal_tower_blocks_ln_post(t : AudioTower; var s : EncoderState; npos : int64) : bool + => metal_tower_blocks_form(t, s, npos, true) + +[hot_path, arch(at="../ARCHITECTURE_IMAGE.md#image-tower-twin-plane")] +def private metal_tower_blocks_form(t : AudioTower; var s : EncoderState; npos : int64; ln_post : bool) : bool { // nolint:STYLE038 — the dispatch chain is the shape if (!g_tw_env_tower) { return tw_decline(MetalTowerDecline.knob) } @@ -365,11 +374,18 @@ def private metal_tower_blocks(t : AudioTower; var s : EncoderState; npos : int6 enc_add_bias_rows(enc, bxb2, 0ul, bblob, uint64(lo.fc2_b * 4l), u_d, u_totd, npos * d) enc_add(enc, bx, 0ul, bxb2, 0ul, u_totd, npos * d) } + if (ln_post) { + enc_ln(enc, bx, bblob, uint64(t.lnp_w_off * 4l), bblob, uint64(t.lnp_b_off * 4l), bxb, 0ul, u_d, u_eps, npos) + } } var served = false if (ran) { - unsafe { - memcpy(addr(s.x[0]), metal_buffer_contents(bx), uint64(npos * d * 4l)) + unsafe { // the post-norm seat lands the normed rows where the CPU post-norm would: `s.xb` + if (ln_post) { + memcpy(addr(s.xb[0]), metal_buffer_contents(bxb), uint64(npos * d * 4l)) + } else { + memcpy(addr(s.x[0]), metal_buffer_contents(bx), uint64(npos * d * 4l)) + } } tw_count_encode(npos, t.n_layer, f16w || bf16w) served = true @@ -4674,6 +4690,7 @@ def dasllama_metal_tower_register { register_qwen3v_gpu(@@metal_qwen3v_blocks, @@metal_tower_serves) register_qwen25v_gpu(@@metal_qwen25v_blocks) register_tower_blocks_gpu(@@metal_tower_blocks, @@metal_tower_serves) + register_tower_blocks_ln_post_gpu(@@metal_tower_blocks_ln_post) register_tower_conv_gpu(@@metal_conv_frontend) register_qwen3a_front_gpu(@@metal_q3a_front) register_qwen3a_mel_gpu(@@metal_q3a_mel) diff --git a/modules/dasLLAMA/followup_metal.md b/modules/dasLLAMA/followup_metal.md index 211e371129..96f81f7933 100644 --- a/modules/dasLLAMA/followup_metal.md +++ b/modules/dasLLAMA/followup_metal.md @@ -895,7 +895,7 @@ Each is a stage and the models it holds for, read off the drivers' own gates: - **The whisper mel** runs on the CPU on every whisper size, and the whisper-class conv stem on a model whose 3 x n_mel im2col width is off the 32 lattice (tiny, base, small, medium: 80 mels). - **The whisper decode step** serves on the device from a text width of 1024 up - (`g_wd_step_min_d`); below it the step runs on the CPU, and the encoder's post-norm on every size. + (`g_wd_step_min_d`); below it the step runs on the CPU. - **The parakeet decode step** (the TDT predictor and joint) is a CPU team publish. - **The q8 encoder lane** declines `quant_mode` on gemma4a, canary and parakeet, so those families serve their f32 lane on the device at the larger image; the whisper-class tower reads q8. diff --git a/modules/dasLLAMA/tests/CLAUDE.md b/modules/dasLLAMA/tests/CLAUDE.md index 1b17edadae..81297e9499 100644 --- a/modules/dasLLAMA/tests/CLAUDE.md +++ b/modules/dasLLAMA/tests/CLAUDE.md @@ -1708,7 +1708,11 @@ token and bar, the batch with its last token changed landing outside the bar as CPU blocks over them transcribe the all-CPU chain's text), `test_whisper_vulkan_wdec_lifetime` (tiny, one session reused the way a serving worker reuses one: a model drop between two transcriptions - the second serves again and reads the same; the decoder knob turned off between two - the second reads as a fresh knob-off session; the block hooks pinned off after a served window - no -handoff for the CPU-encoded windows, the text of the CPU-encoder chain), `test_whisper_metal_wdec_flush` +handoff for the CPU-encoded windows, the text of the CPU-encoder chain), `test_whisper_metal_ln_post` +(tiny, the f32 encoder minted in memory: the blocks-with-post-norm seat declines with the tower +off and serves with it on, one device encode counted, and the device's normed rows are the CPU +post-norm of the device's own block rows within 1e-4 of the largest - reads 3e-7 - with the rows +before the post-norm the control), `test_whisper_metal_wdec_flush` (tiny, the step's floor lifted so the Metal step serves and the window's cross-KV layouts wait on the device: the host layouts zeroed, a 33-row first batch hands the window to the CPU chain, the layouts land first and its logits match the knob-off chain's token and bar, the batch with its diff --git a/modules/dasLLAMA/tests/test_whisper.das b/modules/dasLLAMA/tests/test_whisper.das index a98ac38381..f262aa58d4 100644 --- a/modules/dasLLAMA/tests/test_whisper.das +++ b/modules/dasLLAMA/tests/test_whisper.das @@ -777,6 +777,78 @@ def test_whisper_vulkan_wdec_flush(t : T?) { } } +//! the largest element difference of two row images over the reference's largest magnitude; 2 where a length differs +def private rows_rel_max(got, want : array) : float { + return 2.0 if (length(got) != length(want) || empty(want)) + var diff = 0.0 + var peak = 0.0 + for (g, w in got, want) { + diff = max(diff, abs(g - w)) + peak = max(peak, abs(w)) + } + return diff / max(peak, 1e-30) +} + +//! The Metal tower fills the whisper blocks-with-post-norm seat: the post-norm runs behind the blocks on the device and the +//! normed rows land in `xb`, where the CPU chain's post-norm would put them. +[test] +def test_whisper_metal_ln_post(t : T?) { + t |> run("ggml-tiny.bin: the encoder's post-norm on the Metal tower lands the CPU chain's normed rows") @(tt : T?) { + static_if (typeinfo builtin_module_exists(das_metal)) { + let model_path = path_join(whisper_dir(), "ggml-tiny.bin") + return if (!jit_model_available(tt, model_path)) + let samples <- read_wav_pcm16_mono(path_join(models_dir(), "jfk.wav")) + if (empty(samples)) { + tt |> skip("jfk.wav not present") + return + } + var inscope w <- mint_whisper(model_path, false, false, -1l) + with_dasllama_jobque_() { + var padded : array + var mel = WhisperMel() + log_mel_whisper(samples, w.mel, w.filters, w.enc.n_mel, padded, mel) + var win = MelChunk(n_mel = w.enc.n_mel, n_len = AUDIO_CHUNK_FRAMES) + win.data |> resize(w.enc.n_mel * AUDIO_CHUNK_FRAMES) + whisper_mel_window(mel, 0l, win) + var es <- make_encoder_state(w.enc) + set_metal_tower(false) + tt |> success(!audio_encode_blocks_ln_post(w.enc, es, win), "tower off: the seat declines and the CPU owes the post-norm") + var raw := es.x // the blocks' rows before the post-norm: the control's reference + whisper_encode(w.enc, es, win) + var cpu := es.xb + set_metal_tower(true) + let s0 = metal_tower_stats() + tt |> success(audio_encode_blocks_ln_post(w.enc, es, win), "tower on: the seat served the blocks and the post-norm") + tt |> equal(metal_tower_stats().encodes - s0.encodes, 1l, "one device encode") + let off = rows_rel_max(es.xb, cpu) + let unnormed = rows_rel_max(es.xb, raw) + var normed := es.xb + // the post-norm is the unit: the blocks seat alone gives the device's own block rows, the CPU norms THOSE, and the + // device's normed rows must be that image - the block chain's device-vs-CPU distance stays out of the bar + audio_encode_blocks(w.enc, es, win) + var want : array + want |> resize(length(es.x)) + layernorm_batch(want, es.x, w.enc.fblob, w.enc.lnp_w_off, w.enc.lnp_b_off, w.enc.d_model, w.enc.n_ctx, w.enc.eps) + let ln_off = rows_rel_max(normed, want) + to_log(LOG_INFO, "whisper tiny ln_post: device norm vs the CPU norm of the device's block rows {ln_off}; vs the CPU chain {off}; vs the un-normed rows {unnormed}\n") + tt |> success(ln_off < 1e-4, "the device's post-norm is the CPU post-norm of the same rows ({ln_off})") + tt |> success(unnormed > 0.1, "the control: against the rows before the post-norm the compare misses ({unnormed})") + delete want + delete normed + delete raw + delete cpu + delete es + delete win + delete mel + delete padded + } + metal_tower_shutdown() + } else { + tt |> skip("dasMetal not present") + } + } +} + //! The Metal decoder leaves a window's cross-KV layouts on the device while its step serves; a first batch past the step's //! 32-row cap hands the window to the CPU chain, and the layouts land first. [test] From 20c1a39cea5f4830b8845bd7e634d2b297ec4be8 Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Sat, 3 Oct 2026 23:32:40 -0700 Subject: [PATCH 07/46] dasLLAMA: the Metal whisper decode step serves from a text width of 512 - base and small decode on the device The floor stood at 1024 on a reading that predates the step's current kernels. One request's decode on the M5 Max, CPU against device: tiny 17.8 / 21.8 ms, base 32.6 / 27.9, small 70.9 / 56.4, medium 179.9 / 119.8, large-v3-turbo 49.0 / 33.1. The device step wins from base up and loses on tiny, so the floor is 512. Co-Authored-By: Claude Fable 5.1 --- modules/dasLLAMA/ENVIRONMENT.md | 2 +- modules/dasLLAMA/dasllama/dasllama_env.das | 2 +- modules/dasLLAMA/dasllama/dasllama_metal_asr_dec.das | 4 +++- modules/dasLLAMA/followup_metal.md | 5 +++-- modules/dasLLAMA/tests/test_model_image.das | 2 +- modules/dasLLAMA/tests/test_whisper.das | 4 ++-- 6 files changed, 11 insertions(+), 8 deletions(-) diff --git a/modules/dasLLAMA/ENVIRONMENT.md b/modules/dasLLAMA/ENVIRONMENT.md index 50287e8793..30eb89bd5e 100644 --- a/modules/dasLLAMA/ENVIRONMENT.md +++ b/modules/dasLLAMA/ENVIRONMENT.md @@ -90,7 +90,7 @@ Apple GPU backend. Absent on non-Apple builds, where setting them does nothing. | `DASLLAMA_METAL_TOWER_F16` | flag | on | Serve a tower's weighted GEMMs from a halfword twin of its weights on the crowned tensor mul_mm twins: the image's baked f16 twin plane where the mint bakes one (only for f16-sourced GEMMs, so it holds the file's own values), and the FastConformer chain's device twin of parakeet's f32 blob. Needs a tensor crown - a box without one, or an image without the plane, keeps the f32 route; 0 pins the f32 GEMM route - the A/B rail. | | `DASLLAMA_METAL_TOWER_FLASH` | flag | on | Serve head-72 tower attention at %64 canvases on the lifted flash kernel (no score slab, no rowstat pass, no head restrides - the gemma3v-geometry ViTs); 0 pins the three-pass slab - the A/B rail. | | `DASLLAMA_METAL_WDEC` | flag | on | Serve the whisper decoder side (cross-KV; the decode step under wdec_step) on the Metal ASR-decoder driver; 0 pins the CPU decoder. | -| `DASLLAMA_METAL_WDEC_STEP` | flag | on | Serve the whisper DECODE STEP on the GPU too (needs wdec). Small decoders (n_text_state under the 1024 floor; set_metal_wdec_step_min_d) keep the CPU rail - the per-dispatch latency floor beats them. | +| `DASLLAMA_METAL_WDEC_STEP` | flag | on | Serve the whisper DECODE STEP on the GPU too (needs wdec). Small decoders (n_text_state under the 512 floor; set_metal_wdec_step_min_d) keep the CPU rail - the per-dispatch latency floor beats them. | | `DASLLAMA_METAL_ATTN_D` | flag | on | Fused single-pass decode attention (a head of 64, 96, 128, 256 or 512 on every KV codec; sliding windows, attention sinks, the logit soft cap and a second head class included), on the single row, the speculative verify's rows and the batch; 0 is the A/B rail to the chunked pair. | | `DASLLAMA_METAL_ATTN_SINGLE` | number | 64 | Row count below which attention uses the single-chunk kernel; clamped to 128. | | `DASLLAMA_METAL_MULMM` | flag | on | The mul_mm prefill GEMM; 0 falls back to the legacy per-op path. | diff --git a/modules/dasLLAMA/dasllama/dasllama_env.das b/modules/dasLLAMA/dasllama/dasllama_env.das index 640d5b9e4e..8b7dfdc5bd 100644 --- a/modules/dasLLAMA/dasllama/dasllama_env.das +++ b/modules/dasLLAMA/dasllama/dasllama_env.das @@ -241,7 +241,7 @@ struct public MetalEnv { @clarg_doc = "Serve the whisper decoder side (cross-KV; the decode step under wdec_step) on the Metal ASR-decoder driver; 0 pins the CPU decoder." wdec : bool = true - @clarg_doc = "Serve the whisper DECODE STEP on the GPU too (needs wdec). Small decoders (n_text_state under the 1024 floor; set_metal_wdec_step_min_d) keep the CPU rail - the per-dispatch latency floor beats them." + @clarg_doc = "Serve the whisper DECODE STEP on the GPU too (needs wdec). Small decoders (n_text_state under the 512 floor; set_metal_wdec_step_min_d) keep the CPU rail - the per-dispatch latency floor beats them." wdec_step : bool = true @clarg_doc = "Fused single-pass decode attention (a head of 64, 96, 128, 256 or 512 on every KV codec; sliding windows, attention sinks, the logit soft cap and a second head class included), on the single row, the speculative verify's rows and the batch; 0 is the A/B rail to the chunked pair." diff --git a/modules/dasLLAMA/dasllama/dasllama_metal_asr_dec.das b/modules/dasLLAMA/dasllama/dasllama_metal_asr_dec.das index cf9d3fb0c8..6501efb006 100644 --- a/modules/dasLLAMA/dasllama/dasllama_metal_asr_dec.das +++ b/modules/dasLLAMA/dasllama/dasllama_metal_asr_dec.das @@ -29,7 +29,9 @@ var private g_wd_failed = false var private g_wd_pool : MetalBufferPool var private g_wd_env = true var private g_wd_step_env = true // the decode-step half (DASLLAMA_METAL_WDEC_STEP); windows capture it -var private g_wd_step_min_d = 1024l // the step's serving floor (measured: d=1280 wins 1.6x, d=384 loses) +//! The text width the decode step serves from: at 384 (tiny) the device step is slower than the CPU's, from 512 (base) up it is faster. +let METAL_WDEC_STEP_MIN_D = 512l +var private g_wd_step_min_d = METAL_WDEC_STEP_MIN_D // the attached decoder, keyed on the dq plane (base, length): ONE live whisper decoder per // process is the serving contract — a key mismatch re-attaches; tests switching models call // metal_wdec_shutdown between them (address reuse after an unmap could alias the key) diff --git a/modules/dasLLAMA/followup_metal.md b/modules/dasLLAMA/followup_metal.md index 96f81f7933..9345f13067 100644 --- a/modules/dasLLAMA/followup_metal.md +++ b/modules/dasLLAMA/followup_metal.md @@ -894,8 +894,9 @@ Each is a stage and the models it holds for, read off the drivers' own gates: - **The whisper mel** runs on the CPU on every whisper size, and the whisper-class conv stem on a model whose 3 x n_mel im2col width is off the 32 lattice (tiny, base, small, medium: 80 mels). -- **The whisper decode step** serves on the device from a text width of 1024 up - (`g_wd_step_min_d`); below it the step runs on the CPU. +- **The whisper decode step** serves on the device from a text width of 512 up + (`METAL_WDEC_STEP_MIN_D`); tiny's (384) runs on the CPU, where the device step is the slower + one - 21.8 ms a request's decode against 17.8. - **The parakeet decode step** (the TDT predictor and joint) is a CPU team publish. - **The q8 encoder lane** declines `quant_mode` on gemma4a, canary and parakeet, so those families serve their f32 lane on the device at the larger image; the whisper-class tower reads q8. diff --git a/modules/dasLLAMA/tests/test_model_image.das b/modules/dasLLAMA/tests/test_model_image.das index 649096cc50..58da3f8ba3 100644 --- a/modules/dasLLAMA/tests/test_model_image.das +++ b/modules/dasLLAMA/tests/test_model_image.das @@ -1512,7 +1512,7 @@ def test_whisper_metal_cross_kv(tst : T?) { t |> success(rearm_text == gpu_text, "post-shutdown transcript unchanged") // the serving FLOOR: at the default min_d, tiny cross-KV-serves but never steps, // and the small decline is recorded as policy - set_metal_wdec_step_min_d(1024l) + set_metal_wdec_step_min_d(METAL_WDEC_STEP_MIN_D) let sm0 = metal_wdec_stats() let small_text = wdec_transcribe(w8, samples) let sm1 = metal_wdec_stats() diff --git a/modules/dasLLAMA/tests/test_whisper.das b/modules/dasLLAMA/tests/test_whisper.das index f262aa58d4..043acd7084 100644 --- a/modules/dasLLAMA/tests/test_whisper.das +++ b/modules/dasLLAMA/tests/test_whisper.das @@ -898,7 +898,7 @@ def test_whisper_metal_wdec_flush(t : T?) { tt |> equal(metal_wdec_stats().steps, d1.steps, "the device served no step of the wide batch: the CPU chain decoded it over the flushed memory") dev_logits := sg.w.ds.logits wdec_flush_compare(tt, m.whisper, cpu_logits, dev_logits, cpu_moved) // inside the queue: the argmax is the parallel one - set_metal_wdec_step_min_d(1024l) + set_metal_wdec_step_min_d(METAL_WDEC_STEP_MIN_D) } delete cpu_logits delete cpu_moved @@ -1232,7 +1232,7 @@ def test_whisper_crowned_word_level(t : T?) { set_metal_tower(true) // restore: the defaults later cells' GPU-serving claims see set_metal_wdec(true) set_metal_wdec_step(true) - set_metal_wdec_step_min_d(1024l) // the measured serving floor (the declaration default) + set_metal_wdec_step_min_d(METAL_WDEC_STEP_MIN_D) // the measured serving floor (the declaration default) delete m delete mm } else { From 66f7da48def087f719058fe333b9db96e3585081 Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Sat, 3 Oct 2026 23:40:06 -0700 Subject: [PATCH 08/46] dasLLAMA: a whisper.cpp q8_0 bin loads - the loader reads Q8_0 tensors as each block's scale times its quants, where it refused every file type but f32 and f16 Co-Authored-By: Claude Fable 5.1 --- .../dasLLAMA/dasllama/dasllama_whisper.das | 33 +++++++++++++++---- modules/dasLLAMA/tests/CLAUDE.md | 4 ++- modules/dasLLAMA/tests/test_whisper.das | 32 ++++++++++++++++++ 3 files changed, 61 insertions(+), 8 deletions(-) diff --git a/modules/dasLLAMA/dasllama/dasllama_whisper.das b/modules/dasLLAMA/dasllama/dasllama_whisper.das index a10c46308c..0b0ecfc214 100644 --- a/modules/dasLLAMA/dasllama/dasllama_whisper.das +++ b/modules/dasLLAMA/dasllama/dasllama_whisper.das @@ -392,7 +392,7 @@ def private wm_mm_rq(w : WhisperModel; var y : array; woff : int64; x : a // ===== ggml bin container parsing ===== -// One tensor's location in the bin: ggml ttype (0=f32, 1=f16), element count, absolute +// One tensor's location in the bin: ggml ttype (0=f32, 1=f16, 8=q8_0), element count, absolute // byte offset of the raw data (which follows the name with NO alignment padding). struct private WTensorRec { ttype : int @@ -411,7 +411,18 @@ def private bytes_str(b : array | #; o, n : int64) : string { return s } -// Read a tensor's raw data (f32 or f16) into dst[dst_off..) as fp32. Byte-wise reads — the +let private GGML_FTYPE_MOSTLY_Q8_0 = 7 //!< the file type whisper.cpp's quantizer stamps a q8_0 bin with +let private WT_Q8_BLOCK_ELEMS = 32l +let private WT_Q8_BLOCK_BYTES = 34l //!< a Q8_0 block on disk: an f16 scale and 32 quants + +//! a tensor's bytes in the bin by its ggml type +def private wt_bytes(tt : int; n_elem : int64) : int64 { + if (tt == GGML_TYPE_F32) return n_elem * 4l + if (tt == GGML_TYPE_F16) return n_elem * 2l + return (n_elem / WT_Q8_BLOCK_ELEMS) * WT_Q8_BLOCK_BYTES +} + +// Read a tensor's raw data (f32, f16 or Q8_0 blocks) into dst[dst_off..) as fp32. Byte-wise reads — the // bin stores tensor data unaligned. def private wt_read(recs : table; b : array | #; name : string; var dst : array; dst_off, n : int64) { @@ -426,6 +437,14 @@ def private wt_read(recs : table; b : array | #; name for (i in range64(n)) { dst[dst_off + i] = rd_f32(b, r.off + i * 4l) } + } elif (r.ttype == GGML_TYPE_Q8_0) { // a block: its f16 scale, then 32 quants + for (blk in range64(n / WT_Q8_BLOCK_ELEMS)) { + let at = r.off + blk * WT_Q8_BLOCK_BYTES + let scale = f16_to_f32(rd_u16(b, at)) + for (i in range64(WT_Q8_BLOCK_ELEMS)) { + dst[dst_off + blk * WT_Q8_BLOCK_ELEMS + i] = scale * float(int8(b[at + 2l + i])) + } + } } else { for (i in range64(n)) { dst[dst_off + i] = f16_to_f32(rd_u16(b, r.off + i * 2l)) @@ -802,8 +821,8 @@ def private whisper_index(bytes : array | #; n_bytes : int64; var st : Wh st.e.n_text_layer = int64(rd_i32(bytes, o)); o += 4l let n_mels = int64(rd_i32(bytes, o)); o += 4l let ftype = rd_i32(bytes, o) % 1000; o += 4l - if (ftype != GGML_TYPE_F32 && ftype != GGML_TYPE_F16) { - panic("dasLLAMA whisper: unsupported ftype {ftype} (only f32/f16 bins)") + if (ftype != GGML_TYPE_F32 && ftype != GGML_TYPE_F16 && ftype != GGML_FTYPE_MOSTLY_Q8_0) { + panic("dasLLAMA whisper: unsupported ftype {ftype} (f32, f16 and q8_0 bins load)") } // mel filterbank @@ -866,8 +885,8 @@ def private whisper_index(bytes : array | #; n_bytes : int64; var st : Wh let nl = int64(rd_i32(bytes, o + 4l)) let tt = rd_i32(bytes, o + 8l) o += 12l - if (tt != GGML_TYPE_F32 && tt != GGML_TYPE_F16) { - panic("dasLLAMA whisper: unsupported tensor type {tt} (only f32/f16 bins)") + if (tt != GGML_TYPE_F32 && tt != GGML_TYPE_F16 && tt != GGML_TYPE_Q8_0) { + panic("dasLLAMA whisper: unsupported tensor type {tt} (f32, f16 and q8_0 tensors load)") } var ne = 1l for (_i in range(nd)) { @@ -877,7 +896,7 @@ def private whisper_index(bytes : array | #; n_bytes : int64; var st : Wh let name = bytes_str(bytes, o, nl) o += nl recs[name] = WTensorRec(ttype = tt, n_elem = ne, off = o) - o += ne * (tt == GGML_TYPE_F32 ? 4l : 2l) + o += wt_bytes(tt, ne) } // encoder tower layout -> AudioTower scalars + one job per tensor, in blob-offset order diff --git a/modules/dasLLAMA/tests/CLAUDE.md b/modules/dasLLAMA/tests/CLAUDE.md index 81297e9499..511a6aeb61 100644 --- a/modules/dasLLAMA/tests/CLAUDE.md +++ b/modules/dasLLAMA/tests/CLAUDE.md @@ -1708,7 +1708,9 @@ token and bar, the batch with its last token changed landing outside the bar as CPU blocks over them transcribe the all-CPU chain's text), `test_whisper_vulkan_wdec_lifetime` (tiny, one session reused the way a serving worker reuses one: a model drop between two transcriptions - the second serves again and reads the same; the decoder knob turned off between two - the second reads as a fresh knob-off session; the block hooks pinned off after a served window - no -handoff for the CPU-encoded windows, the text of the CPU-encoder chain), `test_whisper_metal_ln_post` +handoff for the CPU-encoded windows, the text of the CPU-encoder chain), `test_whisper_q8_0_file` (a whisper.cpp q8_0 bin of tiny loads - its +Q8_0 tensors read as each block's scale times its quants - and reads jfk as the f16 bin does; skips +without `ggml-tiny-q8_0.bin`), `test_whisper_metal_ln_post` (tiny, the f32 encoder minted in memory: the blocks-with-post-norm seat declines with the tower off and serves with it on, one device encode counted, and the device's normed rows are the CPU post-norm of the device's own block rows within 1e-4 of the largest - reads 3e-7 - with the rows diff --git a/modules/dasLLAMA/tests/test_whisper.das b/modules/dasLLAMA/tests/test_whisper.das index 043acd7084..f6c23f8115 100644 --- a/modules/dasLLAMA/tests/test_whisper.das +++ b/modules/dasLLAMA/tests/test_whisper.das @@ -777,6 +777,38 @@ def test_whisper_vulkan_wdec_flush(t : T?) { } } +//! A whisper.cpp q8_0 bin loads: its Q8_0 tensors read as their blocks' scale times the quants, and it transcribes what the +//! f16 bin of the same model does. +[test] +def test_whisper_q8_0_file(t : T?) { + t |> run("ggml-tiny-q8_0.bin: a q8_0 bin loads and reads jfk as the f16 bin does") @(tt : T?) { + let f16_path = path_join(whisper_dir(), "ggml-tiny.bin") + let q8_path = path_join(whisper_dir(), "ggml-tiny-q8_0.bin") + return if (!jit_model_available(tt, f16_path) || !jit_model_available(tt, q8_path)) + let samples <- read_wav_pcm16_mono(path_join(models_dir(), "jfk.wav")) + if (empty(samples)) { + tt |> skip("jfk.wav not present") + return + } + set_asr_fp32(false) + set_asr_tower_fp32(false) + var inscope m16 <- mint_asr_whisper(f16_path) + var inscope m8 <- mint_asr_whisper(q8_path) + var from_f16 = "" + var from_q8 = "" + with_dasllama_jobque_() { + var inscope s16 <- create_session(m16) + from_f16 = transcribe(m16, s16, samples) + var inscope s8 <- create_session(m8) + from_q8 = transcribe(m8, s8, samples) + } + to_log(LOG_INFO, "whisper tiny f16 bin: '{from_f16}'\n") + to_log(LOG_INFO, "whisper tiny q8_0 bin: '{from_q8}'\n") + tt |> success(!empty(from_q8), "the q8_0 bin transcribes") + tt |> equal(from_q8, from_f16, "and reads what the f16 bin reads") + } +} + //! the largest element difference of two row images over the reference's largest magnitude; 2 where a length differs def private rows_rel_max(got, want : array) : float { return 2.0 if (length(got) != length(want) || empty(want)) From 172a3a8b5581329dcf40f4bc09bb7dd9df5b70fc Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Sat, 3 Oct 2026 23:46:19 -0700 Subject: [PATCH 09/46] dasLLAMA: the Metal conv stem serves the 80-mel whisper models - the first conv's 240 columns ride the GEMM lattice through a device copy of its rows zero-padded to 256 Co-Authored-By: Claude Fable 5.1 --- modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md | 4 +- .../dasllama/dasllama_metal_tower.das | 53 ++++++++++++++++--- modules/dasLLAMA/followup_metal.md | 3 +- modules/dasLLAMA/tests/test_model_image.das | 2 +- 4 files changed, 50 insertions(+), 12 deletions(-) diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md b/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md index a342fc31c9..861b9e3b2d 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md @@ -60,7 +60,9 @@ transform the ASR-decoder driver uploads its planes through), and every site run ladder (`pf_enc_q8_mm`) with the activations converted to one half panel where that ladder reads one. The device blob is 34/32 of the q8 planes and no image carries it; it drops with the weights epoch and the tower shutdown. The stem's first conv reads the f32 blob on both lanes - its -3 x n_mel columns are not quantized. +3 x n_mel columns are not quantized - and where those columns are off the GEMM's 32 lattice (80 +mels: 240) it reads a device copy of its rows zero-padded to the lattice (`tw_conv1_pad_attach`), +the im2col pass padding its own rows with zeros to the same width. The gemma4a chain ends in the projector tail, in the blocks' command buffer: the out projection and its bias at the encoder's own output width (`gemma4a_mid_dim`, the length of the weightless diff --git a/modules/dasLLAMA/dasllama/dasllama_metal_tower.das b/modules/dasLLAMA/dasllama/dasllama_metal_tower.das index 7a32cacdc9..d71055feba 100644 --- a/modules/dasLLAMA/dasllama/dasllama_metal_tower.das +++ b/modules/dasLLAMA/dasllama/dasllama_metal_tower.das @@ -109,6 +109,7 @@ def metal_tower_shutdown { st2_slab_free(g_tw_q3a_wpad) st2_slab_free(g_tw_fc_twin) st2_slab_free(g_tw_q8) + st2_slab_free(g_tw_conv1_pad) st2_slab_drop() pool_drain(g_tw_pool) g_tw_failed = false @@ -2142,6 +2143,32 @@ def private metal_qwen25v_blocks(t : Qwen25vTower; var s : Qwen25vState; npos : return true } +var private g_tw_conv1_pad : St2SlabBase //!< the served whisper-class tower's first conv with its columns zero-padded to the 32 lattice + +//! The first conv's weight rows at `k1` columns - the file's 3 x n_mel and zeros past them - on the device, once a tower: +//! the im2col pass pads its rows to the same lattice with zeros. Null where the buffer cannot be made. +[cold_path] +def private tw_conv1_pad_attach(t : AudioTower; k1 : int64) : MetalBuffer? { + let key = hash_combine64(tw_plane_key(t.fblob), uint64(t.conv1_w_off)) + return g_tw_conv1_pad.buf if (st2_slab_current(g_tw_conv1_pad, key)) + st2_slab_free(g_tw_conv1_pad) + let d = t.d_model + let k = 3l * t.n_mel + var pad = metal_new_buffer(g_dev, uint64(d * k1 * 4l)) + return null if (pad == null) + unsafe { + var p = reinterpret(metal_buffer_contents(pad)) + for (r in range64(d)) { + for (c in range64(k1)) { + p[r * k1 + c] = c < k ? t.fblob[t.conv1_w_off + r * k + c] : 0.0 + } + } + } + g_tw_conv1_pad.buf = pad + g_tw_conv1_pad.key = key + return pad +} + //! the stem's second conv as a GEMM over its im2col rows: the device q8 blob's region on a q8 tower, the f32 blob otherwise def private conv2_mm(enc : MetalComputeEncoder?; t : AudioTower; bblob, bx2, bx2h, bxo, u_3d, u_d : MetalBuffer?; mp2 : int64) { let d = t.d_model @@ -2159,7 +2186,7 @@ def private conv2_mm(enc : MetalComputeEncoder?; t : AudioTower; bblob, bx2, bx2 // one command buffer, s.x readback. BEST-EFFORT inside the tower knob: any decline is a // silent false (no counter noise — the CPU conv serves and the blocks still ride the GPU). [hot_path] -def private metal_conv_frontend(t : AudioTower; var s : EncoderState; mel : MelChunk) : bool { +def private metal_conv_frontend(t : AudioTower; var s : EncoderState; mel : MelChunk) : bool { // nolint:STYLE038 — the dispatch chain is the shape if (!g_tw_env_tower) { return false } @@ -2167,8 +2194,13 @@ def private metal_conv_frontend(t : AudioTower; var s : EncoderState; mel : MelC let tt = t.n_ctx let d = t.d_model let nm = t.n_mel - // tiny's 3x80 im2col width stays CPU; init re-arms after any shutdown seat - if ((3l * nm) % 32l != 0l || d % 64l != 0l || (3l * d) % 32l != 0l || !metal_tower_init() || (t.q8 && !tw_q8_attach(t))) { + // the first conv's 3 x n_mel columns ride the GEMM's 32 lattice: 80 mels' 240 pad to 256 through a padded weight copy + let k1 = round_up(3l * nm, 32l) + if (d % 64l != 0l || (3l * d) % 32l != 0l || !metal_tower_init() || (t.q8 && !tw_q8_attach(t))) { + return false + } + var bw1 = k1 == 3l * nm ? null : tw_conv1_pad_attach(t, k1) + if (k1 != 3l * nm && bw1 == null) { return false } let mp1 = round_up(tin, 32l) @@ -2176,7 +2208,7 @@ def private metal_conv_frontend(t : AudioTower; var s : EncoderState; mel : MelC let mapped = t.image_map != null var bblob = plane_buffer(g_dev, unsafe(reinterpret(plane_at(t.fblob, 0l))), uint64(length(t.fblob) * 4l), mapped) let bytes_mel = uint64(nm * tin * 4l) - let bytes_x1 = uint64(mp1 * 3l * nm * 4l) + let bytes_x1 = uint64(mp1 * k1 * 4l) let bytes_y1 = uint64(mp1 * d * 4l) let bytes_x2 = uint64(mp2 * 3l * d * 4l) let bytes_xo = uint64(mp2 * d * 4l) @@ -2189,17 +2221,21 @@ def private metal_conv_frontend(t : AudioTower; var s : EncoderState; mel : MelC unsafe { memcpy(metal_buffer_contents(bmel), addr(mel.data[0]), bytes_mel) } - var u_nm3 = uniform_u32(uint(3l * nm)) + var u_nm3 = uniform_u32(uint(k1)) var u_d = uniform_u32(uint(d)) var u_3d = uniform_u32(uint(3l * d)) var u_totd1 = uniform_u32(uint(tin * d)) var u_totd2 = uniform_u32(uint(tt * d)) - var k1 = St2ConvArgs(cin = uint(nm), k = 3u, stride = 1u, pad_l = 1u, dil = 1u, t_in = uint(tin), r0 = 0u, rows = uint(tin), xs = uint(nm)) + var ka1 = St2ConvArgs(cin = uint(nm), k = 3u, stride = 1u, pad_l = 1u, dil = 1u, t_in = uint(tin), r0 = 0u, rows = uint(tin), xs = uint(nm)) var k2 = St2ConvArgs(cin = uint(d), k = 3u, stride = 2u, pad_l = 1u, dil = 1u, t_in = uint(tin), r0 = 0u, rows = uint(tt), xs = uint(d)) var err = "" let ran = with_compute_encoder(g_queue, err) $(enc : MetalComputeEncoder?) { // nolint:PERF026 — the error-text fetch runs only on the failure leg - enc_st2_im2col_cm(enc, bmel, bx1, k1, mp1 * 3l * nm) - enc_f32_mm(enc, bblob, uint64(t.conv1_w_off * 4l), bx1, by1, 0ul, u_nm3, u_d, mp1, d) + enc_st2_im2col_cm(enc, bmel, bx1, ka1, mp1 * k1) + if (bw1 != null) { + enc_f32_mm(enc, bw1, 0ul, bx1, by1, 0ul, u_nm3, u_d, mp1, d) + } else { + enc_f32_mm(enc, bblob, uint64(t.conv1_w_off * 4l), bx1, by1, 0ul, u_nm3, u_d, mp1, d) + } enc_add_bias_rows(enc, by1, 0ul, bblob, uint64(t.conv1_b_off * 4l), u_d, u_totd1, tin * d) if (t.gelu_tanh) { enc_gelu_lut(enc, by1, 0ul, u_totd1, tin * d) @@ -4666,6 +4702,7 @@ def private tw_weights_drop { st2_slab_free(g_tw_q3a_wpad) st2_slab_free(g_tw_fc_twin) st2_slab_free(g_tw_q8) + st2_slab_free(g_tw_conv1_pad) st2_slab_drop() } diff --git a/modules/dasLLAMA/followup_metal.md b/modules/dasLLAMA/followup_metal.md index 9345f13067..de0a363438 100644 --- a/modules/dasLLAMA/followup_metal.md +++ b/modules/dasLLAMA/followup_metal.md @@ -892,8 +892,7 @@ from it. A Metal chain serves every stage of its family on the device (`REVIEW_TOWER.md`); these do not. Each is a stage and the models it holds for, read off the drivers' own gates: -- **The whisper mel** runs on the CPU on every whisper size, and the whisper-class conv stem on a - model whose 3 x n_mel im2col width is off the 32 lattice (tiny, base, small, medium: 80 mels). +- **The whisper mel** runs on the CPU on every whisper size. - **The whisper decode step** serves on the device from a text width of 512 up (`METAL_WDEC_STEP_MIN_D`); tiny's (384) runs on the CPU, where the device step is the slower one - 21.8 ms a request's decode against 17.8. diff --git a/modules/dasLLAMA/tests/test_model_image.das b/modules/dasLLAMA/tests/test_model_image.das index 58da3f8ba3..1b2bccb188 100644 --- a/modules/dasLLAMA/tests/test_model_image.das +++ b/modules/dasLLAMA/tests/test_model_image.das @@ -1303,7 +1303,7 @@ def test_whisper_metal_tower(tst : T?) { t |> equal(chunks, 1l, "jfk is one 30s chunk") t |> equal(s2.blocks - s1.blocks, 4l * chunks, "blocks delta = n_layer x chunks (tiny = 4)") t |> equal(s2.rows - s1.rows, 1500l * chunks, "rows delta = n_ctx x chunks") - t |> equal(s2.convs - s1.convs, 0l, "tiny's 3x80 im2col width keeps the CPU conv (the shape carve-out)") + t |> equal(s2.convs - s1.convs, chunks, "tiny's 3 x 80 conv columns pad to 256: the conv front served on the device") to_log(LOG_INFO, "whisper tiny mtower CPU eyeball: '{cpu.text}'\n") to_log(LOG_INFO, "whisper tiny mtower GPU eyeball: '{gpu.text}'\n") t |> success(!empty(gpu.text), "non-empty transcript") From 4c7bb5cc8a2e4b4237b436afde0350e06d8a67ba Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Sun, 4 Oct 2026 00:04:57 -0700 Subject: [PATCH 10/46] dasLLAMA: a parakeet q8_0 bin loads - the whisper and parakeet bin readers are one reader that takes f32, f16 and Q8_0 tensors Co-Authored-By: Claude Fable 5.1 --- .../dasLLAMA/dasllama/dasllama_convert.das | 35 +++++++++++++++++++ .../dasLLAMA/dasllama/dasllama_parakeet.das | 14 ++------ .../dasLLAMA/dasllama/dasllama_whisper.das | 29 ++------------- modules/dasLLAMA/followup_metal.md | 4 ++- modules/dasLLAMA/tests/CLAUDE.md | 3 +- modules/dasLLAMA/tests/test_whisper.das | 34 ++++++++++++++++++ 6 files changed, 79 insertions(+), 40 deletions(-) diff --git a/modules/dasLLAMA/dasllama/dasllama_convert.das b/modules/dasLLAMA/dasllama/dasllama_convert.das index 5cb790fc0e..6964af8cc3 100644 --- a/modules/dasLLAMA/dasllama/dasllama_convert.das +++ b/modules/dasLLAMA/dasllama/dasllama_convert.das @@ -35,6 +35,41 @@ def rd_f32(b : array | #; o : int64) : float { return unsafe(reinterpret(rd_u32(b, o))) } +let GGML_BIN_Q8_BLOCK_ELEMS = 32l //!< a Q8_0 block of a ggml bin: an f16 scale, then this many quants +let GGML_BIN_Q8_BLOCK_BYTES = 34l + +//! the bytes a tensor of ggml type `ttype` takes in a whisper-style bin: f32, f16, or Q8_0 blocks +def ggml_bin_tensor_bytes(ttype : int; n_elem : int64) : int64 { + if (ttype == GGML_TYPE_F32) return n_elem * 4l + if (ttype == GGML_TYPE_F16) return n_elem * 2l + if (ttype == GGML_TYPE_Q8_0) return (n_elem / GGML_BIN_Q8_BLOCK_ELEMS) * GGML_BIN_Q8_BLOCK_BYTES + panic("dasLLAMA: ggml bin tensor type {ttype} - f32, f16 and q8_0 tensors load") + return 0l +} + +//! `n` elements of a ggml bin tensor at byte `off` into `dst[dst_off..)` as f32: byte-wise, the bin stores its data +//! unaligned; a Q8_0 element is its block's scale times its quant +def ggml_bin_read_f32(b : array | #; ttype : int; off, n : int64; var dst : array; dst_off : int64) { + if (ttype == GGML_TYPE_F32) { + for (i in range64(n)) { + dst[dst_off + i] = rd_f32(b, off + i * 4l) + } + } elif (ttype == GGML_TYPE_F16) { + for (i in range64(n)) { + dst[dst_off + i] = f16_to_f32(rd_u16(b, off + i * 2l)) + } + } else { + for (blk in range64(ggml_bin_tensor_bytes(ttype, n) / GGML_BIN_Q8_BLOCK_BYTES)) { + let at = off + blk * GGML_BIN_Q8_BLOCK_BYTES + let scale = f16_to_f32(rd_u16(b, at)) + for (i in range64(GGML_BIN_Q8_BLOCK_ELEMS)) { + dst[dst_off + blk * GGML_BIN_Q8_BLOCK_ELEMS + i] = scale * float(int8(b[at + 2l + i])) + } + } + } +} + + def rd_i32(b : array | #; o : int64) : int { return int(rd_u32(b, o)) } diff --git a/modules/dasLLAMA/dasllama/dasllama_parakeet.das b/modules/dasLLAMA/dasllama/dasllama_parakeet.das index 050d83ecd2..a5b899aedf 100644 --- a/modules/dasLLAMA/dasllama/dasllama_parakeet.das +++ b/modules/dasLLAMA/dasllama/dasllama_parakeet.das @@ -425,7 +425,7 @@ def private pk_fold_bn(var blob : array; w, b, mean, vr, d : int64) { // ===== bin container ===== // whisper-bin-style tensor records: i32 n_dims, i32 name_len, i32 ftype; i32 dims (ne -// order); name bytes; raw data (f32/f16), unaligned. +// order); name bytes; raw data (f32, f16 or Q8_0 blocks), unaligned. struct private PkTensorRec { ttype : int n_elem : int64 @@ -452,15 +452,7 @@ def private pk_read(recs : table; b : array | #; nam if (r.n_elem != n) { panic("dasLLAMA parakeet: tensor '{name}' has {r.n_elem} elements, want {n}") } - if (r.ttype == GGML_TYPE_F32) { - for (i in range64(n)) { - dst[dst_off + i] = rd_f32(b, r.off + i * 4l) - } - } else { - for (i in range64(n)) { - dst[dst_off + i] = f16_to_f32(rd_u16(b, r.off + i * 2l)) - } - } + ggml_bin_read_f32(b, r.ttype, r.off, n, dst, dst_off) } // GEMM weight read: a q8 load transcodes it from the file straight into qblob/qscales at its @@ -648,7 +640,7 @@ def stage_parakeet_model(path : string; q8 : bool = false) : ParakeetStaging { let name = pk_bytes_str(bytes, o, name_len) o += name_len recs[name] = PkTensorRec(ttype = ttype, n_elem = n_elem, off = o) - o += n_elem * (ttype == GGML_TYPE_F32 ? 4l : 2l) + o += ggml_bin_tensor_bytes(ttype, n_elem) } // blob layout — dual walk: `bo` is the full layout (qblob on a q8 load, blob on an diff --git a/modules/dasLLAMA/dasllama/dasllama_whisper.das b/modules/dasLLAMA/dasllama/dasllama_whisper.das index 0b0ecfc214..80a333e5e7 100644 --- a/modules/dasLLAMA/dasllama/dasllama_whisper.das +++ b/modules/dasLLAMA/dasllama/dasllama_whisper.das @@ -412,15 +412,6 @@ def private bytes_str(b : array | #; o, n : int64) : string { } let private GGML_FTYPE_MOSTLY_Q8_0 = 7 //!< the file type whisper.cpp's quantizer stamps a q8_0 bin with -let private WT_Q8_BLOCK_ELEMS = 32l -let private WT_Q8_BLOCK_BYTES = 34l //!< a Q8_0 block on disk: an f16 scale and 32 quants - -//! a tensor's bytes in the bin by its ggml type -def private wt_bytes(tt : int; n_elem : int64) : int64 { - if (tt == GGML_TYPE_F32) return n_elem * 4l - if (tt == GGML_TYPE_F16) return n_elem * 2l - return (n_elem / WT_Q8_BLOCK_ELEMS) * WT_Q8_BLOCK_BYTES -} // Read a tensor's raw data (f32, f16 or Q8_0 blocks) into dst[dst_off..) as fp32. Byte-wise reads — the // bin stores tensor data unaligned. @@ -433,23 +424,7 @@ def private wt_read(recs : table; b : array | #; name if (r.n_elem != n) { panic("dasLLAMA whisper: tensor '{name}' has {r.n_elem} elements, want {n}") } - if (r.ttype == GGML_TYPE_F32) { - for (i in range64(n)) { - dst[dst_off + i] = rd_f32(b, r.off + i * 4l) - } - } elif (r.ttype == GGML_TYPE_Q8_0) { // a block: its f16 scale, then 32 quants - for (blk in range64(n / WT_Q8_BLOCK_ELEMS)) { - let at = r.off + blk * WT_Q8_BLOCK_BYTES - let scale = f16_to_f32(rd_u16(b, at)) - for (i in range64(WT_Q8_BLOCK_ELEMS)) { - dst[dst_off + blk * WT_Q8_BLOCK_ELEMS + i] = scale * float(int8(b[at + 2l + i])) - } - } - } else { - for (i in range64(n)) { - dst[dst_off + i] = f16_to_f32(rd_u16(b, r.off + i * 2l)) - } - } + ggml_bin_read_f32(b, r.ttype, r.off, n, dst, dst_off) } // Conv1d weight (flat [c_out][c_in][k=3], k contiguous — torch layout) permuted to the @@ -896,7 +871,7 @@ def private whisper_index(bytes : array | #; n_bytes : int64; var st : Wh let name = bytes_str(bytes, o, nl) o += nl recs[name] = WTensorRec(ttype = tt, n_elem = ne, off = o) - o += wt_bytes(tt, ne) + o += ggml_bin_tensor_bytes(tt, ne) } // encoder tower layout -> AudioTower scalars + one job per tensor, in blob-offset order diff --git a/modules/dasLLAMA/followup_metal.md b/modules/dasLLAMA/followup_metal.md index de0a363438..1119a08de9 100644 --- a/modules/dasLLAMA/followup_metal.md +++ b/modules/dasLLAMA/followup_metal.md @@ -896,7 +896,9 @@ Each is a stage and the models it holds for, read off the drivers' own gates: - **The whisper decode step** serves on the device from a text width of 512 up (`METAL_WDEC_STEP_MIN_D`); tiny's (384) runs on the CPU, where the device step is the slower one - 21.8 ms a request's decode against 17.8. -- **The parakeet decode step** (the TDT predictor and joint) is a CPU team publish. +- **The parakeet decode** (the TDT predictor and joint, the pick) and **its mel** run on the CPU: of a + 64 s clip's 78 ms, the decode is 27.0 (260 steps at 78 us) and the mel 7.1, beside 44.9 of encode + on the device. - **The q8 encoder lane** declines `quant_mode` on gemma4a, canary and parakeet, so those families serve their f32 lane on the device at the larger image; the whisper-class tower reads q8. - **Qwen3-ASR, Qwen3-Omni and Canary-Qwen** have had no stage-by-stage check. diff --git a/modules/dasLLAMA/tests/CLAUDE.md b/modules/dasLLAMA/tests/CLAUDE.md index 511a6aeb61..94fbc9a4ca 100644 --- a/modules/dasLLAMA/tests/CLAUDE.md +++ b/modules/dasLLAMA/tests/CLAUDE.md @@ -1708,7 +1708,8 @@ token and bar, the batch with its last token changed landing outside the bar as CPU blocks over them transcribe the all-CPU chain's text), `test_whisper_vulkan_wdec_lifetime` (tiny, one session reused the way a serving worker reuses one: a model drop between two transcriptions - the second serves again and reads the same; the decoder knob turned off between two - the second reads as a fresh knob-off session; the block hooks pinned off after a served window - no -handoff for the CPU-encoded windows, the text of the CPU-encoder chain), `test_whisper_q8_0_file` (a whisper.cpp q8_0 bin of tiny loads - its +handoff for the CPU-encoded windows, the text of the CPU-encoder chain), `test_parakeet_q8_0_file` (the parakeet v3 q8_0 bin loads through the same +reader and reads jfk as the f32 bin does; skips without it), `test_whisper_q8_0_file` (a whisper.cpp q8_0 bin of tiny loads - its Q8_0 tensors read as each block's scale times its quants - and reads jfk as the f16 bin does; skips without `ggml-tiny-q8_0.bin`), `test_whisper_metal_ln_post` (tiny, the f32 encoder minted in memory: the blocks-with-post-norm seat declines with the tower diff --git a/modules/dasLLAMA/tests/test_whisper.das b/modules/dasLLAMA/tests/test_whisper.das index f6c23f8115..363a83e51e 100644 --- a/modules/dasLLAMA/tests/test_whisper.das +++ b/modules/dasLLAMA/tests/test_whisper.das @@ -809,6 +809,40 @@ def test_whisper_q8_0_file(t : T?) { } } +//! one parakeet bin staged and minted in memory on the lane `q8`, reading `samples` +def private parakeet_text(path : string; q8 : bool; samples : array) : string { + var inscope m = AsrModel(kind = AsrKind.parakeet) + var inscope stg <- stage_parakeet_model(path, q8) + mint_parakeet_model(stg, m.pk, q8) + var text = "" + with_dasllama_jobque_() { + var inscope s <- create_session(m, "en") + text = transcribe(m, s, samples) + } + return text +} + +//! A parakeet q8_0 bin loads through the reader the whisper bins share, and reads what the f32 bin of the same model does. +[test] +def test_parakeet_q8_0_file(t : T?) { + t |> run("ggml-parakeet-tdt-0.6b-v3-q8_0.bin: a q8_0 bin loads and reads jfk as the f32 bin does") @(tt : T?) { + let f32_path = path_join(whisper_dir(), "ggml-parakeet-tdt-0.6b-v3-f32.bin") + let q8_path = path_join(whisper_dir(), "ggml-parakeet-tdt-0.6b-v3-q8_0.bin") + return if (!jit_model_available(tt, f32_path) || !jit_model_available(tt, q8_path)) + let samples <- read_wav_pcm16_mono(path_join(models_dir(), "jfk.wav")) + if (empty(samples)) { + tt |> skip("jfk.wav not present") + return + } + let from_f32 = parakeet_text(f32_path, false, samples) + let from_q8 = parakeet_text(q8_path, false, samples) + to_log(LOG_INFO, "parakeet v3 f32 bin: '{from_f32}'\n") + to_log(LOG_INFO, "parakeet v3 q8_0 bin: '{from_q8}'\n") + tt |> success(!empty(from_q8), "the q8_0 bin transcribes") + tt |> equal(from_q8, from_f32, "and reads what the f32 bin reads") + } +} + //! the largest element difference of two row images over the reference's largest magnitude; 2 where a length differs def private rows_rel_max(got, want : array) : float { return 2.0 if (length(got) != length(want) || empty(want)) From 5cd84a05ff5024e64fb26f746dd0fcdd61968c73 Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Sun, 4 Oct 2026 00:33:56 -0700 Subject: [PATCH 11/46] dasLLAMA: the canary encode clocks its stages - the mel, the front, the blocks and the projection - and followup_metal 39 names the ones on the CPU Co-Authored-By: Claude Fable 5.1 --- modules/dasLLAMA/dasllama/dasllama_canary.das | 11 +++++++++++ modules/dasLLAMA/followup_metal.md | 8 +++++++- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/modules/dasLLAMA/dasllama/dasllama_canary.das b/modules/dasLLAMA/dasllama/dasllama_canary.das index ef57e86829..eb2b84ca4d 100644 --- a/modules/dasLLAMA/dasllama/dasllama_canary.das +++ b/modules/dasLLAMA/dasllama/dasllama_canary.das @@ -1160,18 +1160,26 @@ def canary_encode(m : CanaryEncoder; var st : CanaryState) : int64 { dispatch_phase_encode(conv_s2_out(conv_s2_out(conv_s2_out(st.n_len)))) var tt = -1l + var ts = ref_time_ticks() if (g_cn_front_set) { tt = invoke(g_cn_front, m, st) + if (tt >= 0l) { + asr_prof_add("cn.front.gpu", ts) + } } if (tt < 0l) { if (!st.mel_ready) { canary_mel_spectrum(m, st) + asr_prof_add("cn.mel", ts) + ts = ref_time_ticks() } tt = cn_front_cpu(m, st) + asr_prof_add("cn.front", ts) } let ww = 2l * tt - 1l var gpu_served = false + ts = ref_time_ticks() if (g_cn_gpu_set) { gpu_served = invoke(g_cn_gpu, m, st, tt, ww) } @@ -1179,11 +1187,14 @@ def canary_encode(m : CanaryEncoder; var st : CanaryState) : int64 { canary_pos_table(m, st, tt, ww) cn_blocks_cpu(m, st, tt, ww) } + asr_prof_add(gpu_served ? "cn.blocks.gpu" : "cn.blocks", ts) // perception.proj: encoder rows (d) -> decoder embedding (proj_dim) + ts = ref_time_ticks() ensure_length(st.out, tt * m.proj_dim) cn_mm_rq(m, st.out, m.proj_w, st.x, st.xqi, st.xsi, d, m.proj_dim, tt) add_bias_rows(st.out, m.blob, m.proj_b, m.proj_dim, tt) + asr_prof_add("cn.proj", ts) return tt } diff --git a/modules/dasLLAMA/followup_metal.md b/modules/dasLLAMA/followup_metal.md index 1119a08de9..d6c5d3fff9 100644 --- a/modules/dasLLAMA/followup_metal.md +++ b/modules/dasLLAMA/followup_metal.md @@ -901,7 +901,13 @@ Each is a stage and the models it holds for, read off the drivers' own gates: on the device. - **The q8 encoder lane** declines `quant_mode` on gemma4a, canary and parakeet, so those families serve their f32 lane on the device at the larger image; the whisper-class tower reads q8. -- **Qwen3-ASR, Qwen3-Omni and Canary-Qwen** have had no stage-by-stage check. +- **The canary mel and subsample front** run on the CPU (the front seat is filled by the Vulkan + driver alone): of a 64 s clip's 1274 ms they are 239 and 258, beside 80 of blocks on the device, + and the perception projection (7) is on the CPU too. Its stage clock is `cn.mel`, `cn.front`, + `cn.blocks.gpu`, `cn.proj`. +- **The Qwen3-ASR projector** (`q3a.proj`, 1.6 ms a chunk) runs on the CPU, and its mel's windowing + and clamp around the device spectrum. +- **Qwen3-Omni** has had no stage-by-stage check. The work: each stage on the device, and a counter per stage a gate can read, so a served model's row names the stages it ran and where. From 42c8e6f2e784d4bcca65d5c9c79d7b6cab2d0218 Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Sun, 4 Oct 2026 00:53:58 -0700 Subject: [PATCH 12/46] dasLLAMA: canary's mel and subsample front run on the Metal tower - the front seat only the Vulkan driver filled On Metal the mel (a twiddle GEMM on the CPU) and the subsample stack ran on the CPU beside the device's blocks: 497 of a 64 s clip's 1274 ms. The front runs whole in one command buffer off the windowed frames - the DFT, the power spectrum, the mel sums, MetalCnMelNorm (the log and the per-feature normalization), the parakeet front's convs over a tap-major copy of canary's taps, the input projection. The clip reads in 772 ms, the two stages in 8. Co-Authored-By: Claude Fable 5.1 --- modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md | 8 + modules/dasLLAMA/PERF_LEDGER.md | 12 ++ .../dasllama/dasllama_metal_prefill.das | 52 +++++++ .../dasllama/dasllama_metal_tower.das | 141 +++++++++++++++++- modules/dasLLAMA/followup_metal.md | 7 +- modules/dasLLAMA/tests/CLAUDE.md | 10 +- .../tests/test_metal_prefill_kernels.das | 44 ++++++ modules/dasLLAMA/tests/test_model_image.das | 12 +- 8 files changed, 274 insertions(+), 12 deletions(-) diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md b/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md index 861b9e3b2d..ceb40a2b44 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md @@ -71,6 +71,14 @@ embedder is square on E2B (1536 by 1536) and widens on E4B (1536 to 2560), so th read apart off the file - the out projection's rows, the embedder's rows - and never assumed equal. The chain lands the soft tokens in `Gemma4aState.out` and says so (`out_ready`). +Canary's front runs whole on the device ahead of its block seat (`metal_canary_front`, the front +seat): off the CPU-windowed frames, the DFT as a GEMM over the transposed twiddles, the power +spectrum, the mel sums, `MetalCnMelNorm` - the log and the per-feature normalization over the valid +frames, one thread a feature - then the parakeet front's convs and the input projection. The front +kernels read tap-major taps and canary's file keeps them channel-major, so the three conv panels +ride a device copy (`tw_cn_front_attach`); and a stage's valid rows are its conv's image height, so +a row past them reads as the zero the CPU chain masks it to and no mask pass is dispatched. + The FastConformer chain (canary and parakeet share it: one context, one block body over the canary offsets record, parakeet's offsets mapped onto it with no GEMM biases and its tap-major depthwise stamp) runs the rel-pos (Transformer-XL) attention one head at a time on the f32 GEMM diff --git a/modules/dasLLAMA/PERF_LEDGER.md b/modules/dasLLAMA/PERF_LEDGER.md index 00eb170925..b7434a281a 100644 --- a/modules/dasLLAMA/PERF_LEDGER.md +++ b/modules/dasLLAMA/PERF_LEDGER.md @@ -11,6 +11,18 @@ what it costs today and what the fix would change. ## Entries +- **MEASURED (2026-10-04, `direction-grade`) - canary's mel and subsample front run on the Metal tower.** + M5 Max, Metal, canary-qwen-2.5b (the q8_0 decoder, the f32 encoder), in process as the server's + ASR worker runs it (the engine's stage clock, the fourth transcription of a clip), the tune sidecar + predating the binary. The front seat was filled by the Vulkan driver alone, so on Metal the mel + (a twiddle GEMM on the CPU) and the subsample stack ran on the CPU beside 80 ms of blocks on the + device. + - A 64 s clip: 1274 ms before - mel 239, front 258, blocks 80, projection 7, prefill 56, decode + 629 - and 772 after, the mel and the front together 8.0. An 8 s clip: 163 ms before, 106 after + (3.5 for the two). + - The device rows sit 2.1e-3 rel-l2 from the CPU chain's on the jfk clip (3.1e-4 with the CPU + front ahead of the device blocks); the transcripts are equal. + - **MEASURED (2026-10-03, `direction-grade`) - the Metal tower reads a whisper encoder's q8 planes.** M5 Max, Metal, whisper large-v3-turbo, `dasllama-server` with the ASR model alone (`main.das -- --asr ggml-large-v3-turbo.bin`), the reference whisper.cpp `6fc7c33` as diff --git a/modules/dasLLAMA/dasllama/dasllama_metal_prefill.das b/modules/dasLLAMA/dasllama/dasllama_metal_prefill.das index 5fadc5140e..9d66a693cf 100644 --- a/modules/dasLLAMA/dasllama/dasllama_metal_prefill.das +++ b/modules/dasLLAMA/dasllama/dasllama_metal_prefill.das @@ -4271,6 +4271,52 @@ class MetalG4aLogMel { } } +struct CnMelNormArgs { + nmel : uint + nlen : uint //!< frames the mel holds + valid : uint //!< frames of real audio: the statistics' span, the rows past it zeroed + mfloor : float +} + +//! Canary's log and per-feature normalization over the mel GEMM's sums [rows x nmel], in place, one thread a mel +//! feature: log(sum + floor), the mean and the Bessel deviation over the valid frames, every valid frame normalized +//! and the frames past them zero - the CPU form's three passes. +[metal_dispatch(name = "enc_cn_melnorm", pso = "g_pf_pso_cn_melnorm", tg = 64, grid = "nmel/64", params = "nmel : int64")] +class MetalCnMelNorm { + @ssbo @binding = 0 @role = "readwrite" mel : array //!< [rows x nmel] + @uniform @binding = 1 ka : CnMelNormArgs + + [metal_kernel(name="metal_cn_melnorm_msl")] + def metal_cn_melnorm { + let j = gl_GlobalInvocationID.x + if (j >= ka.nmel) { + return + } + var sum = 0.0 + var i = 0u + while (i < ka.valid) { + let v = log(mel[i * ka.nmel + j] + ka.mfloor) + mel[i * ka.nmel + j] = v + sum += v + i++ + } + let mean = sum / float(max(ka.valid, 1u)) + var sq = 0.0 + i = 0u + while (i < ka.valid) { + let dv = mel[i * ka.nmel + j] - mean + sq += dv * dv + i++ + } + let denom = sqrt(sq / max(float(ka.valid) - 1.0, 1.0)) + 1e-5 + i = 0u + while (i < ka.nlen) { + mel[i * ka.nmel + j] = i < ka.valid ? (mel[i * ka.nmel + j] - mean) / denom : 0.0 + i++ + } + } +} + //! The subsample conv's LayerNorm(channels, weight-only) + ReLU: one thread per row — //! mean/variance over ``dim`` at ``ldin`` stride, compact ``dim``-wide rows out at ``ldout``. [metal_dispatch(name = "enc_g4a_lnrelu", pso = "g_pf_pso_g4a_lnrelu", tg = 64, grid = "nrows/64", params = "nrows : int64")] @@ -4477,6 +4523,7 @@ var private g_pf_pso_pk_dw : MetalComputePipeline? //!< the same on parake var private g_pf_pso_pk_conv0 : MetalComputePipeline? //!< the parakeet front: the first conv with its ReLU var private g_pf_pso_pk_dw2d : MetalComputePipeline? //!< its depthwise stride-2 conv var private g_pf_pso_pk_xf : MetalComputePipeline? //!< its feature rows for the pre-projection +var private g_pf_pso_cn_melnorm : MetalComputePipeline? //!< canary's log and per-feature normalization over the mel var private g_pf_pso_bias_relu : MetalComputePipeline? //!< a 1x1 conv's bias and ReLU over its GEMM's rows var private g_pf_pso_g4a_attn : MetalComputePipeline? //!< the gemma4a sliding-window RPE attention var private g_pf_pso_g4a_glu : MetalComputePipeline? //!< its conv-module GLU halves @@ -4906,6 +4953,7 @@ def public metal_prefill_shutdown { release_handles(g_pf_pso_twr_flash, g_pf_pso_twr_kvhc, g_pf_pso_twr_win) release_handles(g_pf_pso_fc_pack, g_pf_pso_fc_softmax, g_pf_pso_fc_unpack, g_pf_pso_cn_dw, g_pf_pso_pk_dw) release_handles(g_pf_pso_pk_conv0, g_pf_pso_pk_dw2d, g_pf_pso_pk_xf, g_pf_pso_bias_relu) + release_handle(g_pf_pso_cn_melnorm) release_handles(g_pf_pso_g4a_attn, g_pf_pso_g4a_glu, g_pf_pso_g4a_dw, g_pf_pso_g4a_silu, g_pf_pso_g4a_axpy) release_handles(g_pf_pso_g4a_mag, g_pf_pso_g4a_logmel, g_pf_pso_g4a_lnrelu) release_handles(g_pf_pso_q3a_shuffle, g_pf_pso_q3a_finish, g_pf_pso_q3a_pow, g_pf_pso_q3a_logmel_t) @@ -5088,6 +5136,7 @@ def private metal_prefill_init : bool { // nolint:STYLE038 — flat one-compil g_pf_pso_pk_conv0 = compile_stamp(metal_pk_conv0_msl, ok) g_pf_pso_pk_dw2d = compile_stamp(metal_pk_dw2d_msl, ok) g_pf_pso_pk_xf = compile_stamp(metal_pk_xf_msl, ok) + g_pf_pso_cn_melnorm = compile_stamp(metal_cn_melnorm_msl, ok) g_pf_pso_bias_relu = compile_stamp(metal_bias_relu_rows_msl, ok) g_pf_pso_g4a_attn = compile_stamp(metal_g4a_attn_msl, ok) g_pf_pso_g4a_glu = compile_stamp(metal_g4a_glu_msl, ok) @@ -5387,6 +5436,9 @@ def pf_fc_ready(tap_major : bool) : bool => psos_ready(fixed_array(g_pf_pso_fc_p //! The parakeet subsample front's kernels compiled - the whole-encode driver's extra device gate. def pf_pk_front_ready() : bool => psos_ready(fixed_array(g_pf_pso_pk_conv0, g_pf_pso_pk_dw2d, g_pf_pso_pk_xf, g_pf_pso_bias_relu, g_pf_pso_addbias)) +//! The canary front's kernels compiled - the parakeet front's, the spectrum's and the mel normalization. +def pf_cn_front_ready() : bool => pf_pk_front_ready() && psos_ready(fixed_array(g_pf_pso_cn_melnorm, g_pf_pso_q3a_pow)) + //! The gemma4a Conformer kernel set compiled — its tower driver's device gate. def pf_g4a_ready() : bool => psos_ready(fixed_array(g_pf_pso_g4a_attn, g_pf_pso_g4a_glu, g_pf_pso_g4a_dw, g_pf_pso_g4a_silu, g_pf_pso_g4a_axpy)) diff --git a/modules/dasLLAMA/dasllama/dasllama_metal_tower.das b/modules/dasLLAMA/dasllama/dasllama_metal_tower.das index d71055feba..da9315a71c 100644 --- a/modules/dasLLAMA/dasllama/dasllama_metal_tower.das +++ b/modules/dasLLAMA/dasllama/dasllama_metal_tower.das @@ -110,6 +110,7 @@ def metal_tower_shutdown { st2_slab_free(g_tw_fc_twin) st2_slab_free(g_tw_q8) st2_slab_free(g_tw_conv1_pad) + st2_slab_free(g_tw_cn_front) st2_slab_drop() pool_drain(g_tw_pool) g_tw_failed = false @@ -1039,7 +1040,7 @@ def private fc_run(blob : PlaneF; mapped : bool; d, n_ff, n_head, nk, n_layer : } } -//! The canary FastConformer block loop as ONE command buffer on the f32 blob; the subsample front, mel and final proj stay CPU. +//! The canary FastConformer block loop as ONE command buffer on the f32 blob; the front seat runs the mel and the subsample front ahead of it, the final proj stays CPU. [hot_path] def private metal_canary_blocks(t : CanaryEncoder; var st : CanaryState; npos, ww : int64) : bool { if (!g_tw_env_tower) { @@ -1055,6 +1056,142 @@ def private metal_canary_blocks(t : CanaryEncoder; var st : CanaryState; npos, w } } +var private @scratch g_tw_cn_bufs : array //!< the canary front's pooled buffers of one encode, taken and given back inside it +var private g_tw_cn_front : St2SlabBase //!< the served canary encoder's three front convs with their taps tap-major: the front kernels' layout + +//! Canary's first conv and its two depthwise convs on the device as [9][ch] panels, one after another - the file keeps +//! them [ch][9] - once an encoder. Null where the buffer cannot be made. +[cold_path] +def private tw_cn_front_attach(t : CanaryEncoder) : MetalBuffer? { + let key = hash_combine64(tw_plane_key(t.blob), uint64(t.c0_w)) + return g_tw_cn_front.buf if (st2_slab_current(g_tw_cn_front, key)) + st2_slab_free(g_tw_cn_front) + let nch = t.n_sub_ch + var taps = metal_new_buffer(g_dev, uint64(3l * 9l * nch * 4l)) + return null if (taps == null) + unsafe { + var p = reinterpret(metal_buffer_contents(taps)) + for (panel, woff in range64(3l), [t.c0_w, t.c2_w, t.c5_w]) { + for (c in range64(nch)) { + for (tap in range64(9l)) { + p[(panel * 9l + tap) * nch + c] = t.blob[woff + c * 9l + tap] + } + } + } + } + g_tw_cn_front.buf = taps + g_tw_cn_front.key = key + return taps +} + +//! The canary front whole, off the windowed frames `st.xw`, as ONE command buffer: the mel, the subsample stack and the +//! input projection into `st.x`, the CPU front's stages in its order. A stage's valid rows are its conv's image height: +//! a row past them reads as the zero the CPU masks it to. Answers the encoder rows, or -1 where the CPU front runs. +[hot_path, arch(at="../ARCHITECTURE_GPU_TOWER.md#tower-encode-chains")] +def private metal_canary_front(t : CanaryEncoder; var st : CanaryState) : int64 { // nolint:STYLE038 — the dispatch chain is the shape + if (!g_tw_env_tower || t.q8) { + return -1l + } + let d = t.d_model + let nch = t.n_sub_ch + let n_mel = t.n_mel + let n_fft = t.n_fft + let n_bins = t.n_fb + let n_len = st.n_len + let t1 = conv_s2_out(n_len) + let w1 = conv_s2_out(n_mel) + let t2 = conv_s2_out(t1) + let w2 = conv_s2_out(w1) + let tt = conv_s2_out(t2) + let w3 = conv_s2_out(w2) + let l0 = conv_s2_out(st.n_valid) //! the valid rows of each stage: the CPU front's length mask + let l2 = conv_s2_out(l0) + let feat = w3 * nch + let bytes_y0 = uint64(t1 * w1 * nch * 4l) + if (n_len < 1l || nch % 64l != 0l || feat % 32l != 0l || n_mel % 64l != 0l || n_fft % 32l != 0l || d % 64l != 0l + || bytes_y0 > uint64(PK_FRONT_MAX_BYTES) || !metal_tower_init() || !pf_cn_front_ready()) { + return -1l + } + var btaps = tw_cn_front_attach(t) + return -1l if (btaps == null) + residency_flush() + let mp = round_up(n_len, 32l) + let rows2 = t2 * w2 + let rows3 = tt * w3 + let mp2 = round_up(rows2, 32l) + let mp3 = round_up(rows3, 32l) + let mpt = round_up(tt, 32l) + let dft_out_pad = round_up(2l * n_bins, 64l) + let ldpow = round_up(n_bins, 32l) + var bblob = plane_buffer(g_dev, unsafe(reinterpret(plane_at(t.blob, 0l))), uint64(length(t.blob) * 4l), t.image_map != null) + let sizes = fixed_array(uint64(mp * n_fft * 4l), uint64(dft_out_pad * n_fft * 4l), uint64(mp * dft_out_pad * 4l), + uint64(mp * ldpow * 4l), uint64(n_mel * ldpow * 4l), uint64(mp * n_mel * 4l), bytes_y0, + uint64(mp2 * nch * 4l), uint64(mp2 * nch * 4l), uint64(mp3 * nch * 4l), uint64(mp3 * nch * 4l), + uint64(mpt * feat * 4l), uint64(mpt * d * 4l)) + g_tw_cn_bufs |> resize(length(sizes)) + assume bufs = g_tw_cn_bufs + for (b, n in bufs, sizes) { + b = pool_acquire_pinned(g_tw_pool, g_dev, n) + } + upload_rows_strided(bufs[0], unsafe(addr(st.xw[0])), n_len, n_fft, n_fft) + zero_rows(bufs[0], n_len, mp, n_fft) + tw_twiddle_panel(bufs[1], plane_at(t.twiddles, 0l), n_bins, n_fft, n_fft, dft_out_pad) + upload_rows_strided(bufs[4], plane_at(t.filters, 0l), n_mel, n_bins, ldpow) + unsafe { //! the depthwise convs write their live rows alone, and the GEMM behind each reads the row pad: zero it, the buffers are pooled + memset8(reinterpret(metal_buffer_contents(bufs[7])) + rows2 * nch * 4l, uint8(0), int((mp2 - rows2) * nch * 4l)) + memset8(reinterpret(metal_buffer_contents(bufs[9])) + rows3 * nch * 4l, uint8(0), int((mp3 - rows3) * nch * 4l)) + } + var u_kfft = uniform_u32(uint(n_fft)) + var u_dft = uniform_u32(uint(dft_out_pad)) + var u_nbins = uniform_u32(uint(n_bins)) + var u_ldpow = uniform_u32(uint(ldpow)) + var u_totpow = uniform_u32(uint(mp * ldpow)) + var u_nmel = uniform_u32(uint(n_mel)) + var u_nch = uniform_u32(uint(nch)) + var u_feat = uniform_u32(uint(feat)) + var u_d = uniform_u32(uint(d)) + var u_tot2 = uniform_u32(uint(rows2 * nch)) + var u_tot3 = uniform_u32(uint(rows3 * nch)) + var u_totd = uniform_u32(uint(tt * d)) + var kn = CnMelNormArgs(nmel = uint(n_mel), nlen = uint(n_len), valid = uint(min(st.n_valid, n_len)), mfloor = 5.960464477539063e-08) //! 2^-24, the CPU form's log guard + var a0 = PkFrontArgs(iw = uint(n_mel), ih = uint(n_len), ow = uint(w1), oh = uint(t1), ch = uint(nch), total = uint(t1 * w1 * nch)) + var a2 = PkFrontArgs(iw = uint(w1), ih = uint(min(l0, t1)), ow = uint(w2), oh = uint(t2), ch = uint(nch), total = uint(rows2 * nch)) + var a5 = PkFrontArgs(iw = uint(w2), ih = uint(min(l2, t2)), ow = uint(w3), oh = uint(tt), ch = uint(nch), total = uint(rows3 * nch)) + var axf = PkFrontArgs(ow = uint(w3), oh = uint(tt), ch = uint(nch), total = uint(mpt * feat)) + let panel = uint64(9l * nch * 4l) + var err = "" + let ran = with_compute_encoder(g_queue, err) $(enc : MetalComputeEncoder?) { // nolint:PERF026 — the error-text fetch runs only on the failure leg + enc_f32_mm(enc, bufs[1], 0ul, bufs[0], bufs[2], 0ul, u_kfft, u_dft, mp, dft_out_pad) + enc_q3a_pow(enc, bufs[2], bufs[3], u_nbins, u_dft, u_ldpow, u_totpow, mp * ldpow) + enc_f32_mm(enc, bufs[4], 0ul, bufs[3], bufs[5], 0ul, u_ldpow, u_nmel, mp, n_mel) + enc_cn_melnorm(enc, bufs[5], kn, n_mel) + enc_pk_conv0(enc, bufs[5], bufs[6], btaps, 0ul, bblob, uint64(t.c0_b * 4l), a0, int64(a0.total)) + enc_pk_dw2d(enc, bufs[6], bufs[7], btaps, panel, bblob, uint64(t.c2_b * 4l), a2, int64(a2.total)) + enc_f32_mm(enc, bblob, uint64(t.c3_w * 4l), bufs[7], bufs[8], 0ul, u_nch, u_nch, mp2, nch) + enc_bias_relu_rows(enc, bufs[8], 0ul, bblob, uint64(t.c3_b * 4l), u_nch, u_tot2, rows2 * nch) + enc_pk_dw2d(enc, bufs[8], bufs[9], btaps, 2ul * panel, bblob, uint64(t.c5_b * 4l), a5, int64(a5.total)) + enc_f32_mm(enc, bblob, uint64(t.c6_w * 4l), bufs[9], bufs[10], 0ul, u_nch, u_nch, mp3, nch) + enc_bias_relu_rows(enc, bufs[10], 0ul, bblob, uint64(t.c6_b * 4l), u_nch, u_tot3, rows3 * nch) + enc_pk_xf(enc, bufs[10], bufs[11], axf, mpt * feat) + enc_f32_mm(enc, bblob, uint64(t.pre_w * 4l), bufs[11], bufs[12], 0ul, u_feat, u_d, mpt, d) + enc_add_bias_rows(enc, bufs[12], 0ul, bblob, uint64(t.pre_b * 4l), u_d, u_totd, tt * d) + } + if (ran) { + ensure_length(st.x, tt * d) + unsafe { + memcpy(addr(st.x[0]), metal_buffer_contents(bufs[12]), uint64(tt * d * 4l)) + } + g_tower_convs++ + } + for (b, n in bufs, sizes) { + pool_release(g_tw_pool, b, n) + } + for (u in fixed_array(u_kfft, u_dft, u_nbins, u_ldpow, u_totpow, u_nmel, u_nch, u_feat, u_d, u_tot2, u_tot3, u_totd)) { + pool_release(g_upool, u, 4ul) + } + return ran ? tt : -1l +} + //! Parakeet's block offsets in the shared chain's layout: the same regions, no GEMM biases (-1). def private pk_fc_layer(lo : PkLayerOffs) : CanaryLayerOffs { return CanaryLayerOffs(nf1_w = lo.nf1_w, nf1_b = lo.nf1_b, @@ -4703,6 +4840,7 @@ def private tw_weights_drop { st2_slab_free(g_tw_fc_twin) st2_slab_free(g_tw_q8) st2_slab_free(g_tw_conv1_pad) + st2_slab_free(g_tw_cn_front) st2_slab_drop() } @@ -4734,6 +4872,7 @@ def dasllama_metal_tower_register { register_gemma4a_gpu(@@metal_gemma4a_blocks, @@metal_tower_serves) register_gemma4a_chunk_gpu(@@metal_gemma4a_chunk) register_canary_gpu(@@metal_canary_blocks, @@metal_tower_serves) + register_canary_front_gpu(@@metal_canary_front) register_parakeet_gpu(@@metal_parakeet_blocks, @@metal_tower_serves) register_parakeet_gpu_encode(@@metal_parakeet_encode) register_styletts2_gpu(metal_styletts2_driver()) diff --git a/modules/dasLLAMA/followup_metal.md b/modules/dasLLAMA/followup_metal.md index d6c5d3fff9..7dc379a43e 100644 --- a/modules/dasLLAMA/followup_metal.md +++ b/modules/dasLLAMA/followup_metal.md @@ -901,10 +901,9 @@ Each is a stage and the models it holds for, read off the drivers' own gates: on the device. - **The q8 encoder lane** declines `quant_mode` on gemma4a, canary and parakeet, so those families serve their f32 lane on the device at the larger image; the whisper-class tower reads q8. -- **The canary mel and subsample front** run on the CPU (the front seat is filled by the Vulkan - driver alone): of a 64 s clip's 1274 ms they are 239 and 258, beside 80 of blocks on the device, - and the perception projection (7) is on the CPU too. Its stage clock is `cn.mel`, `cn.front`, - `cn.blocks.gpu`, `cn.proj`. +- **The canary perception projection** (`cn.proj`, 7 ms of a 64 s clip) runs on the CPU, and so do + the preemphasis and windowing ahead of the device front. A clip whose first conv output passes + 1 GB (about five minutes) keeps the CPU front, and the q8 encoder lane the whole CPU chain. - **The Qwen3-ASR projector** (`q3a.proj`, 1.6 ms a chunk) runs on the CPU, and its mel's windowing and clamp around the device spectrum. - **Qwen3-Omni** has had no stage-by-stage check. diff --git a/modules/dasLLAMA/tests/CLAUDE.md b/modules/dasLLAMA/tests/CLAUDE.md index 94fbc9a4ca..7f6754db0d 100644 --- a/modules/dasLLAMA/tests/CLAUDE.md +++ b/modules/dasLLAMA/tests/CLAUDE.md @@ -245,7 +245,10 @@ Conformer cells, E2B and E4B (f32-lane transcript equality CPU vs GPU + the soft counter deltas, the projector tail run on the device - `out_ready` set on the device leg and clear on the CPU leg - and the two projector widths read off the file: the encoder's 1536 on both, the embedder's 1536 on E2B and 2560 on E4B; the lane pin/reset discipline mirrors qwen3a's), the canary Metal FastConformer cell (the -same discipline over the rel-pos XL block loop; decoder = the q8_0 serving artifact), the +same discipline over the rel-pos XL block loop, the mel and the subsample front served on the +device ahead of it - the conv counter rising - and the control a clip with half its windowed +frames silenced, which the per-feature normalization cannot absorb as it would a scaled clip; +decoder = the q8_0 serving artifact), the parakeet Metal FastConformer cell (the same chain over parakeet's f32 blob, minted in memory; transcript equality CPU vs GPU + the encoder rows' rel-l2 + counter deltas - the subsample front counted on the device at every GPU encode - then the front lever off (`set_metal_parakeet_front`: @@ -400,7 +403,10 @@ front's cells (the same file): the first conv and the depthwise conv on an 11 x along one axis, even along the other, so each edge drops its own taps - against the in-test loops, the sums asserted to take both signs so the first conv's ReLU and the depthwise conv's lack of one both show; the feature permute bit for bit with its row pad zero under a sentinel -fill; the bias-and-ReLU row pass against max(x + b, 0) over sums of both signs. The FastConformer +fill; the bias-and-ReLU row pass against max(x + b, 0) over sums of both signs. Canary's mel +normalization (`cn_melnorm`): the log and the per-feature normalization in place against a double +form at 70 features (off the 64-thread group), 11 frames of which 7 are valid - the rest zero - and +two rows past the mel left as they were. The FastConformer all-heads attention's two stamps (the same file): the half operands (`fc_twin`) bit for bit the RNE narrow of the host's sum or product over all five panels, their pad rows zero, one poisoned input a panel; the rel-shift softmax (`fc_pexp`) over two heads at 24 and 300 keys - the weights within diff --git a/modules/dasLLAMA/tests/test_metal_prefill_kernels.das b/modules/dasLLAMA/tests/test_metal_prefill_kernels.das index 4dec17591d..d1e6cceb00 100644 --- a/modules/dasLLAMA/tests/test_metal_prefill_kernels.das +++ b/modules/dasLLAMA/tests/test_metal_prefill_kernels.das @@ -819,6 +819,49 @@ def private pk_xf_gate(t : T?; var dev, queue) { } } +//! Canary's log and per-feature normalization, in place over the mel sums: log(sum + floor), the mean and the Bessel +//! deviation over the valid frames in double, every valid frame normalized and the frames past them zero; rows past +//! the mel keep their sentinel. +def private cn_melnorm_gate(t : T?; var dev, queue) { + let tag = "cn_melnorm" + with_metal_cell(dev, queue) $(var mc : MetalCell) { + var err : string + let kd = kernel_of(@@enc_cn_melnorm) + var pso = mc |> keep_stamp(kd, err) + return if (!pso_ok(t, tag, pso, err)) + let nmel = 70 // off the 64-thread group: the last group's guard runs + let nlen = 11 + let valid = 7 + let rows = 13 // two rows past the mel: untouched + let mfloor = 5.960464477539063e-08 + var inscope x <- [for (i in range(rows * nmel)); 0.001 + float((i * 37) % 101) * 0.37] + var inscope want := x + for (j in range(nmel)) { + var sum = 0.0lf + for (i in range(valid)) { + sum += log(double(x[i * nmel + j]) + double(mfloor)) + } + let mean = sum / double(valid) + var sq = 0.0lf + for (i in range(valid)) { + let dv = log(double(x[i * nmel + j]) + double(mfloor)) - mean + sq += dv * dv + } + let denom = sqrt(sq / double(valid - 1)) + 1e-5lf + for (i in range(nlen)) { + want[i * nmel + j] = i < valid ? float((log(double(x[i * nmel + j]) + double(mfloor)) - mean) / denom) : 0.0 + } + } + var ka = CnMelNormArgs(nmel = uint(nmel), nlen = uint(nlen), valid = uint(valid), mfloor = mfloor) + let bx = mc |> io_(x) + let ran = mc |> cell_record(t, tag) $(enc : MetalComputeEncoder?) { + invoke(kd.enc, enc, pso, mc.buf[bx], ka, int64(nmel)) + } + return if (!ran) + check_buf(t, tag, mc.buf[bx], want, BAR_APPROX, Shape(row_width = nmel)) + } +} + //! A 1x1 conv's bias and ReLU over its rows, in place: max(x + b, 0) against the same f32 form, the fixture on //! both sides of zero. def private bias_relu_rows_gate(t : T?; var dev, queue) { @@ -2725,6 +2768,7 @@ def test_metal_prefill_kernels(t : T?) { pk_front_conv_gate(t, dev, queue, true) pk_front_conv_gate(t, dev, queue, false) pk_xf_gate(t, dev, queue) + cn_melnorm_gate(t, dev, queue) bias_relu_rows_gate(t, dev, queue) st2_conv_gate(t, dev, queue, false, false) st2_conv_gate(t, dev, queue, true, false) diff --git a/modules/dasLLAMA/tests/test_model_image.das b/modules/dasLLAMA/tests/test_model_image.das index 1b2bccb188..f563ffcb44 100644 --- a/modules/dasLLAMA/tests/test_model_image.das +++ b/modules/dasLLAMA/tests/test_model_image.das @@ -2396,13 +2396,15 @@ def test_canary_tower_metal(t : T?) { canary_encode(enc, stc) set_metal_tower(true) canary_log_mel(enc, samples, stg) - let e0 = metal_tower_stats().encodes + let e0 = metal_tower_stats() canary_encode(enc, stg) - rows_gpu = metal_tower_stats().encodes - e0 + rows_gpu = metal_tower_stats().encodes - e0.encodes + t |> equal(metal_tower_stats().convs - e0.convs, 1l, "the mel and the subsample front served on the device") enc_rel_l2 = rel_l2(stg.out, stc.out) - // the control: the same clip with its mel scaled x3 through the GPU chain must red the bar - for (i in range(length(stg.mel))) { - stg.mel[i] *= 3.0 + // the control: the device front reads the windowed frames, so the clip with the first half of them + // silenced, through the GPU chain, must red the bar (a scaled clip would not: the mel normalizes per feature) + for (i in range(length(stg.xw) / 2)) { + stg.xw[i] = 0.0 } canary_encode(enc, stg) poison_rel_l2 = rel_l2(stg.out, stc.out) From bc1b3787e7e3d73496dceec02e966882a185e936 Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Sun, 4 Oct 2026 01:19:59 -0700 Subject: [PATCH 13/46] dasLLAMA: the audio embedder carries the whisper-class tower - a Qwen2-Audio, Qwen2.5-Omni, Ultravox or Voxtral mmproj arms a chat slot's audio, where only the gemma-4 Conformer did The tower's blocks ride the Metal half twin off an f16 or bf16 mmproj (the twin baked at stage, a twinless image minted again), and a clip of 30 s or less is one mel chunk, as the reference reads it, where the 31 s pre-extend made two. The mmproj probes read files past 2 GB. The embedder test's direct-image cell selects the lane the box serves by the lane-named image, where its scan for a hash-named one skipped every run. Co-Authored-By: Claude Fable 5.1 --- modules/dasLLAMA/dasllama/dasllama_audio.das | 123 ++++++++++++++---- .../dasllama/dasllama_audio_embedder.das | 41 +++++- .../dasLLAMA/dasllama/dasllama_gemma4a.das | 2 +- modules/dasLLAMA/dasllama/dasllama_qwen3a.das | 13 +- modules/dasLLAMA/tests/test_audio.das | 21 ++- .../dasLLAMA/tests/test_audio_embedder.das | 73 ++++++++++- 6 files changed, 224 insertions(+), 49 deletions(-) diff --git a/modules/dasLLAMA/dasllama/dasllama_audio.das b/modules/dasLLAMA/dasllama/dasllama_audio.das index ed53efa4e6..44fd92d99f 100644 --- a/modules/dasLLAMA/dasllama/dasllama_audio.das +++ b/modules/dasLLAMA/dasllama/dasllama_audio.das @@ -198,14 +198,12 @@ def fft_pow2_run(n : int64; rev : int const?; tw : float const?; var reim : floa } //! Whisper log-mel spectrogram over the full padded input, split into full 3000-frame chunks — -//! the mtmd flavor end-to-end (pad to ≥31 s, reflect-200 head, Hann/hop-160 power spectra, -//! slaney mel, log10, global max−8 clamp). Audio ≤30 s yields exactly one chunk. +//! the mtmd flavor end-to-end (reflect-200 head, 30 s of zeros and 200 past the audio, Hann/hop-160 +//! power spectra, slaney mel, log10, global max−8 clamp). Audio under 30 s yields exactly one chunk. def log_mel_chunks(samples : array | #; n_mel : int64) : array { let n_in = length(samples) - // this flavor zero-extends short input to chunk+1 seconds BEFORE the stage pads — the head - // reflect below must read from the extended buffer, not the raw input - let min_samples = int(AUDIO_SAMPLE_RATE * 31l) - let n_base = max(n_in, min_samples) + // the head reflect reads one sample plus half a window: shorter input counts as that many, zero past its end + let n_base = max(n_in, int(AUDIO_N_FFT / 2l) + 1) let head = int(AUDIO_N_FFT / 2l) // reflect pad let tail = int(AUDIO_SAMPLE_RATE * 30l) + head // 30 s zeros + 200 let n_padded = head + n_base + tail @@ -1013,10 +1011,12 @@ def load_audio_tower(path : string; q8 : bool = false) : AudioTower { let img = image_path_for(path, tag) let ts_img = ref_time_ticks() var t = AudioTower() - if (load_image(img, t, tag)) { + // an exact-lane image minted before the half twin rode it carries none: where the file's GEMMs are halves, mint again + if (load_image(img, t, tag) && (q8 || length(t.wblob) > 0l || !audio_tower_half_gemms(path))) { to_log(LOG_INFO, "dasLLAMA: prepared tower image mapped in {get_time_usec(ts_img) / 1000} ms - {img}\n") return <- t } + delete t var inscope st <- stage_audio_tower(path, q8) var built = AudioTower() if (cache_via_image_staged(st, built, img, tag)) { @@ -1032,6 +1032,90 @@ def load_audio_tower(path : string; q8 : bool = false) : AudioTower { return <- built } +//! whether every block GEMM of a whisper-class mmproj - q, k, v, out, the two FFN matrices of each of `n_layer` blocks - is +//! stored as ggml type `gt`: the half twin is baked only off a file whose GEMMs are all one half type +def tower_gemm_types_uniform(m : GGUFMeta; n_layer : int64; gt : int) : bool { + for (l in range64(n_layer)) { + for (nm in ["attn_q", "attn_k", "attn_v", "attn_out", "ffn_up", "ffn_down"]) { + if (gguf_tensor_type(m, "a.blk.{l}.{nm}.weight") != gt) { + return false + } + } + } + return true +} + +//! whether a whisper-class mmproj stores every block GEMM in one half type (f16 or bf16); false for a file that cannot be read +def private audio_tower_half_gemms(path : string) : bool { + let f = fopen(path, "rb") + return false if (f == null) + var half = false + fmap(f) $(var bytes : array#) { + if (long_length(bytes) < 4l || rd_u32(bytes, 0l) != 0x46554747u) { + return + } + var inscope m <- parse_gguf_meta(bytes) + let gt = gguf_tensor_type(m, "a.blk.0.attn_q.weight") + let n_layer = gguf_has(m, "clip.audio.block_count") ? gguf_int(m, bytes, "clip.audio.block_count") : 0l + half = (gt == GGML_TYPE_BF16 || gt == GGML_TYPE_F16) && n_layer > 0l && tower_gemm_types_uniform(m, n_layer, gt) + } + fclose(f) + return half +} + +//! the projector kind an mmproj's `clip.projector_type` names, false for a type the audio tower does not serve +def private audio_tower_projector(proj : string; var kind : AudioProjKind&) : bool { + if (proj == "qwen2a" || proj == "qwen2.5o") { // omni's audio side is qwen2a verbatim + kind = AudioProjKind.qwen2a + } elif (proj == "ultravox") { + kind = AudioProjKind.ultravox + } elif (proj == "voxtral") { + kind = AudioProjKind.voxtral + } else { + return false + } + return true +} + +//! An MLP projector's hidden and output widths off its tensors - `clip.audio.projection_dim` lies for ultravox-1b (4096 +//! against 2048) - zeros where a tensor is missing +def private audio_tower_mlp_shape(m : GGUFMeta; kind : AudioProjKind; mm1_in : int64) : tuple { + let i1 = gguf_find_tensor(m, "mm.a.mlp.1.weight") + let i2 = gguf_find_tensor(m, "mm.a.mlp.2.weight") + return (mm1_out = 0l, proj_dim = 0l) if (i1 < 0 || i2 < 0 || mm1_in <= 0l) + let mm1_out = m.tensors[i1].n_elem / mm1_in + let mm2_in = kind == AudioProjKind.ultravox ? mm1_out / 2l : mm1_out + return (mm1_out = mm1_out, proj_dim = mm2_in > 0l ? m.tensors[i2].n_elem / mm2_in : 0l) +} + +//! The decoder width a whisper-class audio mmproj (qwen2a, qwen2.5o, ultravox, voxtral) projects to; 0 for a missing +//! file, a file that is no GGUF, or an mmproj this tower does not serve. Never panics: the seams gate an audio arm on it. +def audio_tower_probe_proj_dim(path : string) : int64 { + return 0l if (path |> ends_with(".dlim")) + let f = fopen(path, "rb") + return 0l if (f == null) + var dim = 0l + fmap(f) $(var bytes : array#) { + if (long_length(bytes) < 4l || rd_u32(bytes, 0l) != 0x46554747u) { + return + } + var inscope m <- parse_gguf_meta(bytes) + var kind = AudioProjKind.qwen2a + if (!gguf_has(m, "clip.projector_type") || !audio_tower_projector(gguf_str(m, bytes, "clip.projector_type"), kind) + || !gguf_has(m, "clip.audio.embedding_length") || gguf_find_tensor(m, "a.position_embd.weight") < 0) { + return + } + if (kind == AudioProjKind.qwen2a) { + dim = gguf_has(m, "clip.audio.projection_dim") ? gguf_int(m, bytes, "clip.audio.projection_dim") : 0l + } else { + let stack = gguf_has(m, "clip.audio.projector.stack_factor") ? gguf_int(m, bytes, "clip.audio.projector.stack_factor") : 1l + dim = audio_tower_mlp_shape(m, kind, gguf_int(m, bytes, "clip.audio.embedding_length") * stack).proj_dim + } + } + fclose(f) + return dim +} + //! Mint a served tower from a staged read WITHOUT touching disk — the off-rail control for suites. def mint_audio_tower(var st : AudioTowerStaging; var out : AudioTower; q8 : bool = false) { cache_via_image_staged(st, out, "", q8 ? "tower-q8" : "tower-f32") @@ -1048,13 +1132,7 @@ def stage_audio_tower(path : string; q8 : bool = false) : AudioTowerStaging { fmap(f) $(var bytes : array#) { var inscope m <- parse_gguf_meta(bytes) let proj = gguf_str(m, bytes, "clip.projector_type") - if (proj == "qwen2a" || proj == "qwen2.5o") { // omni's audio side is qwen2a verbatim - st.e.proj_kind = AudioProjKind.qwen2a - } elif (proj == "ultravox") { - st.e.proj_kind = AudioProjKind.ultravox - } elif (proj == "voxtral") { - st.e.proj_kind = AudioProjKind.voxtral - } else { + if (!audio_tower_projector(proj, st.e.proj_kind)) { panic("dasLLAMA: audio tower supports qwen2a/qwen2.5o/ultravox/voxtral projectors, got '{proj}'") } st.e.n_mel = gguf_int(m, bytes, "clip.audio.num_mel_bins") @@ -1079,18 +1157,14 @@ def stage_audio_tower(path : string; q8 : bool = false) : AudioTowerStaging { } st.e.n_ctx = m.tensors[pi].n_elem / d - // shapes come from the tensors, not clip.audio.projection_dim — it lies for ultravox-1b (4096 vs actual 2048) - var mm1_in = 0l + let mm1_in = st.e.proj_kind != AudioProjKind.qwen2a ? d * st.e.stack_factor : 0l if (st.e.proj_kind != AudioProjKind.qwen2a) { - mm1_in = d * st.e.stack_factor - let i1 = gguf_find_tensor(m, "mm.a.mlp.1.weight") - let i2 = gguf_find_tensor(m, "mm.a.mlp.2.weight") - if (i1 < 0 || i2 < 0) { + let shape = audio_tower_mlp_shape(m, st.e.proj_kind, mm1_in) + if (shape.proj_dim == 0l) { panic("dasLLAMA: mm.a.mlp.1/2.weight missing from '{proj}' mmproj") } - st.e.mm1_out = m.tensors[i1].n_elem / mm1_in - let mm2_in = st.e.proj_kind == AudioProjKind.ultravox ? st.e.mm1_out / 2l : st.e.mm1_out - st.e.proj_dim = m.tensors[i2].n_elem / mm2_in + st.e.mm1_out = shape.mm1_out + st.e.proj_dim = shape.proj_dim } // fixed blob layout: convs, pos, per-layer region, post-ln, projector @@ -1158,10 +1232,13 @@ def stage_audio_tower(path : string; q8 : bool = false) : AudioTowerStaging { gguf_read_tensor_f32(m, bytes, "mm.a.mlp.1.weight", st.fblob, st.e.proj_w_off, st.e.mm1_out * m1in) gguf_read_tensor_f32(m, bytes, "mm.a.mlp.2.weight", st.fblob, st.e.mm2_w_off, st.e.proj_dim * st.e.mm1_out) } + st.e.blk_bf16 = gguf_tensor_type(m, "a.blk.0.attn_q.weight") == GGML_TYPE_BF16 } fclose(f) if (q8) { tower_quantize(st) + } elif (audio_tower_half_gemms(path)) { // the file's GEMMs are halves: the Metal tower's half GEMMs read them back as they were stored + tower_bake_half_twin(st.fblob, st.wblob, st.e.blk_bf16) } return <- st } diff --git a/modules/dasLLAMA/dasllama/dasllama_audio_embedder.das b/modules/dasLLAMA/dasllama/dasllama_audio_embedder.das index 7fee9a33d5..bda8a9de9a 100644 --- a/modules/dasLLAMA/dasllama/dasllama_audio_embedder.das +++ b/modules/dasLLAMA/dasllama/dasllama_audio_embedder.das @@ -6,6 +6,7 @@ module dasllama_audio_embedder shared public require dasllama/dasllama_lint public require dasllama/dasllama_gemma4a public // re-exported: a requirer that carries the union must see the family to delete it +require dasllama/dasllama_audio public // the same for the whisper-class tower (qwen2a, qwen2.5o, ultravox, voxtral) require dasllama/dasllama_image // image_family_tag — a prepared .dlim routes on its baked tag require strings @@ -16,37 +17,45 @@ require strings enum AudioKind { none gemma4a // the gemma-4 E-series Conformer audio encoder + tower // the whisper-class tower with its projector: Qwen2-Audio, Qwen2.5-Omni, Ultravox, Voxtral } struct AudioEmbedder { kind : AudioKind = AudioKind.none gemma4a : Gemma4aEncoder = Gemma4aEncoder() + tower : AudioTower = AudioTower() } //! Per-encode scratch for whichever family the embedder carries (caller-owned, reused). struct AudioState { gemma4a : Gemma4aState = Gemma4aState() + enc : EncoderState = EncoderState() + enc_made : bool //! `enc` is sized for the embedder's tower: made at the first encode } // Stated rather than inferred: the family carrier holds borrowed planes, and a generated // teardown would force every module that deletes an AudioEmbedder to see the plane finalizers too. def finalize(var e : AudioEmbedder) { delete e.gemma4a + delete e.tower } def finalize(var s : AudioState) { delete s.gemma4a + delete s.enc } //! Soft-token width the encoder projects to (must equal the decoder width); 0 = no audio arm. def audio_proj_dim(e : AudioEmbedder) : int64 { if (e.kind == AudioKind.gemma4a) return e.gemma4a.proj_dim + if (e.kind == AudioKind.tower) return e.tower.proj_dim return 0l } //! The audio family the carrier holds ("gemma4a"; "" for none) — for logs and bench cells. def audio_family(e : AudioEmbedder) : string { if (e.kind == AudioKind.gemma4a) return "gemma4a" + if (e.kind == AudioKind.tower) return "{e.tower.proj_kind}" return "" } @@ -54,7 +63,8 @@ def audio_family(e : AudioEmbedder) : string { //! present, 0 otherwise (missing file, non-GGUF, a vision-only mmproj). Never panics — the //! seams gate their audio arms on this. def audio_probe_proj_dim(path : string) : int64 { - return gemma4a_probe_proj_dim(path) + let g4a = gemma4a_probe_proj_dim(path) + return g4a > 0l ? g4a : audio_tower_probe_proj_dim(path) } //! Load an audio mmproj (gguf, or a prepared ``.dlim`` routed on its baked family tag) as @@ -62,19 +72,44 @@ def audio_probe_proj_dim(path : string) : int64 { //! no audio family serves; probe first where absence is an answer. def load_audio_embedder(path : string) : AudioEmbedder { let is_gemma4a_image = path |> ends_with(".dlim") && image_family_tag(path) |> starts_with("gemma4a-") - if (is_gemma4a_image || audio_probe_proj_dim(path) > 0l) { + if (is_gemma4a_image || gemma4a_probe_proj_dim(path) > 0l) { var e = AudioEmbedder(kind = AudioKind.gemma4a) e.gemma4a <- load_gemma4a_encoder(path) return <- e } - panic("dasLLAMA audio: '{path}' carries no audio tower any family serves (gemma4a)") + let is_tower_image = path |> ends_with(".dlim") && image_family_tag(path) |> starts_with("tower-") + if (is_tower_image || audio_tower_probe_proj_dim(path) > 0l) { + var e = AudioEmbedder(kind = AudioKind.tower) + e.tower <- load_audio_tower(path) + return <- e + } + panic("dasLLAMA audio: '{path}' carries no audio tower any family serves (gemma4a, or a whisper-class tower: qwen2a, qwen2.5o, ultravox, voxtral)") return <- AudioEmbedder() } +//! the whisper-class tower over a whole clip: a 30 s mel chunk at a time, each chunk's soft tokens appended to `out` +def private tower_encode_all(t : AudioTower; var s : AudioState; samples : array | #; var out : array) : int64 { + if (!s.enc_made) { + s.enc <- make_encoder_state(t) + s.enc_made = true + } + var inscope chunks <- log_mel_chunks(samples, t.n_mel) + let rows = audio_tokens_per_chunk(t) + out |> reserve(length(out) + length(chunks) * int(rows * t.proj_dim)) + for (chunk in chunks) { + audio_encode(t, s.enc, chunk) + for (i in range64(rows * t.proj_dim)) { + out |> push(s.enc.out[i]) + } + } + return long_length(chunks) * rows +} + //! The whole audio path for the carried family: 16 kHz mono samples → the family's encode into //! ``out`` = nout × proj_dim soft tokens (returns nout). Panics on the no-audio arm. def encode_audio_(e : AudioEmbedder; var s : AudioState; samples : array | #; var out : array) : int64 { if (e.kind == AudioKind.gemma4a) return gemma4a_encode_all(e.gemma4a, s.gemma4a, samples, out) + if (e.kind == AudioKind.tower) return tower_encode_all(e.tower, s, samples, out) panic("dasLLAMA audio: encode_audio_ on an embedder with no audio arm") return 0l } diff --git a/modules/dasLLAMA/dasllama/dasllama_gemma4a.das b/modules/dasLLAMA/dasllama/dasllama_gemma4a.das index 7754fbd899..3515b31f7a 100644 --- a/modules/dasLLAMA/dasllama/dasllama_gemma4a.das +++ b/modules/dasLLAMA/dasllama/dasllama_gemma4a.das @@ -353,7 +353,7 @@ def gemma4a_probe_proj_dim(path : string) : int64 { } var dim = 0l fmap(f) $(var bytes : array#) { - if (length(bytes) < 4 || !(int(bytes[0]) == 'G' && int(bytes[1]) == 'G' && int(bytes[2]) == 'U' && int(bytes[3]) == 'F')) { + if (long_length(bytes) < 4l || !(int(bytes[0]) == 'G' && int(bytes[1]) == 'G' && int(bytes[2]) == 'U' && int(bytes[3]) == 'F')) { return } var inscope m <- parse_gguf_meta(bytes) diff --git a/modules/dasLLAMA/dasllama/dasllama_qwen3a.das b/modules/dasLLAMA/dasllama/dasllama_qwen3a.das index 1a6ccdf505..638539604e 100644 --- a/modules/dasLLAMA/dasllama/dasllama_qwen3a.das +++ b/modules/dasLLAMA/dasllama/dasllama_qwen3a.das @@ -164,17 +164,6 @@ def private q3a_announce_lane() { } [arch(at="../ARCHITECTURE_IMAGE.md#image-tower-twin-plane")] -def private q3a_gemm_types_uniform(m : GGUFMeta; n_layer : int64; gt : int) : bool { - for (l in range64(n_layer)) { - for (nm in ["attn_q", "attn_k", "attn_v", "attn_out", "ffn_up", "ffn_down"]) { - if (gguf_tensor_type(m, "a.blk.{l}.{nm}.weight") != gt) { - return false - } - } - } - return true -} - //! Load a Qwen3-ASR mmproj GGUF (projector type `qwen3a`, bf16/f16/f32 tensors). Like //! ``load_model``, caches a box/knob-specific PREPARED IMAGE for fast reload; //! ``DASLLAMA_IMAGE=0`` disables it, a ``.dlim`` path loads one directly. @@ -287,7 +276,7 @@ def stage_qwen3a_tower(path : string) : Qwen3aStaging { st.e.core <- c if (q3a_serve_q8()) { tower_quantize(st.e.core, st.core) // arm the shared core's q8 rail (encoder_blocks reads core.q8) - } elif ((gt == GGML_TYPE_BF16 || gt == GGML_TYPE_F16) && q3a_gemm_types_uniform(m, st.e.core.n_layer, gt)) { + } elif ((gt == GGML_TYPE_BF16 || gt == GGML_TYPE_F16) && tower_gemm_types_uniform(m, st.e.core.n_layer, gt)) { tower_bake_half_twin(st.core.fblob, st.core.wblob, st.e.core.blk_bf16) } diff --git a/modules/dasLLAMA/tests/test_audio.das b/modules/dasLLAMA/tests/test_audio.das index 66572dcd2b..557e6632da 100644 --- a/modules/dasLLAMA/tests/test_audio.das +++ b/modules/dasLLAMA/tests/test_audio.das @@ -79,14 +79,14 @@ def test_mel_filterbank(t : T?) { [test] def test_log_mel_chunks(t : T?) { t |> run("whisper mel: 440 Hz sine lands on its mel row, silence clamps") @(t : T?) { - // the mtmd-debug 'preproc 440' fixture: 5 s sine -> exactly 2 full 3000-frame chunks + // 5 s of sine and the 30 s of zeros past it: one full 3000-frame chunk, the partial one dropped var s : array s |> resize(80000) for (i in range(80000)) { s[i] = sin(2.0 * 3.14159265 * 440.0 * float(i) / 16000.0) } let chunks <- log_mel_chunks(s, 128l) - t |> success(length(chunks) == 2, "two chunks (audio + padded silence)") + t |> equal(length(chunks), 1, "a clip under 30 s is one chunk") t |> equal(chunks[0].n_mel, 128l, "n_mel") t |> equal(chunks[0].n_len, 3000l, "n_len") // 440 Hz sits in the slaney linear region: center(m=18) = 19·(m_hi/129)/lin_slope ≈ 444 Hz @@ -108,14 +108,21 @@ def test_log_mel_chunks(t : T?) { mx = max(mx, v) } t |> success(mx - mn <= 2.0 + 1e-6, "max-min <= 8/4 (global clamp)") - // the second chunk is all padded silence: every value at the clamp floor + // past the audio the chunk is padded silence: every value at the clamp floor var uniform = true - for (v in chunks[1].data) { - if (v != chunks[1].data[0]) { - uniform = false + for (m in range(128)) { + for (i in range(600, 3000)) { + if (chunks[0].data[m * 3000 + i] != chunks[0].data[600]) { + uniform = false + } } } - t |> success(uniform, "silence chunk uniform at the floor") + t |> success(uniform, "the frames past the audio are uniform at the floor") + t |> success(chunks[0].data[18 * 3000 + 100] != chunks[0].data[600], "the control: a frame of the sine is off the floor") + // 35 s of audio: the 30 s of zeros past it complete a second chunk + s |> resize(560000) + let two <- log_mel_chunks(s, 128l) + t |> equal(length(two), 2, "a clip between 30 and 60 s is two chunks") } } diff --git a/modules/dasLLAMA/tests/test_audio_embedder.das b/modules/dasLLAMA/tests/test_audio_embedder.das index d2b89af0ec..4c2557bcca 100644 --- a/modules/dasLLAMA/tests/test_audio_embedder.das +++ b/modules/dasLLAMA/tests/test_audio_embedder.das @@ -82,17 +82,18 @@ def test_audio_embedder_gemma4a_arm(t : T?) { // baked family tag, or directory order hands this audio cell a vision image. The // load above already minted the gemma4a arm when the image rail is on, so this // scan only skips under DASLLAMA_IMAGE=0 — a model gate, not an artifact gate - if (empty(image_path) && starts_with(name, "{base_name(mm)}.0x") && ends_with(name, ".dlim")) { + if (empty(image_path) && starts_with(name, "{base_name(mm)}.") && ends_with(name, ".dlim")) { let cand = path_join(models_dir(), name) let tag = image_family_tag(cand) //! only the path the rail mints for THIS box identity is loadable by the direct route - a foreign-identity bake sits beside it under its own hash - if (tag |> starts_with("gemma4a-") && cand == image_path_for(mm, tag)) { + //! and of its two lanes only the one this box serves: the direct route reads under the lane the loader picked above + if (tag == (e.gemma4a.q8 ? "gemma4a-q8" : "gemma4a-f32") && cand == image_path_for(mm, tag)) { image_path = cand } } } if (empty(image_path)) { - t |> skip("no prepared gemma4a-tagged {base_name(mm)}.0x*.dlim beside the mmproj - the direct image route needs a baked image") + t |> skip("no prepared gemma4a-tagged {base_name(mm)}.*.dlim beside the mmproj - the direct image route needs a baked image") } else { t |> equal(audio_probe_proj_dim(image_path), 0l, "the gguf probe answers 0 on a .dlim by pinned contract") var inscope ei <- load_audio_embedder(image_path) @@ -102,6 +103,72 @@ def test_audio_embedder_gemma4a_arm(t : T?) { } } +def private tower_arm_cell(t : T?; file, family : string; width, rows_per_chunk : int64) { + let mm = path_join(models_dir(), file) + if (!model_available(t, mm)) { + return + } + t |> equal(audio_probe_proj_dim(mm), width, "the probe reports the projector's output width off its tensors") + var inscope e <- load_audio_embedder(mm) + t |> success(e.kind == AudioKind.tower, "the carrier holds the whisper-class tower") + t |> equal(audio_family(e), family, "the carrier reports the projector kind") + t |> equal(audio_proj_dim(e), width, "the carrier reports the probed width") + var st = AudioState() + var samples : array + samples |> resize(16000 * 5) + for (i, s in count(), samples) { + s = 0.3 * sin(float(i) * 0.05) + } + var rows : array + t |> equal(encode_audio_(e, st, samples, rows), rows_per_chunk, "a 5 s clip is one mel chunk") + t |> equal(long_length(rows), rows_per_chunk * width, "one row of the decoder's width a soft token") + var peak = 0. + var finite = true + for (v in rows) { + finite &&= is_finite(v) + peak = max(peak, abs(v)) + } + t |> success(finite && peak > 0., "the rows are finite and not all zero (peak {peak})") + var again : array + t |> equal(encode_audio_(e, st, samples, again), rows_per_chunk, "the reused state encodes the clip again") + var moved = 0 + for (a, b in rows, again) { + moved += a != b ? 1 : 0 + } + t |> equal(moved, 0, "the reused state reads the same rows bit for bit") + for (i, s in count(), samples) { + s = 0.3 * sin(float(i) * 0.11) + } + delete again + encode_audio_(e, st, samples, again) + moved = 0 + for (a, b in rows, again) { + moved += a != b ? 1 : 0 + } + t |> success(moved > length(rows) / 2, "another clip reads other rows ({moved} of {length(rows)} differ) - the control") + samples |> resize(16000 * 35) + delete again + t |> equal(encode_audio_(e, st, samples, again), 2l * rows_per_chunk, "a 35 s clip is two chunks, their rows appended") + delete again + delete rows + delete samples + delete st +} + +[test] +def test_audio_embedder_tower_arm(t : T?) { + t |> run("an ultravox mmproj loads as the whisper-class tower") @(t : T?) { + with_dasllama_jobque() { + tower_arm_cell(t, "mmproj-ultravox-v0_5-llama-3_2-1b-f16.gguf", "ultravox", 2048l, 187l) + } + } + t |> run("the Qwen2.5-Omni mmproj's audio side loads as the whisper-class tower beside its vision tensors") @(t : T?) { + with_dasllama_jobque() { + tower_arm_cell(t, "mmproj-Qwen2.5-Omni-3B-f16.gguf", "qwen2a", 2048l, 750l) + } + } +} + // The pre-encoded audio rows on a PLAIN chat (no tower): the E2B family hears through this seam, // and every refusal at it is a panic the caller must never reach by accident. // The rows turn with thinking OFF under the chat sampler preset: the E-series opens a media turn From 10e736eee0e9cae6ee06fd55b3257a1fc07026e1 Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Sun, 4 Oct 2026 01:36:31 -0700 Subject: [PATCH 14/46] gc: the mark walk sizes a range in 64 bits - a borrowed array whose byte length is a whole multiple of 4 GiB read as empty and the validating collect indexed the heap's chunk table by it Co-Authored-By: Claude Fable 5.1 --- src/simulate/simulate_gc.cpp | 6 ++---- tests/gc/test_gc_borrowed_4gib.das | 23 +++++++++++++++++++++++ 2 files changed, 25 insertions(+), 4 deletions(-) create mode 100644 tests/gc/test_gc_borrowed_4gib.das diff --git a/src/simulate/simulate_gc.cpp b/src/simulate/simulate_gc.cpp index 87d919d423..5521bc6b87 100644 --- a/src/simulate/simulate_gc.cpp +++ b/src/simulate/simulate_gc.cpp @@ -217,8 +217,7 @@ namespace das if ( currentRange.empty() ) return true; if ( currentRange.contains(r) ) return false; if ( heapOnly ) { - int ssize = int(r.to-r.from); - ssize = (ssize + 15) & ~15; + uint64_t ssize = (uint64_t(r.to-r.from) + 15) & ~uint64_t(15); return context->heap->isOwnPtr(r.from, ssize); } return true; @@ -706,8 +705,7 @@ namespace das bool result = true; ptrRangeStack.push_back(currentRange); if ( !r.empty() && !currentRange.contains(r) ) { - int ssize = int(r.to-r.from); - ssize = (ssize + 15) & ~15; + uint64_t ssize = (uint64_t(r.to-r.from) + 15) & ~uint64_t(15); if ( validate ) { if ( context->heap->isOwnPtr(r.from, ssize) ) { if ( context->heap->isValidPtr(r.from, ssize) ) { diff --git a/tests/gc/test_gc_borrowed_4gib.das b/tests/gc/test_gc_borrowed_4gib.das new file mode 100644 index 0000000000..9e2729dd6f --- /dev/null +++ b/tests/gc/test_gc_borrowed_4gib.das @@ -0,0 +1,23 @@ +// A borrowed array view (a mapped file plane) whose byte length is a whole multiple of 4 GiB: the mark walk sizes +// the range in 64 bits. Sized in 32, the range read as empty and the validating collect indexed the heap's chunk +// table by it. +options gen2 +options persistent_heap +options gc +options no_aot + +require dastest/testing_boost public + +var g_backing : uint64[16] // global memory, outside the heap, as a mapping is +var g_plane : array + +[test] +def test_gc_borrowed_range_of_4gib(t : T?) { + let n = 1l << 29l // 2^29 uint64 = 4 GiB of range over 16 real elements; the walk reads no element of a uint64 array + unsafe { + _builtin_make_temp_array_i64(g_plane, reinterpret(addr(g_backing[0])), n) + heap_collect(true, true) + } + t |> equal(long_length(g_plane), n, "the view survives a validating collect") + unsafe(_builtin_forget_temp_array(g_plane)) +} From 8356edc677f9f6d55ff6bbc86e3ae24bf95d0777 Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Sun, 4 Oct 2026 02:41:39 -0700 Subject: [PATCH 15/46] dasLLAMA: the chat towers' projector tail and mel run on the Metal tower, and an audio request frees what it allocates The whisper-class block chain gains a tail form: the pair pool, the post-norm and the projector of each kind (qwen2a, ultravox, voxtral) behind the blocks in one command buffer, through the blocks-with-tail seat. The chunked mel asks the whisper mel seat Qwen3-ASR's mel already filled; the seat's two GEMMs ride the exact f32 tiles, since a half tile read a quiet bin 0.085 off. An 11 s clip's encode on an f16 projector file: about 550 ms as served (the tail's CPU GEMMs under the media worker's dispatch) to 102. The chunked mel leaked 18 MB an encode, and the Qwen3-ASR, canary and gemma-4 transcriptions 2 to 3.6 MB a request: their temporaries are scoped now, and the tests hold the heap flat across repeated requests. The embedder also answers whether a projector's rows splice bare (ultravox). Co-Authored-By: Claude Fable 5.1 --- modules/dasLLAMA/ARCHITECTURE_GPU.md | 2 +- modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md | 14 ++ modules/dasLLAMA/ARCHITECTURE_MEDIA.md | 18 +- modules/dasLLAMA/dasllama/dasllama_audio.das | 181 ++++++++++++----- .../dasllama/dasllama_audio_embedder.das | 10 + modules/dasLLAMA/dasllama/dasllama_canary.das | 14 +- .../dasLLAMA/dasllama/dasllama_gemma4a.das | 18 +- .../dasllama/dasllama_metal_common.das | 2 + .../dasllama/dasllama_metal_prefill.das | 51 ++++- .../dasllama/dasllama_metal_tower.das | 188 +++++++++++++++++- modules/dasLLAMA/dasllama/dasllama_qwen3a.das | 66 +++--- .../dasllama/dasllama_vulkan_tower.das | 2 +- modules/dasLLAMA/followup_metal.md | 5 +- modules/dasLLAMA/tests/CLAUDE.md | 51 ++++- modules/dasLLAMA/tests/test_audio.das | 106 +++++++++- .../dasLLAMA/tests/test_audio_embedder.das | 22 ++ .../tests/test_metal_prefill_kernels.das | 45 +++++ modules/dasLLAMA/tests/test_whisper.das | 21 ++ tests/gc/test_gc_borrowed_4gib.das | 2 +- 19 files changed, 684 insertions(+), 134 deletions(-) diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU.md b/modules/dasLLAMA/ARCHITECTURE_GPU.md index deac8f98a7..7e6bebd8f2 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU.md @@ -24,7 +24,7 @@ that a question answered for one backend has an obvious address in the other. Th | `dasllama__decode`
`dasllama_metal_decode`, `dasllama_vulkan_decode` | the resident token-step driver + decode-time arms; on Vulkan, for a hyper-connection MoE (`Config.hyper_conn`), the split token command and the hot pool's device slots, stacks and hit chains (`ARCHITECTURE_GPU_VULKAN_HC.md#hc-token-command`, `#hc-hot-pool`) | kernel bodies | | `dasllama__prefill`
`dasllama_metal_prefill`, `dasllama_vulkan_prefill` | the batched prefill driver + batch arms; on Vulkan, for a hyper-connection MoE, the window chain cut after each router (`ARCHITECTURE_GPU_VULKAN_HC.md#hc-window-chain`) | kernel bodies | | `dasllama__shapes`
`dasllama_metal_shapes` | PORTABLE servability gates - no GPU C++ require, so any box can bake | device calls | -| the tower driver
`dasllama_metal_tower`, `dasllama_vulkan_tower` (the vision ViT chains over the q8 image, qwen25v's over the halfword twin, and the audio chains over the q8 image - the whisper-class towers with their conv stem, the gemma4a Conformer with its chunk front and projector tail, the canary FastConformer with its front, the qwen3a front and mel; on a build without das_metal it fills the gemma4v and gemma3v hook slots, the blocks-only seats `register_qwen3v_gpu_blocks` and `register_qwen25v_gpu_blocks`, the Vulkan seats of those two families, and the audio seats `register_tower_blocks_gpu`, `register_tower_blocks_ln_post_gpu`, `register_tower_conv_gpu`, `register_qwen3a_front_gpu`, `register_qwen3a_mel_gpu`, `register_gemma4a_gpu`, `register_gemma4a_chunk_gpu`, `register_canary_gpu` and `register_canary_front_gpu`; parakeet's seat is Metal's) | one-shot embedder/encoder encodes (gemma4uv chain, the gemma4v ViT, gemma3v SigLIP and qwen3v block loops - qwen3v adds the vision NEOX rope, the fused-qkv weight-offset GEMMs, and the inline deepstack tap + tail merger chains - the whisper-class block loop, the qwen25v window ViT, the gemma4a Conformer chain with its mel/conv front, the FastConformer chain canary and parakeet share - one block body over `CanaryLayerOffs`, parakeet's offsets mapped onto it with no GEMM biases and the tap-major depthwise stamp - the Pocket TTS codec and frame loop (`ARCHITECTURE_GPU_TOWER.md#tower-pocket-codec`, `ARCHITECTURE_GPU_TOWER.md#tower-pocket-frames`) - the whole StyleTTS2 synthesis (kitten and kokoro on both weight lanes: seven seats from PL-BERT to the inverse STFT, the front end on the f32-exact GEMM stamp, the source chain the CPU's operation for operation, `ARCHITECTURE_GPU_TOWER.md#tower-tts-chain`) - the conv frontends + the qwen3a padded-weight slab and GPU front/mel) - no session, no mirror - the Pocket frame loop's per-voice device K/V slot is the one state kept across calls; registers the gemma4uv, gemma4v, gemma3v, qwen3v, qwen25v, encoder_blocks, tower-conv, qwen3a-front, qwen3a-mel, gemma4a, gemma4a-chunk, canary, parakeet (`register_parakeet_gpu` for the blocks, `register_parakeet_gpu_encode` for the whole encode with the subsample front) and StyleTTS2 (`register_styletts2_gpu`, the seven-seat record) and Pocket (`register_pocket_gpu`, the codec and frames seats, `ARCHITECTURE_GPU_TOWER.md#tower-pocket-codec` and `ARCHITECTURE_GPU_TOWER.md#tower-pocket-frames`) hooks; the FastConformer chain borrows `enc_fc_pack` / `enc_fc_softmax` / `enc_fc_unpack` around `enc_f32_mm`, `enc_cn_dw` / `enc_pk_dw`, the prefill's `pf_enc_fc_attn_dev` (every head's attention at once on the device pair, under its crown) and `enc_bias_relu_rows` with the parakeet front's `enc_pk_conv0` / `enc_pk_dw2d` / `enc_pk_xf`, and the `enc_st2_*` row, LSTM, attention and source kernels around `enc_st2_conv_mm` / `enc_st2_conv_exact_mm`, the `enc_pk_*` row, add-and-norm, attention-row and prologue/epilogue GEMV kernels for the Pocket codec and frame loop beside the decode rail's `enc_gemv` / `enc_rpst32_c` and the prefill's `enc_rope`; on Vulkan the whisper-class encoder-output handoff - the served post-norm chain's `xb` plane, kept for the encode in flight alone, which the ASR-decoder driver's cross-KV chain reads (`vulkan_tower_enc_out`) - is the second state kept across calls beside the Pocket slot | LLM or ASR decoder session state | +| the tower driver
`dasllama_metal_tower`, `dasllama_vulkan_tower` (the vision ViT chains over the q8 image, qwen25v's over the halfword twin, and the audio chains over the q8 image - the whisper-class towers with their conv stem, the gemma4a Conformer with its chunk front and projector tail, the canary FastConformer with its front, the qwen3a front and mel; on a build without das_metal it fills the gemma4v and gemma3v hook slots, the blocks-only seats `register_qwen3v_gpu_blocks` and `register_qwen25v_gpu_blocks`, the Vulkan seats of those two families, and the audio seats `register_tower_blocks_gpu`, `register_tower_blocks_ln_post_gpu`, `register_tower_conv_gpu`, `register_qwen3a_front_gpu`, `register_whisper_mel_gpu`, `register_gemma4a_gpu`, `register_gemma4a_chunk_gpu`, `register_canary_gpu` and `register_canary_front_gpu`; parakeet's seat is Metal's) | one-shot embedder/encoder encodes (gemma4uv chain, the gemma4v ViT, gemma3v SigLIP and qwen3v block loops - qwen3v adds the vision NEOX rope, the fused-qkv weight-offset GEMMs, and the inline deepstack tap + tail merger chains - the whisper-class block loop, the qwen25v window ViT, the gemma4a Conformer chain with its mel/conv front, the FastConformer chain canary and parakeet share - one block body over `CanaryLayerOffs`, parakeet's offsets mapped onto it with no GEMM biases and the tap-major depthwise stamp - the Pocket TTS codec and frame loop (`ARCHITECTURE_GPU_TOWER.md#tower-pocket-codec`, `ARCHITECTURE_GPU_TOWER.md#tower-pocket-frames`) - the whole StyleTTS2 synthesis (kitten and kokoro on both weight lanes: seven seats from PL-BERT to the inverse STFT, the front end on the f32-exact GEMM stamp, the source chain the CPU's operation for operation, `ARCHITECTURE_GPU_TOWER.md#tower-tts-chain`) - the conv frontends + the qwen3a padded-weight slab and GPU front/mel) - no session, no mirror - the Pocket frame loop's per-voice device K/V slot is the one state kept across calls; registers the gemma4uv, gemma4v, gemma3v, qwen3v, qwen25v, encoder_blocks, tower-conv, qwen3a-front, qwen3a-mel, gemma4a, gemma4a-chunk, canary, parakeet (`register_parakeet_gpu` for the blocks, `register_parakeet_gpu_encode` for the whole encode with the subsample front) and StyleTTS2 (`register_styletts2_gpu`, the seven-seat record) and Pocket (`register_pocket_gpu`, the codec and frames seats, `ARCHITECTURE_GPU_TOWER.md#tower-pocket-codec` and `ARCHITECTURE_GPU_TOWER.md#tower-pocket-frames`) hooks; the FastConformer chain borrows `enc_fc_pack` / `enc_fc_softmax` / `enc_fc_unpack` around `enc_f32_mm`, `enc_cn_dw` / `enc_pk_dw`, the prefill's `pf_enc_fc_attn_dev` (every head's attention at once on the device pair, under its crown) and `enc_bias_relu_rows` with the parakeet front's `enc_pk_conv0` / `enc_pk_dw2d` / `enc_pk_xf`, and the `enc_st2_*` row, LSTM, attention and source kernels around `enc_st2_conv_mm` / `enc_st2_conv_exact_mm`, the `enc_pk_*` row, add-and-norm, attention-row and prologue/epilogue GEMV kernels for the Pocket codec and frame loop beside the decode rail's `enc_gemv` / `enc_rpst32_c` and the prefill's `enc_rope`; on Vulkan the whisper-class encoder-output handoff - the served post-norm chain's `xb` plane, kept for the encode in flight alone, which the ASR-decoder driver's cross-KV chain reads (`vulkan_tower_enc_out`) - is the second state kept across calls beside the Pocket slot | LLM or ASR decoder session state | | the ASR-decoder driver
`dasllama_metal_asr_dec`, `dasllama_vulkan_asr_dec` | the whisper decoder on the GPU: Metal's 34B weight blob or Vulkan's row-major q8 gather, the f16 resident cross/self K/V, window-granular cross-KV + decode-step serves; registers the whisper cross-KV and decode hooks (`register_whisper_cross_kv_gpu`, `register_whisper_decode_gpu`; family registries in `dasllama_whisper`); on a build without das_metal the Vulkan driver fills them (`ARCHITECTURE_GPU_TOWER_VULKAN.md#vk-asr-decoder`) | kernel bodies, LLM session state | | the TTS driver
`dasllama_vulkan_tts` (Vulkan; the Metal tower driver's own row carries the TTS seats on an Apple build) | the StyleTTS2 synthesis seats and the Pocket codec and frames seats on the tower's knob - the f32 weight slabs the shared host writer (`dasllama_tts_slab`) lays out, the front end on the f32-exact tile GEMM, the seats registered through `register_styletts2_gpu` and `register_pocket_gpu` on a build without das_metal; the seats serve only while the tier's want arms the device (`DASLLAMA_GPU=1`, off by default), so a run with no flags and no environment overrides synthesizes on the CPU chain on a Vulkan build, where on an Apple build the Metal tower driver's TTS seats serve by default (`DASLLAMA_METAL_TOWER` on) | kernel bodies, the CPU chain's arithmetic (the CPU chain is the specification), LLM or ASR session state | | the assistant-drafter driver
`dasllama_metal_mtp_gemma` | the gemma-4 assistant drafter on Metal: the sidecar blob upload, the Q-only layer chain reading the TARGET mirror at the two capture layers with the decode's own attention kernels, the speculative round over the batch driver's same-slab verify; registers the `metal` round override and delegates head-less-drafter-less models to `metal_mtp_spec_round` | kernel bodies, mirror ownership | diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md b/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md index ceb40a2b44..ed22013483 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_TOWER.md @@ -64,6 +64,20 @@ epoch and the tower shutdown. The stem's first conv reads the f32 blob on both l mels: 240) it reads a device copy of its rows zero-padded to the lattice (`tw_conv1_pad_attach`), the im2col pass padding its own rows with zeros to the same width. +The whisper-class chain has three forms over one block loop (`TwForm`): the blocks alone, the +blocks with the tower's post-norm (whisper's encoder output), and the blocks with the projector +tail - the chat towers' soft tokens (`tw_tail_body`). The tail is the CPU tail's steps at its +widths, by projector kind: qwen2a pools row pairs (`MetalTwPool2`), norms, and runs one biased +linear; voxtral pools, norms, and runs the stacked rows through a GELU MLP; ultravox norms every +position, rms-norms the stacked rows at the projector's own eps, and runs the gated MLP +(`MetalTwSwigluRows`: silu of a row's second half times its first). A stack is no kernel: the +stacked row is the same memory read at the wider row. The tail's GEMMs read the f32 blob on the +f32 tile, so the tail seat serves it on an f32 tower whose widths sit on that tile's lattice +(`tw_tail_shape_ok`) and serves the blocks alone otherwise - a q8 tower's projector planes are not +on the device - answering which it served (`TowerTailServed`), so the CPU tail runs exactly when +the device one did not. Its engage counter is `metal_tower_tail_encodes`, its lever +`set_metal_tower_tail`. + The gemma4a chain ends in the projector tail, in the blocks' command buffer: the out projection and its bias at the encoder's own output width (`gemma4a_mid_dim`, the length of the weightless norm's ones row), that norm, then the audio embedder to the decoder's width (`proj_dim`). The diff --git a/modules/dasLLAMA/ARCHITECTURE_MEDIA.md b/modules/dasLLAMA/ARCHITECTURE_MEDIA.md index 9a0917aa77..7edd4d96d8 100644 --- a/modules/dasLLAMA/ARCHITECTURE_MEDIA.md +++ b/modules/dasLLAMA/ARCHITECTURE_MEDIA.md @@ -135,7 +135,12 @@ carriers name, and no TTS family file (`dasllama_kitten`, `dasllama_kokoro`, `da - **`dasllama_audio_embedder.das`** - the audio carrier: `AudioEmbedder` / `AudioState`, the vision carrier's audio twin - one union through every seam (server media worker, facade `encode_audio`, tutorials), the family probed from the mmproj's audio tensor (or a `.dlim`'s - baked tag) at load, one-line arms. Outside a family's own file, an audio family type is named + baked tag) at load, one-line arms. It carries two kinds: the gemma-4 Conformer, and the + whisper-class tower with its projector (`AudioKind.tower`: Qwen2-Audio, Qwen2.5-Omni, Ultravox, + Voxtral), a clip encoded one 30 s mel chunk at a time into its state's `EncoderState`. The + projector says how its rows splice: an ultravox span takes no marker (`audio_span_bare`), since + the stock decoder it pairs with declares none, and every other kind takes the markers the + decoder's chat template declares. Outside a family's own file, an audio family type is named only here, in `dasllama_asr.das`'s union field and one-line arms (the ASR rail's own carrier), in the GPU family hooks (`dasllama_metal_tower.das`, `dasllama_metal_asr_dec.das`, `dasllama_vulkan_tower.das`, `dasllama_vulkan_asr_dec.das` - each fills the family's own seat @@ -187,14 +192,19 @@ blocks and merger alike - because the Metal tower reads f32 planes or the baked A family file owns the hook SLOT for a stage the GPU can serve - a `var private` function pointer plus a `register_*` entry - and a tower driver fills it at `[init]`: the Metal driver on a Metal build, the Vulkan driver (`dasllama_vulkan_tower.das`) on a build without das_metal, for the -blocks seats it serves and the front seats it fills (qwen3a's mel and conv front, gemma4a's +blocks seats it serves and the front seats it fills (the whisper preprocessor's mel +`register_whisper_mel_gpu`, which Qwen3-ASR's mel and the chat towers' chunked mel both ask; +qwen3a's conv front, gemma4a's whole chunk, canary's front, parakeet's whole encode - the Metal driver's, asked ahead of the CPU front with the blocks seat behind a decline - and the whisper-class blocks-with-post-norm seat `register_tower_blocks_ln_post_gpu`, which the whisper encode asks ahead of its CPU block loop and post-norm - a decline asks no second seat, its driver's blocks seat declining the same way; both tower drivers fill it, the post-norm one more row pass in the blocks' command buffer; the seat serves the family whose post-norm follows the blocks with nothing between, whisper, while the -families that pool before it, and ultravox, keep the CPU post-norm). The direction is forced: the +families that pool before it, and ultravox, take the blocks-with-tail seat +`register_tower_blocks_tail_gpu`, which `audio_encode` asks ahead of the blocks and the CPU +projector tail and which answers what it served - nothing, the blocks, or the blocks and the tail +(`TowerTailServed`); the Metal driver fills it). The direction is forced: the driver requires the family file for its types, so the family cannot require the driver back. A box with no driver leaves the slot empty and the CPU form runs. A seat taken over a filled slot (a test's stub through @@ -203,7 +213,7 @@ and serve counters are a `TtsGpuSeats` of `dasllama_tts_types.das`) gives the di `unregister` - one level: the record keeps the registration it displaced, not a stack of them - so the driver's seats survive the test. -Every hook answers "declined" in its own return - `false` for the block hooks and qwen3a's mel and +Every hook answers "declined" in its own return - `false` for the block hooks and the whisper mel and qwen3a's front hooks, `-1` for the hooks that return a row count (gemma4a's whole chunk, canary's front) - so a decline is a fallback, never an outage, and the CPU form stays the reference. A family calls its hook on either lane: the Metal driver declines the q8 encoder of every family but the diff --git a/modules/dasLLAMA/dasllama/dasllama_audio.das b/modules/dasLLAMA/dasllama/dasllama_audio.das index 44fd92d99f..9bae21c710 100644 --- a/modules/dasLLAMA/dasllama/dasllama_audio.das +++ b/modules/dasLLAMA/dasllama/dasllama_audio.das @@ -207,7 +207,7 @@ def log_mel_chunks(samples : array | #; n_mel : int64) : array let head = int(AUDIO_N_FFT / 2l) // reflect pad let tail = int(AUDIO_SAMPLE_RATE * 30l) + head // 30 s zeros + 200 let n_padded = head + n_base + tail - var padded : array + var inscope padded : array padded |> reserve(n_padded) padded |> resize(n_padded) // resize zero-fills for (i in range(n_in)) { @@ -220,12 +220,12 @@ def log_mel_chunks(samples : array | #; n_mel : int64) : array let n_fft = AUDIO_N_FFT let n_bins = n_fft / 2l + 1l let n_len = (int64(n_padded) - n_fft) / AUDIO_HOP + 1l - let hann <- build_hann_window(n_fft) - let twiddles <- build_dft_twiddles(n_fft) - let filters <- build_mel_filterbank(n_mel, n_fft, AUDIO_SAMPLE_RATE) + var inscope hann <- build_hann_window(n_fft) + var inscope twiddles <- build_dft_twiddles(n_fft) + var inscope filters <- build_mel_filterbank(n_mel, n_fft, AUDIO_SAMPLE_RATE) // windowed frames [n_len × n_fft] - var xw : array + var inscope xw : array xw |> reserve_resize(int(n_len * n_fft)) for (i in range64(n_len)) { let off = i * AUDIO_HOP @@ -234,45 +234,22 @@ def log_mel_chunks(samples : array | #; n_mel : int64) : array } } - // DFT as one GEMM: [n_len × n_fft] · [n_fft × 2·n_bins] -> interleaved re/im - var reim : array - reim |> reserve(int(n_len * n_bins * 2l)) - reim |> resize(int(n_len * n_bins * 2l)) // gemm accumulates: starts zeroed - gemm_f32(unsafe(addr(reim[0])), unsafe(addr(xw[0])), unsafe(addr(twiddles[0])), - n_len, n_fft, n_bins * 2l) - - // power spectra -> mel (float products, double sum in bin order), log10 - let mel_floor = 5.960464477539063e-08lf // 2^-24, the fixed filterbank floor - let log10_e = 0.43429448190325176lf - var mel : array - mel |> reserve(int(n_mel * n_len)) - mel |> resize(int(n_mel * n_len)) // mel-major [n_mel][n_len] - var power : array - power |> resize(int(n_bins)) - for (i in range64(n_len)) { - for (k in range64(n_bins)) { - let re = reim[i * n_bins * 2l + k * 2l] - let im = reim[i * n_bins * 2l + k * 2l + 1l] - power[k] = re * re + im * im - } - for (m in range64(n_mel)) { - var sum = 0.0lf - for (k in range64(n_bins)) { - sum += double(power[k] * filters[m * n_bins + k]) - } - mel[m * n_len + i] = float(log(max(sum, mel_floor)) * log10_e) - } + var inscope wm = WhisperMel(n_mel = n_mel, n_len = n_len, n_len_org = n_len) // mel-major [n_mel][n_len] + wm.data |> reserve(n_mel * n_len) + wm.data |> resize(n_mel * n_len) + if (!whisper_mel_gpu(xw, n_len, n_fft, n_bins, n_mel, twiddles, filters, wm)) { + log_mel_spectrum_cpu(xw, twiddles, filters, n_len, n_fft, n_bins, n_mel, wm.data) } // whisper-style dynamic clamp + scale, GLOBAL over the padded spectrogram; the clamp value // round-trips through float before the (x+4)/4 double math - the round-trip is what keeps // the mel plane bit-exact against the reference var mmax = -1e20lf - for (v in mel) { + for (v in wm.data) { mmax = max(mmax, double(v)) } mmax -= 8.0lf - for (v in mel) { + for (v in wm.data) { let clamped = double(v) < mmax ? float(mmax) : v v = float((double(clamped) + 4.0lf) / 4.0lf) } @@ -285,7 +262,7 @@ def log_mel_chunks(samples : array | #; n_mel : int64) : array c.data |> resize(int(n_mel * AUDIO_CHUNK_FRAMES)) for (m in range64(n_mel)) { for (i in range64(AUDIO_CHUNK_FRAMES)) { - c.data[m * AUDIO_CHUNK_FRAMES + i] = mel[m * n_len + off + i] + c.data[m * AUDIO_CHUNK_FRAMES + i] = wm.data[m * n_len + off + i] } } chunks |> emplace(c) @@ -294,6 +271,53 @@ def log_mel_chunks(samples : array | #; n_mel : int64) : array return <- chunks } +//! The whisper preprocessor's spectrum on the CPU: the DFT as one GEMM over the windowed frames `xw`, the power spectra, +//! the mel sums in bin order and their log10 into the mel-major `mel` [n_mel][n_len] - what the mel seat serves on a device. +def private log_mel_spectrum_cpu(xw, twiddles, filters : array; n_len, n_fft, n_bins, n_mel : int64; var mel : array) { + // DFT as one GEMM: [n_len × n_fft] · [n_fft × 2·n_bins] -> interleaved re/im + var inscope reim : array + reim |> reserve(n_len * n_bins * 2l) + reim |> resize(n_len * n_bins * 2l) // gemm accumulates: starts zeroed + gemm_f32(unsafe(addr(reim[0])), unsafe(addr(xw[0])), unsafe(addr(twiddles[0])), + n_len, n_fft, n_bins * 2l) + + // power spectra -> mel (float products, double sum in bin order), log10 + let mel_floor = 5.960464477539063e-08lf // 2^-24, the fixed filterbank floor + let log10_e = 0.43429448190325176lf + var inscope power : array + power |> resize(n_bins) + for (i in range64(n_len)) { + for (k in range64(n_bins)) { + let re = reim[i * n_bins * 2l + k * 2l] + let im = reim[i * n_bins * 2l + k * 2l + 1l] + power[k] = re * re + im * im + } + for (m in range64(n_mel)) { + var sum = 0.0lf + for (k in range64(n_bins)) { + sum += double(power[k] * filters[m * n_bins + k]) + } + mel[m * n_len + i] = float(log(max(sum, mel_floor)) * log10_e) + } + } +} + +//! the GPU mel hook: DFT, power and log10 mel off the CPU-windowed `xw` frames into `m.data` (mel-major, no clamp) +typedef WhisperMelGpuFn = function<(xw : array; n_len, n_fft, n_bins, n_mel : int64; + twiddles, filters : array; var m : WhisperMel) : bool> +var private g_mel_gpu : WhisperMelGpuFn +var private g_mel_gpu_set = false + +[arch(at="../ARCHITECTURE_MEDIA.md#tower-gpu-hook")] +def register_whisper_mel_gpu(fn : WhisperMelGpuFn) { + g_mel_gpu = fn + g_mel_gpu_set = true +} + +//! the mel seat asked: true where a driver landed the log10 mel in `m.data`, false with no seat or on its decline +def whisper_mel_gpu(xw : array; n_len, n_fft, n_bins, n_mel : int64; twiddles, filters : array; var m : WhisperMel) : bool + => g_mel_gpu_set && invoke(g_mel_gpu, xw, n_len, n_fft, n_bins, n_mel, twiddles, filters, m) + //! Full-length whisper-flavor log-mel spectrogram: total frame count (audio + the 30 s //! zero tail), the real-audio frame count (``n_len_org`` = the driver's seek_end), and //! mel-major ``[n_mel][n_len]`` data. @@ -1088,13 +1112,14 @@ def private audio_tower_mlp_shape(m : GGUFMeta; kind : AudioProjKind; mm1_in : i return (mm1_out = mm1_out, proj_dim = mm2_in > 0l ? m.tensors[i2].n_elem / mm2_in : 0l) } -//! The decoder width a whisper-class audio mmproj (qwen2a, qwen2.5o, ultravox, voxtral) projects to; 0 for a missing -//! file, a file that is no GGUF, or an mmproj this tower does not serve. Never panics: the seams gate an audio arm on it. -def audio_tower_probe_proj_dim(path : string) : int64 { - return 0l if (path |> ends_with(".dlim")) +//! What a whisper-class audio mmproj (qwen2a, qwen2.5o, ultravox, voxtral) says of itself, read off its metadata: the +//! decoder width it projects to and its projector kind. `dim` 0 for a missing file, a file that is no GGUF, or an mmproj +//! this tower does not serve. Never panics. +def private audio_tower_probe(path : string) : tuple { + var res = (dim = 0l, kind = AudioProjKind.qwen2a) + return res if (path |> ends_with(".dlim")) let f = fopen(path, "rb") - return 0l if (f == null) - var dim = 0l + return res if (f == null) fmap(f) $(var bytes : array#) { if (long_length(bytes) < 4l || rd_u32(bytes, 0l) != 0x46554747u) { return @@ -1105,15 +1130,30 @@ def audio_tower_probe_proj_dim(path : string) : int64 { || !gguf_has(m, "clip.audio.embedding_length") || gguf_find_tensor(m, "a.position_embd.weight") < 0) { return } + res.kind = kind if (kind == AudioProjKind.qwen2a) { - dim = gguf_has(m, "clip.audio.projection_dim") ? gguf_int(m, bytes, "clip.audio.projection_dim") : 0l + res.dim = gguf_has(m, "clip.audio.projection_dim") ? gguf_int(m, bytes, "clip.audio.projection_dim") : 0l } else { let stack = gguf_has(m, "clip.audio.projector.stack_factor") ? gguf_int(m, bytes, "clip.audio.projector.stack_factor") : 1l - dim = audio_tower_mlp_shape(m, kind, gguf_int(m, bytes, "clip.audio.embedding_length") * stack).proj_dim + res.dim = audio_tower_mlp_shape(m, kind, gguf_int(m, bytes, "clip.audio.embedding_length") * stack).proj_dim } } fclose(f) - return dim + return res +} + +//! The decoder width a whisper-class audio mmproj projects to; 0 where the tower does not serve the file. Never panics: +//! the seams gate an audio arm on it. +def audio_tower_probe_proj_dim(path : string) : int64 => audio_tower_probe(path).dim + +//! Whether a tower's soft tokens sit in the user turn with no span marker around them: the ultravox projector pairs with +//! a stock decoder whose template and vocab know no audio marker +def audio_span_bare(kind : AudioProjKind) : bool => kind == AudioProjKind.ultravox + +//! `audio_span_bare` for an mmproj on disk; false where the tower does not serve the file +def audio_tower_probe_span_bare(path : string) : bool { + let p = audio_tower_probe(path) + return p.dim > 0l && audio_span_bare(p.kind) } //! Mint a served tower from a staged read WITHOUT touching disk — the off-rail control for suites. @@ -1439,6 +1479,48 @@ def audio_encode_blocks_ln_post(t : AudioTower; var s : EncoderState; mel : MelC return false } +//! What the blocks-with-tail seat served: nothing, the blocks alone (`s.x` holds their rows), or the blocks and the projector +//! tail behind them (`s.out` holds the soft tokens) +enum TowerTailServed { + declined + blocks + tail +} + +typedef TowerBlocksTailGpuFn = function<(t : AudioTower; var s : EncoderState; npos : int64) : TowerTailServed> +var private g_tw_blocks_tail : TowerBlocksTailGpuFn +var private g_tw_blocks_tail_set = false + +[arch(at="../ARCHITECTURE_MEDIA.md#tower-gpu-hook")] +def register_tower_blocks_tail_gpu(fn : TowerBlocksTailGpuFn) { + g_tw_blocks_tail = fn + g_tw_blocks_tail_set = true +} + +//! the stem, then the blocks with the projector tail behind them where the tail seat serves both (true: `s.out` holds the soft +//! tokens); false leaves the blocks' rows in `s.x` for the CPU tail, whichever chain ran them +def private audio_encode_blocks_tail(t : AudioTower; var s : EncoderState; mel : MelChunk) : bool { + audio_encode_stem(t, s, mel) + let tt = t.n_ctx + if (!g_tw_blocks_tail_set) { + encoder_blocks(t, s, tt) + return false + } + dispatch_phase_encode(tt) + var served = TowerTailServed.declined + if (empty(g_audio_ref_dir)) { + let te = ref_time_ticks() + served = invoke(g_tw_blocks_tail, t, s, tt) + if (served != TowerTailServed.declined) { + asr_prof_add("enc.blocks.gpu", te) + } + } + if (served == TowerTailServed.declined) { + encoder_blocks_cpu(t, s, tt) + } + return served == TowerTailServed.tail +} + var private g_tw_blocks_ln_post : TowerBlocksGpuFn var private g_tw_blocks_ln_post_set = false @@ -1595,11 +1677,15 @@ def private rms_rows_blob_at(var x : array; wp : float const?; dim, rows //! Stacking floors rows — ragged tail positions are dropped, not zero-padded. [hot_path] def audio_encode(t : AudioTower; var s : EncoderState; mel : MelChunk) { - audio_encode_blocks(t, s, mel) + let rows = audio_tokens_per_chunk(t) + if (audio_encode_blocks_tail(t, s, mel)) { + diff_stage("out", s.out, rows * t.proj_dim) + return + } trace_tag(TRACE_TAG_ENC_PROJ) + let t_tail = ref_time_ticks() let tt = t.n_ctx let d = t.d_model - let rows = audio_tokens_per_chunk(t) if (t.proj_kind == AudioProjKind.qwen2a) { // avg-pool pairs over time (BEFORE the post-LN, not after), then ln_post @@ -1636,6 +1722,7 @@ def audio_encode(t : AudioTower; var s : EncoderState; mel : MelChunk) { gelu_erf(s.ph, rows * t.mm1_out) tw_mm_rq(t, s.out, t.mm2_w_off, s.ph, s.xqi, s.xsi, t.mm1_out, t.proj_dim, rows) } + asr_prof_add("enc.tail", t_tail) trace_tag(0) // clear: the decoder's dispatches must not inherit the projector's color diff_stage("out", s.out, rows * t.proj_dim) } diff --git a/modules/dasLLAMA/dasllama/dasllama_audio_embedder.das b/modules/dasLLAMA/dasllama/dasllama_audio_embedder.das index bda8a9de9a..381c2ac673 100644 --- a/modules/dasLLAMA/dasllama/dasllama_audio_embedder.das +++ b/modules/dasLLAMA/dasllama/dasllama_audio_embedder.das @@ -7,6 +7,7 @@ module dasllama_audio_embedder shared public require dasllama/dasllama_lint public require dasllama/dasllama_gemma4a public // re-exported: a requirer that carries the union must see the family to delete it require dasllama/dasllama_audio public // the same for the whisper-class tower (qwen2a, qwen2.5o, ultravox, voxtral) +require dasllama/dasllama_tower // asr_prof_add: the stage clock require dasllama/dasllama_image // image_family_tag — a prepared .dlim routes on its baked tag require strings @@ -67,6 +68,13 @@ def audio_probe_proj_dim(path : string) : int64 { return g4a > 0l ? g4a : audio_tower_probe_proj_dim(path) } +//! Whether the mmproj at `path` splices its soft tokens with no span marker (see `audio_span_bare`); false for a file +//! no audio family serves. Never panics. +def audio_probe_span_bare(path : string) : bool => audio_tower_probe_span_bare(path) + +//! Whether the carried family splices its soft tokens with no span marker around them +def audio_span_bare(e : AudioEmbedder) : bool => e.kind == AudioKind.tower && audio_span_bare(e.tower.proj_kind) + //! Load an audio mmproj (gguf, or a prepared ``.dlim`` routed on its baked family tag) as //! whichever family it is — probed from the file, never from a knob. Panics by name on a file //! no audio family serves; probe first where absence is an answer. @@ -93,7 +101,9 @@ def private tower_encode_all(t : AudioTower; var s : AudioState; samples : array s.enc <- make_encoder_state(t) s.enc_made = true } + let t_mel = ref_time_ticks() var inscope chunks <- log_mel_chunks(samples, t.n_mel) + asr_prof_add("enc.mel", t_mel) let rows = audio_tokens_per_chunk(t) out |> reserve(length(out) + length(chunks) * int(rows * t.proj_dim)) for (chunk in chunks) { diff --git a/modules/dasLLAMA/dasllama/dasllama_canary.das b/modules/dasLLAMA/dasllama/dasllama_canary.das index eb2b84ca4d..8a72f6b712 100644 --- a/modules/dasLLAMA/dasllama/dasllama_canary.das +++ b/modules/dasLLAMA/dasllama/dasllama_canary.das @@ -1235,11 +1235,11 @@ def canary_transcribe(dec : Model; enc : CanaryEncoder; var s : CanaryAsrSession let im_start = special_id(dec, "<|im_start|>") let im_end = special_id(dec, "<|im_end|>") // = the text EOS (151645) // SALM "qwen" prompt: <|im_start|>user\nTranscribe the following: