From ad2baa7539d6256642bd231a09180ab6d8811187 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Sun, 16 Aug 2026 16:49:28 +0200 Subject: [PATCH 01/14] =?UTF-8?q?[WIP]=20Add=20VoxCPM=20v1=20=E2=80=94=20l?= =?UTF-8?q?ightweight=20VoxCPM=20TTS=20support=20(0.5B=20/=201.5B)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CMakeLists.txt | 15 + docs/reports/voxcpm1_pr.md | 243 ++++++++++ docs/tts.md | 36 ++ include/engine/models/voxcpm2/assets.h | 3 +- include/engine/models/voxcpm2/loader.h | 1 + model_specs/voxcpm1.json | 118 +++++ src/models/voxcpm2/assets.cpp | 647 ++++++++++++++++++++++++- src/models/voxcpm2/generator.cpp | 65 ++- src/models/voxcpm2/loader.cpp | 115 ++++- src/models/voxcpm2/minicpm.cpp | 16 +- 10 files changed, 1220 insertions(+), 39 deletions(-) create mode 100644 docs/reports/voxcpm1_pr.md create mode 100644 model_specs/voxcpm1.json diff --git a/CMakeLists.txt b/CMakeLists.txt index 80dcdd5b..01c8964a 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -696,6 +696,21 @@ audiocpp_add_model(voxcpm2 engine::models::voxcpm2::make_voxcpm2_loader ) +audiocpp_add_model(voxcpm1 + SOURCES + src/models/voxcpm2/assets.cpp + src/models/voxcpm2/audiovae.cpp + src/models/voxcpm2/generator.cpp + src/models/voxcpm2/loader.cpp + src/models/voxcpm2/minicpm.cpp + src/models/voxcpm2/session.cpp + src/models/voxcpm2/tokenizer_text.cpp + INCLUDES + engine/models/voxcpm2/loader.h + LOADERS + engine::models::voxcpm2::make_voxcpm1_loader +) + audiocpp_add_model(vibevoice SOURCES src/models/vibevoice/assets.cpp diff --git a/docs/reports/voxcpm1_pr.md b/docs/reports/voxcpm1_pr.md new file mode 100644 index 00000000..a5d3b70e --- /dev/null +++ b/docs/reports/voxcpm1_pr.md @@ -0,0 +1,243 @@ +# PR: VoxCPM1 — lightweight VoxCPM TTS support (0.5B / 1.5B) + +> **Status: first porting attempt — runtime works end-to-end, output quality NOT yet acceptable** +> +> The port successfully loads and runs all three VoxCPM v1 GGUF variants (anchors pass, graphs +> execute, WAV files are produced at the correct sample rates/durations with active signal). +> **Known issue:** the generated audio is almost pure noise with only a faint trace of human +> voice. The pipeline is correct mechanically, but output quality requires further debugging +> (hypotheses and investigation plan in [Known issue](#known-issue-noisy-output)). + +--- + +## 1. Overview + +This PR adds support for the **OpenBMB VoxCPM v1** family of lightweight TTS models to +audio.cpp, reusing the existing and already-released `voxcpm2` model tree: + +| Model | Params | Output sample rate | GGUF file | +|---|---|---|---| +| VoxCPM-0.5B | 0.5B | **16 kHz** | `voxcpm-0.5b-q8_0-audiovae-f16.gguf` | +| VoxCPM-1.5B | 1.5B | **44.1 kHz** | `voxcpm1.5-q8_0.gguf` | +| VoxCPM-1.5B | 1.5B | **44.1 kHz** | `voxcpm1.5-q4_k-audiovae-f16.gguf` | + +The three models are architecturally **different variants** (they cannot share one config): + +- **0.5B:** VAE encoder 128 / decoder 1536, encoder_rates `[2,5,8,8]`, decoder_rates + `[8,8,5,2]`, patch_size 2, residual_lm 6 layers, encoder/dit 4 layers, 16 kHz, max_len 4096. +- **1.5B:** VAE encoder 64 / decoder 2048, encoder_rates `[2,3,6,7,7]`, decoder_rates + `[7,7,6,3,2]`, patch_size 4, residual_lm 8 layers, encoder/dit 8 layers, 44.1 kHz, max_len 8192. + +Since the v1 GGUFs store a different tensor convention than v2 (folded AudioVAE weights, no +`weight_v`/`weight_g` split, no `sr_cond_model` tensors, `voxcpm` architecture name), the port +wraps the v2 loader with a GGUF tensor-adaptation layer and adds `config.v1`-guarded branches +in the generator, mirroring the reference implementation (`VoxCPM.cpp`). + +--- + +## 2. Porting activities + +1. **Regenerated the 0.5B `config.json` from the GGUF metadata** — the previously shipped + sidecar was wrong on ~8 axes (patch, residual_lm/encoder/dit layer counts, VAE dims and + rates, sample rate 44.1 kHz vs the actual 16 kHz, max_len). +2. **Diagnosed the v1 GGUF conventions** (tensor dump + reference converter analysis): + - AudioVAE conv weights are stored **already folded** (weight-norm folded), with no + `weight_v`/`weight_g` decomposition and no `sr_cond_model.*` tensors. + - GGUF file dims == ggml `ne` order; the v1 GGUFs carry **no** `audiocpp.tensor_shapes` + override metadata (v2 does), so the adapter must present shapes itself. + - The 1.5B **Q8_0** file stores VAE conv weights **2D-flattened** (`{out, in·k}`, kernel + folded into dim1) while Q4_K and 0.5B store 3D `{out, in, k}` — both must load. +3. **Designed the identity-fold adapter** (see §4) so the existing `load_vae_weights` loader + works unchanged against folded v1 weights byte-for-byte. +4. **Mirrored the reference generator math** for the no-fusion (no `fusion_concat_proj`) + case: elementwise-add fusion inputs, elementwise-add dit-mu, and a real residual_lm + autoregressive step. +5. **Set up per-variant model directories** (`VoxCPM1-GGUF/` for 0.5B, `VoxCPM1.5-GGUF/` + for 1.5B) each with a config regenerated from its own GGUF metadata + tokenizer sidecars, + and updated `model_specs/voxcpm1.json` package targets accordingly. +6. **Verified end-to-end runs** for all three GGUFs on the CPU backend (see + [Validation](#6-validation-performed)). + +--- + +## 3. Changes per file + +| File | Change | +|---|---| +| `CMakeLists.txt` | Added `audiocpp_add_model(voxcpm1 ...)` reusing the 7 voxcpm2 sources; registers `engine::models::voxcpm2::make_voxcpm1_loader`. | +| `include/engine/models/voxcpm2/loader.h` | Declared `make_voxcpm1_loader()`. | +| `include/engine/models/voxcpm2/assets.h` | Added `VoxCPM2Config::v1 = false`; `load_voxcpm2_assets()` now takes `bool is_v1`. | +| `src/models/voxcpm2/loader.cpp` | Added `VoxCPM1Loader` (family `"voxcpm1"`), `load_voxcpm1_model()`, `make_voxcpm1_loader()`, `metadata_v1` / `capabilities_v1` / `cli_v1`. Offline-only TTS + speaker-reference clone, `text_prefix` policy, GGUF via `load_voxcpm2_assets(path, is_v1=true)`. | +| `src/models/voxcpm2/assets.cpp` | Added `TransformingTensorSource` v1 adapter (biggest chunk):
• v1→v2 tensor-name rename map (`token_embd.weight`→`base_lm.embed_tokens.weight`, gguf `blk.N.*`→`base_lm.layers.N.*` / `feat_encoder.encoder.layers.*` / `feat_decoder.estimator.decoder.layers.*` / `residual_lm.layers.*`, `attn_norm`→`input_layernorm`, `ffn_norm`→`post_attention_layernorm`, `attn_*`→`self_attn.*_proj`, `ffn_*`→`mlp.*_proj`, `time_mlp.*` (preserving `.linear_N`), `output_norm.weight`→`base_lm.norm.weight`, projection/fsq/stop mappings)
• **Folded weight-norm synthesis**: for every `audio_vae.*.weight` conv, `X.weight_v` → folded tensor data as-is, `X.weight_g` → per-row L2 norms (identity fold, see §4)
• Identity `decoder.sr_cond_model.{2..5}.scale_embed.weight` (ones) / `.bias_embed.weight` (zeros) since v1 GGUFs carry no SR-conditioning tensors
• Synthesized missing v1 tensors (`feat_encoder.scale_embed/bias_embed`, `feat_encoder.fc_logvar`, `feat_encoder.diag`, `feat_encoder.merge`, `token_embd.extra_bias`, `fusion_concat_proj.weight/bias`, `stop_proj.weight`, `stop_head.weight`)
• Rank-tolerant `require_f32` (accept element-count-equal, shape-different fetches — handles 2D-flattened convs and `{C,1}` alphas) + relaxed-rank VAE weight_v anchors for v1
• `has_tensor` / `require_metadata` / `require_tensor_data` folded + synthesized lookups
• **Anchor fix:** `encoder.fc_mu.weight_v` now uses computed encoder-in (`encoder_dim << #rates` = 2048), not `decoder_dim` (1536) | +| `src/models/voxcpm2/generator.cpp` | • v1 fusion guard: residual input = `AddModule(lm_hidden, current_embed)` / `AddModule(fsq, current_embed)` instead of concat+linear (matches reference `build_residual_fusion_input`)
• Added `add_dit_mu()` helper; v1 `mu` = elementwise add of `current_lm_dit_hidden + residual_dit_hidden` (matches reference `build_dit_mu`, `mu_dim = hidden·(fusion?2:1)`, v1 → hidden)
• CFM `mu` size check is now v1-aware (`hidden_dim * (v1 ? 1 : 2)`)
• v1 decode loop runs `residual_lm_.run_step(next_projected.residual_input).hidden` (the earlier `fsq_lm_dit_hidden` shortcut removed — v1 GGUFs have 6/8 residual_lm layers) | +| `src/models/voxcpm2/minicpm.cpp` | Prompt-prefill graph: v1 `residual_input` = `AddModule(lm_hidden, masked_current)` instead of concat+linear; residual_lm always runs (previously the concat path would have produced a wrong-dimension residual input for v1). | +| `model_specs/voxcpm1.json` | Package targets: `voxcpm1_0.5b_q8_0` → `VoxCPM1-GGUF`; `voxcpm1_1.5b_q4_k` and `voxcpm1_1.5b_q8_0` → `VoxCPM1.5-GGUF` (per-variant config/tokenizer). | +| `docs/tts.md` | Added VoxCPM1 section + TOC entry (usage, options, sample-rate notes). | +| `README.md` | Added `voxcpm1` row to the supported-model table. | +| `docs/reports/voxcpm1_port_status.md` | Port status log (analysis, decisions, timestamps, remaining tasks). | +| `models/VoxCPM1-GGUF/config.json` | **Regenerated** from 0.5B GGUF metadata. | +| `models/VoxCPM1.5-GGUF/config.json` | **New**, regenerated from 1.5B GGUF metadata. | +| `models/VoxCPM1.5-GGUF/tokenizer.json` (+config/special tokens) | Copied from 0.5B dir (same 73,448-vocab BPE tokenizer). | + +--- + +## 4. Key design: the identity-fold adapter + +The v1 GGUF (OpenBMB reference converter) stores AudioVAE conv weights **already folded** +(`weight = weight_g · weight_v / ‖weight_v‖`), with no `weight_v`/`weight_g` split, while +`audiovae.cpp` requests the decomposed names directly via `require_f32`. The adapter solves +this without touching the VAE loader: + +``` +X.weight_v := folded GGUF tensor data (as-is) +X.weight_g := per-row L2 norms of the folded tensor, + computed with the loader's own row grouping + (groups = expected_shape.front(), inner = elements/groups) +``` + +Because `fold_weight_norm` multiplies row `d0` by `weight_g[d0] / ‖row d0‖ = 1`, the loader +output equals the GGUF data **byte-for-byte** — an exact identity, with no layout drift +relative to the reference runtime's consumption of the same bytes. The same mechanism works +for 3D `{out, in, k}` and 2D-flattened `{out, in·k}` conversions (element counts must match; +ranks may differ, covered by rank-tolerant `require_f32` + relaxed-rank anchors). + +--- + +## 5. Usage + +### Build + +```bash +scripts/build_linux.sh --backend cpu --target audiocpp_cli +# or, with the standard full model set: +cmake -S . -B build/linux-cpu-release -DCMAKE_BUILD_TYPE=Release +cmake --build build/linux-cpu-release --target audiocpp_cli -j 8 +``` + +### Run — 0.5B (16 kHz output) + +```bash +build/linux-cpu-release/bin/audiocpp_cli \ + --task tts --family voxcpm1 \ + --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf \ + --backend cpu --text "Hello from VoxCPM1." --out out.wav +``` + +### Run — 1.5B (44.1 kHz output) + +```bash +build/linux-cpu-release/bin/audiocpp_cli \ + --task tts --family voxcpm1 \ + --model models/VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf \ + --backend cpu --text "Hello from VoxCPM1." --out out.wav +``` + +### Options + +| Option | Values | Default | Meaning | +|---|---:|---:|---| +| `--task` | `tts` | required | Task kind. | +| `--family` | `voxcpm1` | auto-detect | Selects the v1 loader. | +| `--backend` | `cpu`, `cuda`, `vulkan`, `metal`, `hip`, `best` | `best` | Backend. | +| `--voice-ref` | WAV path | not set | Reference speaker audio (clone). | +| `--max-tokens` | integer | `4096` | Maximum generated AR tokens. | +| `--num-inference-steps` | integer | `10` | Flow-matching steps. | +| `--guidance-scale` | float | `2.0` | CFG strength. | +| `--session-option voxcpm1.mem_saver=true\|false` | bool | `false` | Tighter graph workspaces + release request graphs after completion. | +| `--session-option voxcpm1.prompt_cache_slots=` | integer | `1` | Prompt/prompt-audio embedding cache slots. | +| `--text-chunk-mode` | `default`, `tag_aware`, `japanese`, `endline` | `tag_aware` | Long-form chunking mode. | + +--- + +## 6. Validation performed + +- **Load + anchors:** all three GGUFs pass `validate_weight_anchors` and + `load_vae_weights`/`load_model_weights` on CPU. This includes the 0.5B 3D convs, the 1.5B + Q4_K 3D convs, and the 1.5B Q8_0 2D-flattened convs. +- **End-to-end:** `--task tts` completes for all three models; outputs are written as WAV at + the correct sample rate (16 kHz for 0.5B, 44.1 kHz for 1.5B) with active signal and + speech-plausible duration/envelope. +- **Regression:** the released voxcpm2 path is untouched (guard style `config.v1`, v2 default + `false`); voxcpm2 was not re-benchmarked but the changed code paths are v1-gated or + v1/v2-neutral. + +> ⚠️ **Quality caveat:** "end-to-end completes" does **not** mean the output is usable yet. +> See the known issue below — the audio is predominantly noise. + +--- + +## 7. Supported modes + +| Mode | Supported | Notes | +|---|---|---| +| **Offline TTS** | ✅ implemented | Default and only advertised mode. | +| **Streaming** | ❌ not implemented for v1 | `voxcpm1` advertises offline-only. Streaming is a v2 capability; it has not been validated (or enabled) for v1. | +| **Voice clone** | ⚠️ surface present | Speaker-reference options are advertised (`--voice-ref`), but quality is gated on the same known issue as plain TTS. | + +--- + +## Known issue: noisy output + +**Symptom.** Generated v1 voices are almost pure noise with a little human voice mixed in — +the signal is dominated by broadband/noise content. This affects all three GGUFs. + +**What is confirmed working.** Model loading, tensor adaptation, anchor validation, graph +construction, graph execution, and WAV output plumbing are all correct (no crashes, no +shape/size errors, correct sample rates and durations). The failure is therefore in the +**numerics of synthesis**, i.e. the audio content itself. + +**Most likely causes (in rough priority order).** + +1. **Weight data interpretation** — the identity fold preserves bytes, but if some AudioVAE + layer's storage layout (depthwise vs pointwise handling, 2D-flattened Q8_0, transposed + decoder `{in,out,k}` conventions, per-group row ordering of `weight_g`) differs from what + `ggml_conv_1d` / `conv_transpose` expects, the VAE decoder outputs garbage while loading + still "succeeds" (element counts match). +2. **Synthesized tensor semantics** — `feat_encoder.scale_embed/bias_embed`, `merge`, + `diag`, `extra_bias`, `fusion_concat_proj`, `stop_*`, and the identity `sr_cond_model` + tensors were synthesized with plausible but unverified semantics; if any is required to be + learned/zero-`scale` (or absent entirely in the reference runtime), the feature stream + feeding the LM/CFM is wrong. +3. **Graph parity vs the reference** — fusion = add and dit-mu = add were taken from + reference `build_residual_fusion_input`/`build_dit_mu`, but adjacent details (masking, + slice indices, position ids, prompt handling, FSQ rounding, CFM conditioning inputs, + ordering of `nn.Module` sub-blocks in the residual_lm stack) may differ. +4. **Sample-rate/codec mismatch** — 0.5B output asserted 16 kHz but the reference may expect + a specific internal feature rate; patch_size/feat_dim interplay (2·64 vs 4·64) feeding the + CFM estimator could be off by a constant factor, producing frozen-then-noisy patches. +5. **Quantization path** — the 1.5B Q8_0 GGUF quantizes the VAE itself (2D-flattened); + dequantized values feed `require_f32`, but a transpose or block-order mismatch would + corrupt every activation. + +**Debugging plan (next iteration).** + +- [ ] Port a small deterministic parity harness: run the same prompt through the reference + `VoxCPM.cpp` and audio.cpp, dump intermediate tensors (lm hidden, residual hidden, + CFM mu, VAE latent, decoder output) at each major stage, and diff numerically. +- [ ] Verify `encoder.fc_mu` / `decoder.model.{0,1,N}` folded data against the Python + reference weights with a strict per-element comparison on non-quantized tensors + (f16 VAE files), including row-grouping of `weight_g`. +- [ ] Check whether the reference runtime actually instantiates `sr_cond_model` and + `feat_encoder` synthesizable blocks for v1; remove or zero-scale any block the + reference does not run. +- [ ] Experimentally force one suspected block to a no-op (e.g. sr_cond identity, merge + zeros, scale_embed 0/1) and measure whether noise level drops. +- [ ] Validate CFM mu dimension/conditioning against the reference expectation for + `patch=2` (0.5B) and `patch=4` (1.5B). +- [ ] After the numerics match, run a human listening + loudness/spectral sanity check + (the current output has a spectral envelope consistent with noise + faint voice). + +--- + +## 8. Remaining tasks + +- [x] Loader registration, tensor adaptation, generator v1 branches, configs, model spec +- [x] End-to-end execution for 0.5B Q8_0, 1.5B Q4_K, 1.5B Q8_0 +- [ ] **Fix noisy output (known issue above) — top priority** +- [ ] Numerical parity harness vs `VoxCPM.cpp` reference (stage-by-stage tensor diff) +- [ ] `tests/voxcpm1/` automated path tests mirroring `tests/voxcpm2/` +- [ ] WebUI catalog entry (`webui/configs/models_catalog.json`) +- [ ] `docs/gguf.md` support-table entry +- [ ] CUDA-backend verification + RTF measurement (expect voxcpm2-like speedups) +- [ ] Streaming support for v1 (only meaningful after numerics are fixed) +- [ ] Commit + release packaging for `audio.cpp-gguf` (0.5B and 1.5B packages) \ No newline at end of file diff --git a/docs/tts.md b/docs/tts.md index 770b6f6f..e77ccd8f 100644 --- a/docs/tts.md +++ b/docs/tts.md @@ -14,6 +14,7 @@ | NeuTTS | `neutts` | `tts` | [NeuTTS](#neutts) | | OmniVoice | `omnivoice` | `tts` | [OmniVoice](#omnivoice), [full guide](models/omnivoice.md) | | PocketTTS | `pocket_tts` | `tts` | [PocketTTS](#pockettts) | +| VoxCPM1 | `voxcpm1` | `tts` | [VoxCPM1](#voxcpm1) | | VoxCPM2 | `voxcpm2` | `tts`, `vdes` | [VoxCPM2](#voxcpm2) | | Higgs Audio v3 TTS | `higgs_audio_tts` | `tts` | [Higgs Audio v3 TTS](#higgs-audio-v3-tts) | | Fish Audio S2 Pro | `fish_audio` | `tts` | [Fish Audio S2 Pro](#fish-audio-s2-pro) | @@ -399,6 +400,41 @@ audiocpp_cli --task tts --family pocket_tts --model models/pocket-tts --backend | `--text-chunk-size` | integer chars | `256` | Long-form chunk size. | | `--session-option pocket_tts.voice_state_cache_slots=` | integer slots | `4` | Prepared voice-state cache slots; set `0` to disable reuse. | +## VoxCPM1 + +VoxCPM1 supports offline TTS. It reuses the VoxCPM2 runtime tree with a GGUF tensor-adaptation layer that understands the OpenBMB folded AudioVAE weights, so the same `--family voxcpm1` path serves both the 16 kHz 0.5B model and the 44.1 kHz 1.5B variants. + +| Field | Value | +|---|---| +| Family | `voxcpm1` | +| Model directory | `models/VoxCPM1-GGUF` (0.5B), `models/VoxCPM1.5-GGUF` (1.5B) | +| Task | `tts` | +| Modes | `offline` | +| Languages | Model auto-handles supported languages | +| Voice input | Optional reference WAV | +| Built-in voices | Not exposed | + +Text to speech: + +```bash +audiocpp_cli --task tts --family voxcpm1 --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf --backend cpu --text "Hello from VoxCPM1." --out out.wav +``` + +1.5B variant (44.1 kHz output): + +```bash +audiocpp_cli --task tts --family voxcpm1 --model models/VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf --backend cpu --text "Hello from VoxCPM1." --out out.wav +``` + +| Option | Values | Default | Meaning | +|---|---:|---:|---| +| `--voice-ref` | WAV path | not set | Reference speaker audio. | +| `--session-option voxcpm1.mem_saver=true\|false` | bool | `false` | Use tighter graph workspaces and release MiniCPM/AudioVAE request graphs after completion to reduce resident VRAM. | +| `--session-option voxcpm1.prompt_cache_slots=` | integer | `1` | Prompt and prompt-audio embedding cache slots. Set to `0` to disable prompt caching. | +| `--max-tokens` | integer | `4096` | Maximum generated AR tokens. | +| `--num-inference-steps` | integer | `10` | Flow matching steps. | +| `--guidance-scale` | float | `2.0` | CFG strength. | + ## VoxCPM2 VoxCPM2 supports plain TTS, voice design, controllable voice cloning, and an ultimate-clone style that uses both prompt audio and transcript. The CLI expresses voice design with the same text convention as the upstream examples: put the voice/style description in parentheses at the start of `--text`. diff --git a/include/engine/models/voxcpm2/assets.h b/include/engine/models/voxcpm2/assets.h index 89fe156b..19581ed3 100644 --- a/include/engine/models/voxcpm2/assets.h +++ b/include/engine/models/voxcpm2/assets.h @@ -85,6 +85,7 @@ struct VoxCPM2Config { int64_t max_length = 8192; std::string device = "cuda"; std::string dtype = "bfloat16"; + bool v1 = false; }; struct VoxCPM2Assets { @@ -94,6 +95,6 @@ struct VoxCPM2Assets { std::shared_ptr audiovae_weights; }; -std::shared_ptr load_voxcpm2_assets(const std::filesystem::path & model_path); +std::shared_ptr load_voxcpm2_assets(const std::filesystem::path & model_path, bool is_v1); } // namespace engine::models::voxcpm2 diff --git a/include/engine/models/voxcpm2/loader.h b/include/engine/models/voxcpm2/loader.h index 4c588482..72f7f3c5 100644 --- a/include/engine/models/voxcpm2/loader.h +++ b/include/engine/models/voxcpm2/loader.h @@ -29,5 +29,6 @@ class VoxCPM2LoadedModel final : public runtime::ILoadedVoiceModel { std::unique_ptr load_voxcpm2_model(const std::filesystem::path &model_path); std::shared_ptr make_voxcpm2_loader(); +std::shared_ptr make_voxcpm1_loader(); } // namespace engine::models::voxcpm2 diff --git a/model_specs/voxcpm1.json b/model_specs/voxcpm1.json new file mode 100644 index 00000000..905bd1df --- /dev/null +++ b/model_specs/voxcpm1.json @@ -0,0 +1,118 @@ +{ + "family": "voxcpm1", + "display_name": "VoxCPM1", + "description": "OpenBMB VoxCPM 0.5B and 1.5B tokenizer-free TTS models with 24kHz output.", + "category": "tts", + "status": "supported", + "tasks": [ + "tts", + "clone" + ], + "modes": [ + "offline" + ], + "languages": [ + "zh", + "en", + "ja", + "ko" + ], + "capabilities": { + "clone": [ + "speaker_reference" + ] + }, + "runtime": { + "tags": [ + "gguf" + ] + }, + "ui": { + "recommended_package": "voxcpm1_0.5b_q8_0", + "tags": [ + "TTS", + "Clone", + "GGUF" + ], + "docs": [ + "docs/tts.md", + "docs/gguf.md" + ] + }, + "package_defaults": { + "download": { + "kind": "huggingface_snapshot", + "repo": "audio-cpp/audio.cpp-gguf", + "revision": "main", + "gated": false + } + }, + "packages": [ + { + "id": "voxcpm1_0.5b_q8_0", + "display_name": "VoxCPM 0.5B Q8_0 GGUF", + "default": true, + "format": "gguf", + "precision": "q8_0", + "target_directory": "VoxCPM1-GGUF", + "files": [ + "VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf", + "VoxCPM1-GGUF/config.json", + "VoxCPM1-GGUF/tokenizer.json", + "VoxCPM1-GGUF/tokenizer_config.json" + ], + "strip_prefix": "VoxCPM1-GGUF" + }, + { + "id": "voxcpm1_1.5b_q4_k", + "display_name": "VoxCPM 1.5B Q4_K GGUF", + "format": "gguf", + "precision": "q4_k", + "target_directory": "VoxCPM1.5-GGUF", + "files": [ + "VoxCPM1.5-GGUF/voxcpm1.5-q4_k-audiovae-f16.gguf", + "VoxCPM1.5-GGUF/config.json", + "VoxCPM1.5-GGUF/tokenizer.json", + "VoxCPM1.5-GGUF/tokenizer_config.json" + ], + "strip_prefix": "VoxCPM1.5-GGUF" + }, + { + "id": "voxcpm1_1.5b_q8_0", + "display_name": "VoxCPM 1.5B Q8_0 GGUF", + "format": "gguf", + "precision": "q8_0", + "target_directory": "VoxCPM1.5-GGUF", + "files": [ + "VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf", + "VoxCPM1.5-GGUF/config.json", + "VoxCPM1.5-GGUF/tokenizer.json", + "VoxCPM1.5-GGUF/tokenizer_config.json" + ], + "strip_prefix": "VoxCPM1.5-GGUF" + } + ], + "sources": [ + { + "format": "gguf", + "roots": { + "model": ".", + "weights": "$gguf" + }, + "files": { + "config": "model:config.json", + "tokenizer_config": "model:tokenizer_config.json", + "tokenizer_json": "model:tokenizer.json", + "special_tokens_map": "model:special_tokens_map.json" + }, +"tensors": { + "weights": { + "source": "weights:" + }, + "audiovae_weights": { + "source": "weights:" + } + } + } + ] +} diff --git a/src/models/voxcpm2/assets.cpp b/src/models/voxcpm2/assets.cpp index 7d4ae2c4..997f54ee 100644 --- a/src/models/voxcpm2/assets.cpp +++ b/src/models/voxcpm2/assets.cpp @@ -2,11 +2,18 @@ #include "engine/framework/model_spec/package.h" #include "engine/framework/assets/resource_bundle.h" +#include "engine/framework/assets/tensor_source.h" #include "engine/framework/io/config.h" #include "engine/framework/io/json.h" #include #include +#include +#include +#include +#include +#include +#include namespace engine::models::voxcpm2 { namespace json = engine::io::json; @@ -138,8 +145,8 @@ VoxCPM2Config parse_config(const assets::ResourceBundle & resources) { const auto root = resources.parse_json("config"); VoxCPM2Config config; config.architecture = json::require_string(root, "architecture"); - if (config.architecture != "voxcpm2") { - throw std::runtime_error("VoxCPM2 config architecture mismatch: " + config.architecture); + if (config.architecture != "voxcpm2" && config.architecture != "voxcpm") { + throw std::runtime_error("VoxCPM config architecture mismatch: " + config.architecture); } config.lm = parse_lm_config(root.require("lm_config")); config.patch_size = json::optional_i64(root, "patch_size", config.patch_size); @@ -172,6 +179,616 @@ VoxCPM2Config parse_config(const assets::ResourceBundle & resources) { return config; } +namespace assets = engine::assets; + +namespace { +core::TensorShape make_tensor_shape(const std::vector & dims) { + if (dims.empty() || dims.size() > core::kMaxTensorRank) { + throw std::runtime_error("tensor rank must be between 1 and 4"); + } + switch (dims.size()) { + case 1: + return core::TensorShape::from_dims({dims[0]}); + case 2: + return core::TensorShape::from_dims({dims[0], dims[1]}); + case 3: + return core::TensorShape::from_dims({dims[0], dims[1], dims[2]}); + case 4: + return core::TensorShape::from_dims({dims[0], dims[1], dims[2], dims[3]}); + default: + throw std::runtime_error("unsupported tensor rank"); + } +} +} // namespace + +class TransformingTensorSource final : public assets::TensorSource { +public: + TransformingTensorSource( + std::shared_ptr source, + const VoxCPM2Config & config, + bool is_v1) + : source_(std::move(source)), config_(config), is_v1_(is_v1) { + build_routes(); + } + + const std::filesystem::path & source_path() const noexcept override { + return source_->source_path(); + } + + bool has_tensor(std::string_view name) const noexcept override { + const std::string key{std::string(name)}; + if (routes_.find(key) != routes_.end() || + synthesized_tensors_.find(key) != synthesized_tensors_.end()) { + return true; + } + if (is_v1_) { + // v1 GGUF stores folded AudioVAE conv weights; the loader asks for + // decomposed weight_v/weight_g names which we synthesize from the + // folded tensors on demand. + const auto base = folded_base_name(key); + if (!base.empty() && folded_convs_.find(base) != folded_convs_.end()) { + return true; + } + } + return false; + } + + assets::TensorMetadata require_metadata(std::string_view name) const override { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it != synthesized_tensors_.end()) { + return it->second; + } + const std::string key{std::string(name)}; + if (is_v1_) { + const auto base = folded_base_name(key); + if (!base.empty()) { + const auto folded_it = folded_convs_.find(base); + if (folded_it != folded_convs_.end()) { + auto metadata = source_->require_metadata(folded_it->second); + metadata.name = key; + if (has_suffix(key, ".weight_g") && !metadata.shape.empty()) { + metadata.shape = {metadata.shape.front(), 1, 1}; + } + return metadata; + } + } + } + const auto route_it = routes_.find(key); + if (route_it == routes_.end()) { + throw std::runtime_error("missing tensor: " + std::string(name)); + } + auto metadata = source_->require_metadata(route_it->second); + metadata.name = key; + // Apply shape transformations if needed + if (reshape_map_.find(key) != reshape_map_.end()) { + metadata.shape = reshape_map_.at(key); + } + return metadata; + } + + std::vector tensors() const override { + std::vector out; + out.reserve(routes_.size() + synthesized_tensors_.size()); + for (const auto & [name, route] : routes_) { + out.push_back(require_metadata(name)); + } + for (const auto & [name, metadata] : synthesized_tensors_) { + out.push_back(metadata); + } + std::sort(out.begin(), out.end(), + [](const assets::TensorMetadata & lhs, const assets::TensorMetadata & rhs) { + return lhs.name < rhs.name; + }); + return out; + } + + void release_storage() const override { source_->release_storage(); } + + assets::RawTensorData require_tensor_data(std::string_view name) const override { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it != synthesized_tensors_.end()) { + return generate_synthesized_tensor(name); + } + const std::string key{std::string(name)}; + if (is_v1_) { + const auto base = folded_base_name(key); + if (!base.empty() && folded_convs_.find(base) != folded_convs_.end()) { + auto data = source_->require_tensor_data(folded_convs_.at(base)); + data.metadata.name = key; + return data; + } + } + const auto route_it = routes_.find(key); + if (route_it == routes_.end()) { + throw std::runtime_error("missing tensor: " + std::string(name)); + } + auto data = source_->require_tensor_data(route_it->second); + data.metadata.name = key; + // Apply transformations + if (reshape_map_.find(key) != reshape_map_.end()) { + const auto & target_shape = reshape_map_.at(key); + if (data.metadata.shape != target_shape) { + // Reshape the data + data = reshape_tensor_data(data, target_shape); + } + } + return data; + } + + std::vector require_f32( + std::string_view name, + const std::optional> & expected_shape) const override { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it != synthesized_tensors_.end()) { + return generate_synthesized_f32(name); + } + const std::string key{std::string(name)}; + if (is_v1_) { + const auto base = folded_base_name(key); + if (!base.empty()) { + const auto folded_it = folded_convs_.find(base); + if (folded_it != folded_convs_.end()) { + const auto folded = source_->require_f32(folded_it->second, std::nullopt); + if (has_suffix(key, ".weight_g")) { + return folded_weight_g(folded, folded_it->second, expected_shape); + } + return folded; + } + } + } + const auto route_it = routes_.find(key); + if (route_it == routes_.end()) { + throw std::runtime_error("missing tensor: " + std::string(name)); + } + if (is_v1_ && expected_shape.has_value()) { + const auto meta = source_->require_metadata(route_it->second); + const int64_t expected_elems = checked_element_count("expected", *expected_shape); + const int64_t actual_elems = checked_element_count(route_it->second, meta.shape); + if (expected_elems == actual_elems && meta.shape != *expected_shape) { + return source_->require_f32(route_it->second, std::nullopt); + } + } + // Check if we need to reshape + if (reshape_map_.find(key) != reshape_map_.end()) { + const auto & target_shape = reshape_map_.at(key); + if (expected_shape.has_value() && *expected_shape != target_shape) { + // We'll fetch with target shape and then it will be validated + } + return source_->require_f32(route_it->second, target_shape); + } + return source_->require_f32(route_it->second, expected_shape); + } + + std::optional> optional_f32( + std::string_view name, + const std::optional> & expected_shape) const override { + if (!has_tensor(name)) return std::nullopt; + return require_f32(name, expected_shape); + } + + void set_backend_tensor( + ggml_tensor * tensor, + std::string_view name, + assets::TensorStorageType storage_type, + const std::vector & expected_shape) const override { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it != synthesized_tensors_.end()) { + const auto values = generate_synthesized_f32(name); + engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, values, + make_tensor_shape(expected_shape), + engine::assets::ggml_type_for_tensor_storage(storage_type)); + return; + } + const auto route_it = routes_.find(std::string(name)); + if (route_it == routes_.end()) { + throw std::runtime_error("missing tensor: " + std::string(name)); + } + // Check for weight norm decomposition (weight_v + weight_g) + const std::string logical_name = std::string(name); + if (weight_norm_map_.find(logical_name) != weight_norm_map_.end()) { + const auto & wn = weight_norm_map_.at(logical_name); + const auto weight_v = source_->require_f32(wn.weight_v_name, wn.weight_v_shape); + const auto weight_g = source_->require_f32(wn.weight_g_name, wn.weight_g_shape); + const auto folded = fold_weight_norm(weight_v, weight_g, wn.out_channels, wn.in_channels, wn.kernel_size); + const auto shape = make_tensor_shape(expected_shape); + const ggml_type type = engine::assets::ggml_type_for_tensor_storage( + engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); + engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, folded, shape, type); + return; + } + // Check for reshape + if (reshape_map_.find(logical_name) != reshape_map_.end()) { + const auto & target_shape = reshape_map_.at(logical_name); + const auto values = source_->require_f32(route_it->second, target_shape); + const auto shape = make_tensor_shape(expected_shape); + const ggml_type type = engine::assets::ggml_type_for_tensor_storage( + engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); + engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, values, shape, type); + return; + } + source_->set_backend_tensor(tensor, route_it->second, storage_type, expected_shape); + } + + void set_backend_f32_tensor( + ggml_tensor * tensor, + std::string_view name, + const std::vector & expected_shape) const override { + set_backend_tensor(tensor, name, assets::TensorStorageType::F32, expected_shape); + } + + int64_t require_i64_scalar(std::string_view name) const override { + return source_->require_i64_scalar(name); + } + +private: + struct WeightNormInfo { + std::string weight_v_name; + std::string weight_g_name; + std::vector weight_v_shape; + std::vector weight_g_shape; + int64_t out_channels = 0; + int64_t in_channels = 0; + int64_t kernel_size = 0; + }; + + void build_routes() { + // V1 -> V2 tensor name mapping + std::unordered_map rename_map = { + // LM embeddings + {"token_embd.weight", "base_lm.embed_tokens.weight"}, + // LM blocks + {"blk.", "base_lm.layers."}, + {"attn_q.weight", "self_attn.q_proj.weight"}, + {"attn_k.weight", "self_attn.k_proj.weight"}, + {"attn_v.weight", "self_attn.v_proj.weight"}, + {"attn_norm.weight", "input_layernorm.weight"}, + {"attn_output.weight", "self_attn.o_proj.weight"}, + {"ffn_norm.weight", "post_attention_layernorm.weight"}, + {"ffn_gate.weight", "mlp.gate_proj.weight"}, + {"ffn_up.weight", "mlp.up_proj.weight"}, + {"ffn_down.weight", "mlp.down_proj.weight"}, + // Output norm + {"output_norm.weight", "base_lm.norm.weight"}, + // Residual LM + {"residual_lm.blk.", "residual_lm.layers."}, + {"residual_lm.output_norm.weight", "residual_lm.norm.weight"}, + // Local encoder (feat_encoder) + {"locenc.in_proj.weight", "feat_encoder.in_proj.weight"}, + {"locenc.in_proj.bias", "feat_encoder.in_proj.bias"}, + {"locenc.special_token", "feat_encoder.special_token"}, + {"locenc.blk.", "feat_encoder.encoder.layers."}, + {"locenc.output_norm.weight", "feat_encoder.encoder.norm.weight"}, + // Local DiT (feat_decoder) + {"locdit.in_proj.weight", "feat_decoder.estimator.in_proj.weight"}, + {"locdit.in_proj.bias", "feat_decoder.estimator.in_proj.bias"}, + {"locdit.cond_proj.weight", "feat_decoder.estimator.cond_proj.weight"}, + {"locdit.cond_proj.bias", "feat_decoder.estimator.cond_proj.bias"}, + {"locdit.out_proj.weight", "feat_decoder.estimator.out_proj.weight"}, + {"locdit.out_proj.bias", "feat_decoder.estimator.out_proj.bias"}, + {"locdit.time_mlp.linear_1.weight", "feat_decoder.estimator.time_mlp.linear_1.weight"}, + {"locdit.time_mlp.linear_1.bias", "feat_decoder.estimator.time_mlp.linear_1.bias"}, + {"locdit.time_mlp.linear_2.weight", "feat_decoder.estimator.time_mlp.linear_2.weight"}, + {"locdit.time_mlp.linear_2.bias", "feat_decoder.estimator.time_mlp.linear_2.bias"}, + {"locdit.delta_time_mlp.linear_1.weight", "feat_decoder.estimator.delta_time_mlp.linear_1.weight"}, + {"locdit.delta_time_mlp.linear_1.bias", "feat_decoder.estimator.delta_time_mlp.linear_1.bias"}, + {"locdit.delta_time_mlp.linear_2.weight", "feat_decoder.estimator.delta_time_mlp.linear_2.weight"}, + {"locdit.delta_time_mlp.linear_2.bias", "feat_decoder.estimator.delta_time_mlp.linear_2.bias"}, + {"locdit.output_norm.weight", "feat_decoder.estimator.decoder.norm.weight"}, + {"locdit.blk.", "feat_decoder.estimator.decoder.layers."}, + // Projections + {"proj.enc_to_lm.weight", "enc_to_lm_proj.weight"}, + {"proj.enc_to_lm.bias", "enc_to_lm_proj.bias"}, + {"proj.lm_to_dit.weight", "lm_to_dit_proj.weight"}, + {"proj.lm_to_dit.bias", "lm_to_dit_proj.bias"}, + {"proj.res_to_dit.weight", "res_to_dit_proj.weight"}, + {"proj.res_to_dit.bias", "res_to_dit_proj.bias"}, + {"fusion_concat_proj.weight", "fusion_concat_proj.weight"}, + {"stop.stop_proj.weight", "stop_proj.weight"}, + {"stop.stop_proj.bias", "stop_proj.bias"}, + {"stop.stop_head.weight", "stop_head.weight"}, + // FSQ + {"fsq.in_proj.weight", "fsq_layer.in_proj.weight"}, + {"fsq.in_proj.bias", "fsq_layer.in_proj.bias"}, + {"fsq.out_proj.weight", "fsq_layer.out_proj.weight"}, + {"fsq.out_proj.bias", "fsq_layer.out_proj.bias"}, + // Audio VAE (prefixed with audio_vae.) + {"audio_vae.encoder.block.", "encoder.block."}, + {"audio_vae.encoder.fc_mu", "encoder.fc_mu"}, + {"audio_vae.decoder.model.", "decoder.model."}, + {"audio_vae.decoder.sr_cond_model.", "decoder.sr_cond_model."}, + }; + + // Build routes by scanning source tensors + for (const auto & tensor : source_->tensors()) { + std::string v1_name = tensor.name; + std::string v2_name = v1_name; + + // Apply prefix replacements + for (const auto & [from, to] : rename_map) { + if (v2_name.rfind(from, 0) == 0) { + v2_name = to + v2_name.substr(from.size()); + break; + } + } + + // Handle blk.N.* -> layers.N.* (base LM, residual LM, locenc, locdit) + constexpr std::string_view kBlk = "blk."; + const size_t blk_pos = v1_name.find(kBlk); + if (blk_pos != std::string::npos) { + const size_t layer_start = blk_pos + kBlk.size(); + const size_t dot = v1_name.find('.', layer_start); + if (dot != std::string::npos) { + const std::string layer_idx = v1_name.substr(layer_start, dot - layer_start); + const std::string rest = v1_name.substr(dot + 1); + if (v1_name.rfind("residual_lm.", 0) == 0) { + v2_name = "residual_lm.layers." + layer_idx + "." + rest; + } else if (v1_name.rfind("locenc.", 0) == 0) { + v2_name = "feat_encoder.encoder.layers." + layer_idx + "." + rest; + } else if (v1_name.rfind("locdit.", 0) == 0) { + v2_name = "feat_decoder.estimator.decoder.layers." + layer_idx + "." + rest; + } else { + v2_name = "base_lm.layers." + layer_idx + "." + rest; + } + // Further sub-replacements + for (const auto & [from, to] : rename_map) { + size_t pos = v2_name.find(from); + if (pos != std::string::npos) { + v2_name.replace(pos, from.size(), to); + } + } + } + } + + routes_[v2_name] = v1_name; + } + + // Reshape map + reshape_map_ = { + // feat_quant: {N, F} -> {N, F, 1} + // merge: {N, D} -> {N, D, 1} + // downsample/upsample: {out, in} -> {out, in, k, k} (k=3 for 3x3) + }; + + // Folded AudioVAE conv weights: v1 GGUF stores weight-norm weights + // already folded into a single `.weight` tensor, while the v2 loader + // requests decomposed `.weight_v`/`.weight_g` names. Register every + // audio_vae conv weight so those logical names resolve to the folded + // data (weight_v) and its per-channel row norms (weight_g), which makes + // the loader's fold_weight_norm an exact identity. + if (is_v1_) { + std::vector> folded; + for (const auto & [logical, source] : routes_) { + if (source.rfind("audio_vae.", 0) == 0 && has_suffix(logical, ".weight")) { + folded.emplace_back( + logical.substr(0, logical.size() - 7), source); + } + } + for (const auto & [base, source] : folded) { + folded_convs_[base] = source; + } + } + + // Synthesized tensors for V1 + const int64_t encoder_hidden = config_.encoder.hidden_dim; + const int64_t feat_dim = config_.feat_dim; + const int64_t lm_hidden = config_.lm.hidden_size; + + // feat_encoder.scale_embed (identity buckets) + synthesized_tensors_["feat_encoder.scale_embed.weight"] = + assets::TensorMetadata{"feat_encoder.scale_embed.weight", "F32", {32, encoder_hidden}}; + synthesized_tensors_["feat_encoder.bias_embed.weight"] = + assets::TensorMetadata{"feat_encoder.bias_embed.weight", "F32", {32, encoder_hidden}}; + + // feat_encoder.fc_logvar (zeros) + synthesized_tensors_["feat_encoder.fc_logvar.weight"] = + assets::TensorMetadata{"feat_encoder.fc_logvar.weight", "F32", {feat_dim, encoder_hidden}}; + + // feat_encoder.diag (identity) + synthesized_tensors_["feat_encoder.diag"] = + assets::TensorMetadata{"feat_encoder.diag", "F32", {feat_dim}}; + + // feat_encoder.special_token (from token_embd) + synthesized_tensors_["feat_encoder.special_token"] = + assets::TensorMetadata{"feat_encoder.special_token", "F32", {1, 1, 1, encoder_hidden}}; + + // token_embd.extra_bias (from logit_scale or zeros) + synthesized_tensors_["token_embd.extra_bias"] = + assets::TensorMetadata{"token_embd.extra_bias", "F32", {lm_hidden}}; + + // feat_encoder.merge (zeros) + synthesized_tensors_["feat_encoder.merge.weight"] = + assets::TensorMetadata{"feat_encoder.merge.weight", "F32", {encoder_hidden, feat_dim}}; + + // Identity SR-condition embeddings for V1 decoder blocks. VoxCPM1 + // GGUFs contain no sr_cond_model tensors (no SR conditioning), but the + // shared decoder loader requires scale_embed/bias_embed. + { + const auto & vae = config_.audio_vae; + const size_t num_blocks = vae.decoder_rates.size(); + for (size_t i = 0; i < num_blocks; ++i) { + const int64_t input_channels = + vae.decoder_dim / (int64_t{1} << static_cast(i)); + const std::string prefix = + "decoder.sr_cond_model." + std::to_string(i + 2) + "."; + synthesized_tensors_[prefix + "scale_embed.weight"] = + assets::TensorMetadata{prefix + "scale_embed.weight", "F32", {1, input_channels}}; + synthesized_tensors_[prefix + "bias_embed.weight"] = + assets::TensorMetadata{prefix + "bias_embed.weight", "F32", {1, input_channels}}; + } + } + + // Missing projection weights for V1 (not in VoxCPM1 GGUF) + synthesized_tensors_["fusion_concat_proj.weight"] = + assets::TensorMetadata{"fusion_concat_proj.weight", "F32", {lm_hidden, lm_hidden * 2}}; + synthesized_tensors_["fusion_concat_proj.bias"] = + assets::TensorMetadata{"fusion_concat_proj.bias", "F32", {lm_hidden}}; + synthesized_tensors_["stop_proj.weight"] = + assets::TensorMetadata{"stop_proj.weight", "F32", {lm_hidden, lm_hidden}}; + synthesized_tensors_["stop_head.weight"] = + assets::TensorMetadata{"stop_head.weight", "F32", {2, lm_hidden}}; + } + + std::vector fold_weight_norm( + const std::vector & weight_v, + const std::vector & weight_g, + int64_t out_channels, int64_t in_channels, int64_t kernel_size) const { + if (static_cast(weight_v.size()) != out_channels * in_channels * kernel_size || + static_cast(weight_g.size()) != out_channels) { + throw std::runtime_error("VoxCPM1 weight-norm shape mismatch"); + } + std::vector out(weight_v.size(), 0.0F); + for (int64_t d0 = 0; d0 < out_channels; ++d0) { + const size_t base = static_cast(d0 * in_channels * kernel_size); + double norm_sq = 0.0; + for (int64_t i = 0; i < in_channels * kernel_size; ++i) { + const double value = weight_v[base + static_cast(i)]; + norm_sq += value * value; + } + const float scale = weight_g[static_cast(d0)] / + static_cast(std::sqrt(norm_sq + 1e-8)); + for (int64_t i = 0; i < in_channels * kernel_size; ++i) { + out[base + static_cast(i)] = weight_v[base + static_cast(i)] * scale; + } + } + return out; + } + + assets::RawTensorData reshape_tensor_data(const assets::RawTensorData & data, + const std::vector & target_shape) const { + // For now, just return the data as-is (validation happens elsewhere) + // The actual reshape happens in require_f32 + return data; + } + + assets::RawTensorData generate_synthesized_tensor(std::string_view name) const { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it == synthesized_tensors_.end()) { + throw std::runtime_error("no synthesized tensor: " + std::string(name)); + } + const auto & metadata = it->second; + const int64_t num_elements = std::accumulate(metadata.shape.begin(), metadata.shape.end(), 1, std::multiplies()); + std::vector bytes(num_elements * sizeof(float)); + std::memset(bytes.data(), 0, bytes.size()); + return {metadata, std::move(bytes)}; + } + + std::vector generate_synthesized_f32(std::string_view name) const { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it == synthesized_tensors_.end()) { + throw std::runtime_error("no synthesized tensor: " + std::string(name)); + } + const auto & metadata = it->second; + const int64_t num_elements = std::accumulate(metadata.shape.begin(), metadata.shape.end(), 1, std::multiplies()); + if (name == "feat_encoder.diag") { + std::vector out(num_elements, 1.0F); + return out; + } + if (std::string_view prefix = "decoder.sr_cond_model."; + name.rfind(prefix, 0) == 0 && has_suffix(name, ".scale_embed.weight")) { + return std::vector(num_elements, 1.0F); + } + if (name == "feat_encoder.scale_embed.weight" || name == "feat_encoder.bias_embed.weight") { + // Identity-like initialization + std::vector out(num_elements, 0.0F); + // Fill with small values + for (size_t i = 0; i < out.size(); ++i) { + out[i] = 0.01F; + } + return out; + } + return std::vector(num_elements, 0.0F); + } + + static bool has_suffix(std::string_view value, std::string_view suffix) { + return value.size() >= suffix.size() && + value.substr(value.size() - suffix.size()) == suffix; + } + + std::string folded_base_name(const std::string & key) const { + constexpr std::string_view kWeightV = ".weight_v"; + constexpr std::string_view kWeightG = ".weight_g"; + if (has_suffix(key, kWeightV)) { + return key.substr(0, key.size() - kWeightV.size()); + } + if (has_suffix(key, kWeightG)) { + return key.substr(0, key.size() - kWeightG.size()); + } + return ""; + } + + std::vector folded_weight_g( + const std::vector & folded, + const std::string & folded_source_name, + const std::optional> & expected_shape) const { + const auto meta = source_->require_metadata(folded_source_name); + const int64_t groups = expected_shape.has_value() && !expected_shape->empty() + ? expected_shape->front() + : (meta.shape.empty() ? 0 : meta.shape.front()); + const int64_t rows = checked_element_count(folded_source_name, meta.shape); + if (groups <= 0 || rows == 0 || rows % groups != 0) { + throw std::runtime_error("folded weight_g shape mismatch: " + folded_source_name); + } + const int64_t inner = rows / groups; + std::vector out(static_cast(groups), 0.0F); + for (int64_t g = 0; g < groups; ++g) { + double norm_sq = 0.0; + for (int64_t i = 0; i < inner; ++i) { + const float value = folded[static_cast(g * inner + i)]; + norm_sq += static_cast(value) * static_cast(value); + } + out[static_cast(g)] = static_cast(std::sqrt(norm_sq)); + } + return out; + } + + static int64_t checked_element_count(std::string_view name, const std::vector & shape) { + int64_t count = 1; + for (const int64_t dim : shape) { + if (dim <= 0) { + throw std::runtime_error("tensor shape contains a non-positive dimension: " + std::string(name)); + } + if (count > std::numeric_limits::max() / dim) { + throw std::runtime_error("tensor element count overflow: " + std::string(name)); + } + count *= dim; + } + return count; + } + + std::shared_ptr source_; + VoxCPM2Config config_; + bool is_v1_; + std::unordered_map routes_; + std::unordered_map> reshape_map_; + std::unordered_map weight_norm_map_; + std::unordered_map synthesized_tensors_; + std::unordered_map folded_convs_; +}; + +void require_vae_weight_v_shape(const assets::TensorSource & source, + std::string_view name, + const std::vector & expected_shape, + bool relaxed_rank) { + const auto metadata = source.require_metadata(name); + if (metadata.shape == expected_shape) { + return; + } + if (!relaxed_rank) { + throw std::runtime_error("tensor shape mismatch for " + std::string(name)); + } + int64_t expected_elems = 1; + for (const int64_t dim : expected_shape) { + expected_elems *= dim; + } + int64_t actual_elems = 1; + for (const int64_t dim : metadata.shape) { + actual_elems *= dim; + } + if (actual_elems != expected_elems) { + throw std::runtime_error("tensor element count mismatch for " + std::string(name)); + } +} + void validate_weight_anchors(const VoxCPM2Assets & assets) { const auto & config = assets.config; const auto & weights = *assets.model_weights; @@ -199,22 +816,34 @@ void validate_weight_anchors(const VoxCPM2Assets & assets) { assets::require_tensor_shape(weights, "stop_head.weight", {2, config.lm.hidden_size}); const auto & vae = *assets.audiovae_weights; - assets::require_tensor_shape(vae, "encoder.fc_mu.weight_v", {config.audio_vae.latent_dim, config.audio_vae.decoder_dim, 3}); + int64_t encoder_in_channels = config.audio_vae.encoder_dim; + for (size_t i = 0; i < config.audio_vae.encoder_rates.size(); ++i) { + encoder_in_channels *= 2; + } + require_vae_weight_v_shape(vae, "encoder.fc_mu.weight_v", {config.audio_vae.latent_dim, encoder_in_channels, 3}, config.v1); assets::require_tensor_shape(vae, "encoder.fc_mu.bias", {config.audio_vae.latent_dim}); - assets::require_tensor_shape(vae, "decoder.model.0.weight_v", {config.audio_vae.latent_dim, 1, 7}); - assets::require_tensor_shape(vae, "decoder.model.1.weight_v", {config.audio_vae.decoder_dim, config.audio_vae.latent_dim, 1}); + require_vae_weight_v_shape(vae, "decoder.model.0.weight_v", {config.audio_vae.latent_dim, 1, 7}, config.v1); + require_vae_weight_v_shape(vae, "decoder.model.1.weight_v", {config.audio_vae.decoder_dim, config.audio_vae.latent_dim, 1}, config.v1); } } -std::shared_ptr load_voxcpm2_assets(const std::filesystem::path & model_path) { +std::shared_ptr load_voxcpm2_assets(const std::filesystem::path & model_path, bool is_v1) { auto out = std::make_shared(); out->resources = engine::model_spec::load_resource_bundle( model_path, - engine::model_spec::default_spec_path("voxcpm2")); + engine::model_spec::default_spec_path(is_v1 ? "voxcpm1" : "voxcpm2")); out->config = parse_config(out->resources); - out->model_weights = out->resources.open_tensor_source("weights"); - out->audiovae_weights = out->resources.open_tensor_source("audiovae_weights"); + out->config.v1 = is_v1; + auto raw_model_weights = out->resources.open_tensor_source("weights"); + auto raw_audiovae_weights = out->resources.open_tensor_source("audiovae_weights"); + if (is_v1) { + out->model_weights = std::make_shared(raw_model_weights, out->config, true); + out->audiovae_weights = std::make_shared(raw_audiovae_weights, out->config, true); + } else { + out->model_weights = raw_model_weights; + out->audiovae_weights = raw_audiovae_weights; + } validate_weight_anchors(*out); return out; } diff --git a/src/models/voxcpm2/generator.cpp b/src/models/voxcpm2/generator.cpp index aeffe256..81f60911 100644 --- a/src/models/voxcpm2/generator.cpp +++ b/src/models/voxcpm2/generator.cpp @@ -115,6 +115,18 @@ std::vector concat_dit_mu(const std::vector &lm, return out; } +std::vector add_dit_mu(const std::vector &lm, + const std::vector &residual) { + if (lm.size() != residual.size()) { + throw std::runtime_error("VoxCPM1 dit mu inputs must have equal size"); + } + std::vector out(lm.size(), 0.0F); + for (size_t i = 0; i < lm.size(); ++i) { + out[i] = lm[i] + residual[i]; + } + return out; +} + void append_patch(std::vector &features, const std::vector &patch, int64_t expected_size) { if (static_cast(patch.size()) != expected_size) { @@ -401,23 +413,32 @@ class VoxCPM2StepProjectionRuntime::Impl { .build(ctx, fsq, proj.fsq_out_proj); fsq_hidden_output_ = fsq.tensor; - auto current_residual_concat = - engine::modules::ConcatModule({1}).build(ctx, lm_hidden, current_embed); - auto current_residual_input = - engine::modules::LinearModule( - binding::linear_config(config.lm.hidden_size * 2, - config.lm.hidden_size, true)) - .build(ctx, current_residual_concat, proj.fusion_concat_proj); - current_residual_input_output_ = current_residual_input.tensor; - - auto residual_concat = - engine::modules::ConcatModule({1}).build(ctx, fsq, current_embed); - auto residual_input = - engine::modules::LinearModule( - binding::linear_config(config.lm.hidden_size * 2, - config.lm.hidden_size, true)) - .build(ctx, residual_concat, proj.fusion_concat_proj); - residual_input_output_ = residual_input.tensor; + if (config.v1) { + current_residual_input_output_ = + engine::modules::AddModule() + .build(ctx, lm_hidden, current_embed) + .tensor; + residual_input_output_ = + engine::modules::AddModule().build(ctx, fsq, current_embed).tensor; + } else { + auto current_residual_concat = + engine::modules::ConcatModule({1}).build(ctx, lm_hidden, current_embed); + auto current_residual_input = + engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size * 2, + config.lm.hidden_size, true)) + .build(ctx, current_residual_concat, proj.fusion_concat_proj); + current_residual_input_output_ = current_residual_input.tensor; + + auto residual_concat = + engine::modules::ConcatModule({1}).build(ctx, fsq, current_embed); + auto residual_input = + engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size * 2, + config.lm.hidden_size, true)) + .build(ctx, residual_concat, proj.fusion_concat_proj); + residual_input_output_ = residual_input.tensor; + } auto current_lm_dit = engine::modules::LinearModule( @@ -1024,7 +1045,8 @@ class VoxCPM2CFMRuntime::Impl { throw std::runtime_error("VoxCPM2 CFM received non-finite scalar input"); } const int64_t patch_elems = config.feat_dim * config.patch_size; - if (static_cast(mu.size()) != config.dit.hidden_dim * 2) { + const int64_t mu_dim = config.dit.hidden_dim * (config.v1 ? 1 : 2); + if (static_cast(mu.size()) != mu_dim) { throw std::runtime_error("VoxCPM2 CFM mu size mismatch"); } if (static_cast(cond_patch.size()) != patch_elems) { @@ -1556,8 +1578,11 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { for (int64_t index = 0; index < max_tokens; ++index) { const auto projected = projection_.run(lm_hidden, residual_hidden, zero_hidden); - const auto mu = concat_dit_mu(projected.current_lm_dit_hidden, - projected.residual_dit_hidden); + const auto mu = assets_->config.v1 + ? add_dit_mu(projected.current_lm_dit_hidden, + projected.residual_dit_hidden) + : concat_dit_mu(projected.current_lm_dit_hidden, + projected.residual_dit_hidden); const auto patch = cfm_.generate_patch( mu, prefix_cond, options.num_inference_steps, options.guidance_scale, options.seed, patch_noise_start, options.cfm_noise_file); diff --git a/src/models/voxcpm2/loader.cpp b/src/models/voxcpm2/loader.cpp index 3821dba7..33df9175 100644 --- a/src/models/voxcpm2/loader.cpp +++ b/src/models/voxcpm2/loader.cpp @@ -1,6 +1,7 @@ #include "engine/models/voxcpm2/loader.h" #include "engine/framework/model_spec/package.h" +#include "engine/models/voxcpm2/assets.h" #include "engine/models/voxcpm2/session.h" #include @@ -75,7 +76,7 @@ class VoxCPM2Loader final : public runtime::IVoiceModelLoader { runtime::ModelInspection inspect(const runtime::ModelLoadRequest &request) const override { const auto assets = - load_voxcpm2_assets(request.model_path); + load_voxcpm2_assets(request.model_path, false); runtime::ModelInspection inspection; inspection.model_root = assets->resources.model_root(); inspection.metadata = metadata(*assets); @@ -99,6 +100,104 @@ class VoxCPM2Loader final : public runtime::IVoiceModelLoader { } }; +// VoxCPM1 Loader +runtime::CapabilitySet capabilities_v1(const VoxCPM2Assets &) { + runtime::CapabilitySet out; + out.supported_tasks = { + {runtime::VoiceTaskKind::Tts, + {runtime::RunMode::Offline}}, + }; + out.languages = {"Auto"}; + out.supports_speaker_reference = true; + return out; +} + +runtime::ModelMetadata metadata_v1(const VoxCPM2Assets &assets) { + runtime::ModelMetadata out; + out.family = "voxcpm1"; + out.variant = assets.config.architecture; + out.description = "VoxCPM1 loaded from GGUF assets."; + return out; +} + +runtime::ModelCliInterface cli_v1(const VoxCPM2Assets &) { + runtime::ModelCliInterface out; + out.request_options = { + {"text_chunk_mode", "default|tag_aware|japanese|endline", + "Text chunking mode; default tag_aware."}, + }; + out.session_options = { + {"voxcpm1.mem_saver", "true|false", + "Use tighter graph workspaces and release request runtime graphs; default false."}, + {"voxcpm1.prompt_cache_slots", "n", + "Prompt and prompt-audio embedding cache slots; default 1."}, + }; + return out; +} + +std::unique_ptr +load_voxcpm1_model(const std::filesystem::path &model_path) { + auto assets = load_voxcpm2_assets(model_path, true); + return std::make_unique( + metadata_v1(*assets), capabilities_v1(*assets), std::move(assets)); +} + +class VoxCPM1Loader final : public runtime::IVoiceModelLoader { +public: + std::string family() const override { return "voxcpm1"; } + + runtime::CapabilitySet advertised_capabilities() const override { + runtime::CapabilitySet out; + out.supported_tasks = { + {runtime::VoiceTaskKind::Tts, + {runtime::RunMode::Offline}}, + }; + out.supports_speaker_reference = true; + return out; + } + + std::string advertised_instructions_policy() const override { + return "text_prefix"; + } + + bool can_load(const runtime::ModelLoadRequest &request) const override { + try { + (void)engine::model_spec::load_resource_bundle( + request.model_path, + engine::model_spec::default_spec_path(family())); + return !request.family_hint.has_value() || *request.family_hint == family(); + } catch (...) { + return false; + } + } + + runtime::ModelInspection + inspect(const runtime::ModelLoadRequest &request) const override { + const auto assets = + load_voxcpm2_assets(request.model_path, true); + runtime::ModelInspection inspection; + inspection.model_root = assets->resources.model_root(); + inspection.metadata = metadata_v1(*assets); + inspection.capabilities = capabilities_v1(*assets); + inspection.cli = cli_v1(*assets); + const auto spec_path = engine::model_spec::default_spec_path(family()); + inspection.discovered_configs = runtime::discover_named_assets_from_package_spec( + request.model_path, + spec_path, + engine::model_spec::ResourceKind::Files); + inspection.discovered_weights = runtime::discover_named_assets_from_package_spec( + request.model_path, + spec_path, + engine::model_spec::ResourceKind::Tensors); + return inspection; + } + + std::unique_ptr + load(const runtime::ModelLoadRequest &request) const override { + return load_voxcpm1_model(request.model_path); + } +}; + } // namespace VoxCPM2LoadedModel::VoxCPM2LoadedModel( @@ -136,7 +235,7 @@ VoxCPM2LoadedModel::create_task_session( std::unique_ptr load_voxcpm2_model(const std::filesystem::path &model_path) { - auto assets = load_voxcpm2_assets(model_path); + auto assets = load_voxcpm2_assets(model_path, false); return std::make_unique( metadata(*assets), capabilities(*assets), std::move(assets)); } @@ -145,4 +244,16 @@ std::shared_ptr make_voxcpm2_loader() { return std::make_shared(); } +// VoxCPM1 model loading +std::unique_ptr +load_voxcpm1_model(const std::filesystem::path &model_path) { + auto assets = load_voxcpm2_assets(model_path, true); + return std::make_unique( + metadata_v1(*assets), capabilities_v1(*assets), std::move(assets)); +} + +std::shared_ptr make_voxcpm1_loader() { + return std::make_shared(); +} + } // namespace engine::models::voxcpm2 diff --git a/src/models/voxcpm2/minicpm.cpp b/src/models/voxcpm2/minicpm.cpp index 5830c29e..e03a02e7 100644 --- a/src/models/voxcpm2/minicpm.cpp +++ b/src/models/voxcpm2/minicpm.cpp @@ -789,13 +789,15 @@ class VoxCPM2PromptPrefillRuntime::Impl { auto masked_current = mask_sequence(ctx, current_embeddings, audio_mask); auto residual_input = - engine::modules::ConcatModule({2}).build(ctx, lm_hidden, masked_current); - residual_input = - engine::modules::LinearModule( - binding::linear_config(config.lm.hidden_size * 2, - config.lm.hidden_size, true)) - .build(ctx, residual_input, - model_weights.projections.fusion_concat_proj); + config.v1 + ? engine::modules::AddModule{}.build(ctx, lm_hidden, masked_current) + : engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size * 2, + config.lm.hidden_size, true)) + .build(ctx, + engine::modules::ConcatModule({2}).build( + ctx, lm_hidden, masked_current), + model_weights.projections.fusion_concat_proj); auto residual_hidden = residual_input; for (const auto &layer : model_weights.residual_lm.layers) { From 11f37a70fff715dcd436846e8de86a8c836adc2b Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Mon, 17 Aug 2026 11:59:48 +0200 Subject: [PATCH 02/14] fix(voxcpm1): resolve partialy pure noise output by fixing synthesized weight handling and embedding transpose MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bug: VoxCPM1 model produced pure noise ("elloそ。") instead of speech due to: 1. Synthesized `fusion_concat_proj` weight (Xavier init) treated as learned weight → wrong concat+linear fusion 2. Embedding weight transposed in V1 GGUF: `token_embd.weight` stored as [hidden, vocab] but audio.cpp expects [vocab, hidden] Fix: - Add `is_synthesized()` to TensorSource interface to distinguish loaded vs synthesized weights- Implement in TransformingTensorSource for V1 models- Add embedding weight transpose in set_backend_tensor() for `base_lm.embed_tokens.weight` - Update 5 `has_fusion_proj` checks to exclude synthesized weights - Test: "This is a test run for the fix" now transcribes as "This is a test." (was pure noise) --> but still wrong. --- .../engine/framework/assets/tensor_source.h | 1 + src/models/voxcpm2/assets.cpp | 49 ++- src/models/voxcpm2/generator.cpp | 319 ++++++++++++++++-- src/models/voxcpm2/minicpm_blocks.h | 16 +- 4 files changed, 349 insertions(+), 36 deletions(-) diff --git a/include/engine/framework/assets/tensor_source.h b/include/engine/framework/assets/tensor_source.h index 6735f208..d5c28bba 100644 --- a/include/engine/framework/assets/tensor_source.h +++ b/include/engine/framework/assets/tensor_source.h @@ -118,6 +118,7 @@ class TensorSource { [[nodiscard]] std::string require_tensor_name( std::initializer_list candidates) const; [[nodiscard]] virtual int64_t require_i64_scalar(std::string_view name) const = 0; + [[nodiscard]] virtual bool is_synthesized(std::string_view name) const noexcept { return false; } }; [[nodiscard]] TensorStorageType parse_tensor_storage_type(std::string_view value); diff --git a/src/models/voxcpm2/assets.cpp b/src/models/voxcpm2/assets.cpp index 997f54ee..cc827af6 100644 --- a/src/models/voxcpm2/assets.cpp +++ b/src/models/voxcpm2/assets.cpp @@ -233,6 +233,11 @@ class TransformingTensorSource final : public assets::TensorSource { return false; } + [[nodiscard]] bool is_synthesized(std::string_view name) const noexcept override { + const std::string key{std::string(name)}; + return synthesized_tensors_.find(key) != synthesized_tensors_.end(); + } + assets::TensorMetadata require_metadata(std::string_view name) const override { const auto it = synthesized_tensors_.find(std::string(name)); if (it != synthesized_tensors_.end()) { @@ -406,6 +411,32 @@ class TransformingTensorSource final : public assets::TensorSource { engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, values, shape, type); return; } + // Special handling for V1 embedding weight: token_embd.weight is transposed in GGUF + // V1 GGUF stores [hidden_size, vocab_size] but we need [vocab_size, hidden_size] + if (is_v1_ && logical_name == "base_lm.embed_tokens.weight") { + const auto source_values = source_->require_f32(route_it->second, std::nullopt); + const auto source_meta = source_->require_metadata(route_it->second); + if (source_meta.shape.size() == 2) { + const int64_t src_rows = source_meta.shape[0]; + const int64_t src_cols = source_meta.shape[1]; + const int64_t dst_rows = expected_shape.size() > 0 ? expected_shape[0] : src_cols; + const int64_t dst_cols = expected_shape.size() > 1 ? expected_shape[1] : src_rows; + if (src_rows == dst_cols && src_cols == dst_rows) { + // Transpose the weight matrix + std::vector transposed(static_cast(dst_rows * dst_cols)); + for (int64_t i = 0; i < src_rows; ++i) { + for (int64_t j = 0; j < src_cols; ++j) { + transposed[static_cast(j * dst_rows + i)] = source_values[static_cast(i * src_cols + j)]; + } + } + const auto shape = make_tensor_shape(expected_shape); + const ggml_type type = engine::assets::ggml_type_for_tensor_storage( + engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); + engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, transposed, shape, type); + return; + } + } + } source_->set_backend_tensor(tensor, route_it->second, storage_type, expected_shape); } @@ -482,6 +513,9 @@ class TransformingTensorSource final : public assets::TensorSource { {"proj.lm_to_dit.bias", "lm_to_dit_proj.bias"}, {"proj.res_to_dit.weight", "res_to_dit_proj.weight"}, {"proj.res_to_dit.bias", "res_to_dit_proj.bias"}, + // V1→V2 mapping for fusion_concat_proj (critical for V1 models with fusion) + {"proj.fusion_concat.weight", "fusion_concat_proj.weight"}, + {"proj.fusion_concat.bias", "fusion_concat_proj.bias"}, {"fusion_concat_proj.weight", "fusion_concat_proj.weight"}, {"stop.stop_proj.weight", "stop_proj.weight"}, {"stop.stop_proj.bias", "stop_proj.bias"}, @@ -622,10 +656,6 @@ class TransformingTensorSource final : public assets::TensorSource { assets::TensorMetadata{"fusion_concat_proj.weight", "F32", {lm_hidden, lm_hidden * 2}}; synthesized_tensors_["fusion_concat_proj.bias"] = assets::TensorMetadata{"fusion_concat_proj.bias", "F32", {lm_hidden}}; - synthesized_tensors_["stop_proj.weight"] = - assets::TensorMetadata{"stop_proj.weight", "F32", {lm_hidden, lm_hidden}}; - synthesized_tensors_["stop_head.weight"] = - assets::TensorMetadata{"stop_head.weight", "F32", {2, lm_hidden}}; } std::vector fold_weight_norm( @@ -696,6 +726,17 @@ class TransformingTensorSource final : public assets::TensorSource { } return out; } + if (name == "fusion_concat_proj.weight") { + // Xavier/Glorot initialization for fusion_concat_proj weight + // shape is [lm_hidden, lm_hidden * 2] + std::vector out(num_elements); + const float scale = std::sqrt(2.0f / (config_.lm.hidden_size + config_.lm.hidden_size * 2)); + for (size_t i = 0; i < out.size(); ++i) { + // Simple uniform distribution in [-scale, scale] + out[i] = (static_cast(std::rand()) / RAND_MAX * 2.0f - 1.0f) * scale; + } + return out; + } return std::vector(num_elements, 0.0F); } diff --git a/src/models/voxcpm2/generator.cpp b/src/models/voxcpm2/generator.cpp index 81f60911..f79cf77a 100644 --- a/src/models/voxcpm2/generator.cpp +++ b/src/models/voxcpm2/generator.cpp @@ -413,14 +413,15 @@ class VoxCPM2StepProjectionRuntime::Impl { .build(ctx, fsq, proj.fsq_out_proj); fsq_hidden_output_ = fsq.tensor; - if (config.v1) { - current_residual_input_output_ = - engine::modules::AddModule() - .build(ctx, lm_hidden, current_embed) - .tensor; - residual_input_output_ = - engine::modules::AddModule().build(ctx, fsq, current_embed).tensor; - } else { + // Check if fusion_concat_proj weight exists and was loaded (not synthesized) + // This matches VoxCPM.cpp behavior which checks weight existence + // For V1 models, synthesized weights (Xavier init) should not count as present + const bool has_fusion_proj = + proj.fusion_concat_proj.weight.tensor != nullptr && + !weights_->assets().model_weights->is_synthesized("fusion_concat_proj.weight"); + + if (has_fusion_proj) { + // Concat + Linear (used by V2 and some V1 models trained with fusion) auto current_residual_concat = engine::modules::ConcatModule({1}).build(ctx, lm_hidden, current_embed); auto current_residual_input = @@ -438,6 +439,14 @@ class VoxCPM2StepProjectionRuntime::Impl { config.lm.hidden_size, true)) .build(ctx, residual_concat, proj.fusion_concat_proj); residual_input_output_ = residual_input.tensor; + } else { + // Simple ADD (true V1 without fusion_concat_proj) + current_residual_input_output_ = + engine::modules::AddModule() + .build(ctx, lm_hidden, current_embed) + .tensor; + residual_input_output_ = + engine::modules::AddModule().build(ctx, fsq, current_embed).tensor; } auto current_lm_dit = @@ -758,6 +767,12 @@ class VoxCPM2DiTEstimatorRuntime::Impl { const std::vector &time_embedding, const std::vector &delta_time_embedding) { const auto &config = weights_->assets().config; + // Check if fusion_concat_proj weight exists and was loaded (not synthesized) + // This matches VoxCPM.cpp behavior which checks weight existence + // For V1 models, synthesized weights (Xavier init) should not count as present + const bool has_fusion_proj = + weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && + !weights_->assets().model_weights->is_synthesized("fusion_concat_proj.weight"); const int64_t patch_elems = 2 * config.feat_dim * config.patch_size; if (static_cast(x.size()) != patch_elems) { throw std::runtime_error("VoxCPM2 DiT estimator x size mismatch"); @@ -765,7 +780,9 @@ class VoxCPM2DiTEstimatorRuntime::Impl { if (static_cast(cond.size()) != patch_elems) { throw std::runtime_error("VoxCPM2 DiT estimator cond size mismatch"); } - if (static_cast(mu.size()) != 2 * config.dit.hidden_dim * 2) { + const int64_t expected_mu = + has_fusion_proj ? 2 * config.dit.hidden_dim * 2 : 2 * config.dit.hidden_dim; + if (static_cast(mu.size()) != expected_mu) { throw std::runtime_error("VoxCPM2 DiT estimator mu size mismatch"); } if (static_cast(time_embedding.size()) != @@ -795,6 +812,125 @@ class VoxCPM2DiTEstimatorRuntime::Impl { std::vector output(static_cast(patch_elems), 0.0F); ggml_backend_tensor_get(output_, output.data(), 0, output.size() * sizeof(float)); + if (std::getenv("VOXCPM_DUMP_NORM0") != nullptr) { + ggml_tensor *tn = ggml_get_tensor(ctx_.get(), "dump_norm0"); + if (tn != nullptr) { + const size_t nn = static_cast(ggml_nelements(tn)); + std::vector buf(nn, 0.0F); + ggml_backend_tensor_get(tn, buf.data(), 0, buf.size() * sizeof(float)); + FILE *f = std::fopen("/tmp/opencode/ours_norm0.bin", "wb"); + if (f != nullptr) { + std::fwrite(buf.data(), sizeof(float), std::min(nn, 5120), f); + std::fclose(f); + } + } + } + if (std::getenv("VOXCPM_DUMP_DECODER_LAYERS") != nullptr) { + for (int li = 0; li < 8; ++li) { + char name[32]; + snprintf(name, sizeof(name), "dump_layer_%d", li); + ggml_tensor *t = ggml_get_tensor(ctx_.get(), name); + if (t == nullptr) { + continue; + } + const size_t n = static_cast(ggml_nelements(t)); + std::vector buf(n, 0.0F); + ggml_backend_tensor_get(t, buf.data(), 0, buf.size() * sizeof(float)); + if (li == 0) { + FILE *f = std::fopen("/tmp/opencode/ours_branch0.bin", "wb"); + if (f != nullptr) { + std::fwrite(buf.data(), sizeof(float), std::min(n, 5120), f); + std::fclose(f); + } + } else if (li == 1) { + FILE *f = std::fopen("/tmp/opencode/ours_branch1.bin", "wb"); + if (f != nullptr) { + std::fwrite(buf.data(), sizeof(float), std::min(n, 5120), f); + std::fclose(f); + } + } + double s = 0.0, l2 = 0.0; + for (float v : buf) { + s += v; + l2 += static_cast(v) * v; + } + // batch-local stats for the branch-0 (first ne1*ne0 elements) + double s0 = 0.0, l20 = 0.0; + const size_t branch_elems = static_cast(t->ne[0]) * static_cast(t->ne[1]); + for (size_t i = 0; i < std::min(branch_elems, buf.size()); ++i) { + s0 += buf[i]; + l20 += static_cast(buf[i]) * buf[i]; + } + fprintf(stderr, + "[DEC_LAYER] input#%d ne0=%lld ne1=%lld ne2=%lld ne3=%lld " + "sum=%.6g l2=%.6g branch0_sum=%.6g branch0_l2=%.6g " + "first4=%.6g %.6g %.6g %.6g\n", + li, static_cast(t->ne[0]), + static_cast(t->ne[1]), + static_cast(t->ne[2]), + static_cast(t->ne[3]), s, std::sqrt(l2), s0, + std::sqrt(l20), + buf.empty() ? 0.0 : static_cast(buf[0]), + buf.size() < 2 ? 0.0 : static_cast(buf[1]), + buf.size() < 3 ? 0.0 : static_cast(buf[2]), + buf.size() < 4 ? 0.0 : static_cast(buf[3])); + } + } + if (std::getenv("VOXCPM_DUMP_LOCDIT_WEIGHTS") != nullptr) { + const auto &dw = weights_->weights().dit; + auto dump_w = [](const char *tag, ggml_tensor *t) { + if (t == nullptr) { + fprintf(stderr, "[LOCDIT_W] %s \n", tag); + return; + } + const size_t nbytes = static_cast(ggml_nbytes(t)); + const size_t nelems = static_cast(ggml_nelements(t)); + std::vector raw(nbytes, 0); + ggml_backend_tensor_get(t, raw.data(), 0, nbytes); + fprintf(stderr, "[LOCDIT_W] %s ne0=%lld ne1=%lld type=%d nbytes=%zu " + "v[0..7]=", + tag, static_cast(t->ne[0]), + static_cast(t->ne[1]), static_cast(t->type), + nbytes); + double vals[8]; + for (size_t i = 0; i < 8; ++i) { + if (t->type == GGML_TYPE_Q8_0) { + const size_t block = i / 32; + const size_t in_block = i % 32; + const float scale = + ggml_fp16_to_fp32( + *reinterpret_cast( + raw.data() + block * 34)); + vals[i] = static_cast( + scale * + static_cast( + *reinterpret_cast( + raw.data() + block * 34 + 2 + in_block))); + } else if (t->type == GGML_TYPE_F32) { + vals[i] = static_cast( + *reinterpret_cast(raw.data() + i * 4)); + } else if (t->type == GGML_TYPE_F16) { + vals[i] = static_cast(ggml_fp16_to_fp32( + *reinterpret_cast(raw.data() + i * 2))); + } else { + vals[i] = 0.0; + } + } + (void)nelems; + for (size_t i = 0; i < 8; ++i) { + fprintf(stderr, "%.6g ", vals[i]); + } + fprintf(stderr, "\n"); + }; + dump_w("in_proj", dw.in_proj.weight.tensor); + dump_w("cond_proj", dw.cond_proj.weight.tensor); + dump_w("out_proj", dw.out_proj.weight.tensor); + dump_w("time_mlp1", dw.time_mlp_1.weight.tensor); + dump_w("decoder.l0.q", dw.decoder.layers[0].q_proj.weight.tensor); + dump_w("decoder.l0.k", dw.decoder.layers[0].k_proj.weight.tensor); + dump_w("decoder.l0.o", dw.decoder.layers[0].o_proj.weight.tensor); + dump_w("decoder.norm", dw.decoder.norm.weight->tensor); + } return output; } @@ -830,9 +966,19 @@ class VoxCPM2DiTEstimatorRuntime::Impl { if (mem_saver_) { ggml_set_input(cond_); } + // Check if fusion_concat_proj weight exists and was loaded (not synthesized) + // This matches VoxCPM.cpp behavior which checks weight existence + // For V1 models, synthesized weights (Xavier init) should not count as present + const bool has_fusion_proj = + weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && + !weights_->assets().model_weights->is_synthesized("fusion_concat_proj.weight"); mu_ = engine::core::make_tensor( ctx, GGML_TYPE_F32, - engine::core::TensorShape::from_dims({2, 2, config.hidden_dim})) + has_fusion_proj + ? engine::core::TensorShape::from_dims( + {2, 2, config.hidden_dim}) + : engine::core::TensorShape::from_dims( + {2, config.hidden_dim})) .tensor; if (mem_saver_) { ggml_set_input(mu_); @@ -902,19 +1048,38 @@ class VoxCPM2DiTEstimatorRuntime::Impl { binding::linear_config(config.hidden_dim, config.hidden_dim, true)) .build(ctx, dt, weights.delta_time_mlp_2); time = engine::modules::AddModule{}.build(ctx, time, dt); - time = engine::core::reshape_tensor( - ctx, time, - engine::core::TensorShape::from_dims({2, 1, config.hidden_dim})); - auto mu = engine::core::wrap_tensor( - mu_, engine::core::TensorShape::from_dims({2, 2, config.hidden_dim}), - GGML_TYPE_F32); - auto hidden = engine::modules::ConcatModule({1}).build(ctx, mu, time); + const int64_t prefix_token_count = + has_fusion_proj ? 2 + 1 : 1; + auto hidden = time; + if (!has_fusion_proj) { + // True V1 (no fusion projection): the DiT conditioning mu is a single + // hidden vector that is ADDED into the timestep token. Batch 0 carries + // mu (conditioned branch); batch 1 carries zeros (unconditioned branch), + // mirroring LocDiTModel::forward_cfg_pair_projected with mu_tokens == 1. + auto mu = engine::core::wrap_tensor( + mu_, engine::core::TensorShape::from_dims({2, config.hidden_dim}), + GGML_TYPE_F32); + hidden = engine::modules::AddModule{}.build(ctx, hidden, mu); + } + hidden = engine::core::reshape_tensor( + ctx, hidden, + engine::core::TensorShape::from_dims({2, 1, config.hidden_dim})); + if (has_fusion_proj) { + // V2 (or V1 with fusion projection): mu is two hidden vectors concatenated as + // separate prefix tokens before the timestep token, matching + // LocDiTModel with mu_tokens == 2. + auto mu = engine::core::wrap_tensor( + mu_, engine::core::TensorShape::from_dims({2, 2, config.hidden_dim}), + GGML_TYPE_F32); + hidden = engine::modules::ConcatModule({1}).build(ctx, mu, hidden); + } hidden = engine::modules::ConcatModule({1}).build(ctx, hidden, cond); hidden = engine::modules::ConcatModule({1}).build(ctx, hidden, x); - positions_ = ggml_new_tensor_1d(ctx_.get(), GGML_TYPE_I32, - 2 + 1 + root_config.patch_size * 2); + positions_ = ggml_new_tensor_1d( + ctx_.get(), GGML_TYPE_I32, + prefix_token_count + root_config.patch_size * 2); if (mem_saver_) { ggml_set_input(positions_); ggml_set_output(positions_); @@ -922,12 +1087,14 @@ class VoxCPM2DiTEstimatorRuntime::Impl { auto positions = engine::core::wrap_tensor(positions_, engine::core::TensorShape::from_dims( - {2 + 1 + root_config.patch_size * 2}), + {prefix_token_count + + root_config.patch_size * 2}), GGML_TYPE_I32); hidden = minicpm_transformer(ctx, hidden, positions, weights.decoder, false); hidden = engine::modules::SliceModule( - {1, 2 + 1 + root_config.patch_size, root_config.patch_size}) + {1, prefix_token_count + root_config.patch_size, + root_config.patch_size}) .build(ctx, hidden); hidden = engine::modules::LinearModule( binding::linear_config(config.hidden_dim, root_config.feat_dim, @@ -963,7 +1130,8 @@ class VoxCPM2DiTEstimatorRuntime::Impl { "failed to allocate VoxCPM2 DiT estimator graph"); } std::vector positions_data( - static_cast(2 + 1 + root_config.patch_size * 2), 0); + static_cast(prefix_token_count + root_config.patch_size * 2), + 0); for (int64_t i = 0; i < static_cast(positions_data.size()); ++i) { positions_data[static_cast(i)] = static_cast(i); } @@ -1038,6 +1206,12 @@ class VoxCPM2CFMRuntime::Impl { const std::string &noise_file, float temperature) { const auto &config = weights_->assets().config; + // Check if fusion_concat_proj weight exists and was loaded (not synthesized) + // This matches VoxCPM.cpp behavior which checks weight existence + // For V1 models, synthesized weights (Xavier init) should not count as present + const bool has_fusion_proj = + weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && + !weights_->assets().model_weights->is_synthesized("fusion_concat_proj.weight"); if (timesteps <= 0) { throw std::runtime_error("VoxCPM2 CFM requires positive timesteps"); } @@ -1045,7 +1219,7 @@ class VoxCPM2CFMRuntime::Impl { throw std::runtime_error("VoxCPM2 CFM received non-finite scalar input"); } const int64_t patch_elems = config.feat_dim * config.patch_size; - const int64_t mu_dim = config.dit.hidden_dim * (config.v1 ? 1 : 2); + const int64_t mu_dim = config.dit.hidden_dim * (has_fusion_proj ? 2 : 1); if (static_cast(mu.size()) != mu_dim) { throw std::runtime_error("VoxCPM2 CFM mu size mismatch"); } @@ -1077,12 +1251,18 @@ class VoxCPM2CFMRuntime::Impl { for (float &value : x) { value *= temperature; } + x = patch_major_to_channel_major(x); const std::vector cond = patch_major_to_channel_major(cond_patch); std::vector x_in(static_cast(2 * patch_elems), 0.0F); std::vector cond_in(static_cast(2 * patch_elems), 0.0F); - std::vector mu_in(static_cast(4 * config.dit.hidden_dim), - 0.0F); + const int64_t mu_elements = + has_fusion_proj ? 4 * config.dit.hidden_dim : 2 * config.dit.hidden_dim; + std::vector mu_in(static_cast(mu_elements), 0.0F); std::copy(mu.begin(), mu.end(), mu_in.begin()); + if (std::getenv("VOXCPM_TEST_BATCH_MU") != nullptr) { + std::copy(mu.begin(), mu.end(), + mu_in.begin() + static_cast(mu.size())); + } std::copy(cond.begin(), cond.end(), cond_in.begin()); std::copy(cond.begin(), cond.end(), cond_in.begin() + static_cast(patch_elems)); @@ -1127,6 +1307,29 @@ class VoxCPM2CFMRuntime::Impl { const auto estimator = estimator_.run(x_in, mu_in, cond_in, time_embedding, delta_embedding); const float scale = optimized_cfg_scale(estimator, patch_elems); + if (std::getenv("VOXCPM_DUMP_DPHI") != nullptr && + step == 2) { + double s0 = 0.0, s1 = 0.0, n0 = 0.0, n1 = 0.0; + std::vector combined(static_cast(patch_elems)); + double c2 = 0.0; + for (int64_t i = 0; i < patch_elems; ++i) { + const float p = estimator[static_cast(i)]; + const float m = estimator[static_cast(patch_elems + i)]; + const double d = static_cast(m) * scale + + cfg_value * (static_cast(p) - + static_cast(m) * scale); + combined[static_cast(i)] = d; + s0 += p; s1 += m; n0 += p * p; n1 += m * m; + c2 += d * d; + } + fprintf(stderr, + "[DUMP_DPHI] t=%.6f dt=%.6f pos_l2=%.6g neg_l2=%.6g " + "combined_l2=%.6g scale=%.6g combined[0..3]=%.6g %.6g %.6g %.6g\n", + static_cast(t), static_cast(dt), + std::sqrt(n0), std::sqrt(n1), std::sqrt(c2), + static_cast(scale), combined[0], combined[1], + combined[2], combined[3]); + } for (int64_t i = 0; i < patch_elems; ++i) { const size_t index = static_cast(i); const float positive = estimator[index]; @@ -1552,6 +1755,24 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { prefill_input.audio_mask.push_back(row.audio_mask ? 1.0F : 0.0F); } const auto prefill_output = prefill_.run(prefill_input); + if (std::getenv("VOXCPM_DUMP_PREFILL") != nullptr) { + auto dump_vec = [](const char *tag, const std::vector &v) { + double sum = 0.0; + double sum2 = 0.0; + for (float x : v) { + sum += x; + sum2 += static_cast(x) * x; + } + fprintf(stderr, "[DUMP_PREFILL] %s n=%zu sum=%.6g l2=%.6g", tag, + v.size(), sum, std::sqrt(sum2)); + for (size_t i = 0; i < std::min(8, v.size()); ++i) { + fprintf(stderr, " v[%zu]=%.6g", i, static_cast(v[i])); + } + fprintf(stderr, "\n"); + }; + dump_vec("lm_hidden", prefill_output.lm_hidden); + dump_vec("residual_hidden", prefill_output.residual_hidden); + } base_lm_.import_state(prefill_output.base_state); residual_lm_.import_state(prefill_output.residual_state); std::vector lm_hidden = prefill_output.lm_hidden; @@ -1578,14 +1799,45 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { for (int64_t index = 0; index < max_tokens; ++index) { const auto projected = projection_.run(lm_hidden, residual_hidden, zero_hidden); - const auto mu = assets_->config.v1 - ? add_dit_mu(projected.current_lm_dit_hidden, - projected.residual_dit_hidden) - : concat_dit_mu(projected.current_lm_dit_hidden, - projected.residual_dit_hidden); + if (index == 0 && std::getenv("VOXCPM_DUMP_PREFILL") != nullptr) { + auto dump_vec = [](const char *tag, const std::vector &v) { + double sum = 0.0; + double sum2 = 0.0; + for (float x : v) { + sum += x; + sum2 += static_cast(x) * x; + } + fprintf(stderr, "[DUMP_PREFILL] %s n=%zu sum=%.6g l2=%.6g", tag, + v.size(), sum, std::sqrt(sum2)); + for (size_t i = 0; i < std::min(8, v.size()); ++i) { + fprintf(stderr, " v[%zu]=%.6g", i, static_cast(v[i])); + } + fprintf(stderr, "\n"); + }; + dump_vec("lm_to_dit", projected.current_lm_dit_hidden); + dump_vec("res_to_dit", projected.residual_dit_hidden); + } + // Check if fusion_concat_proj weight exists and was loaded (not synthesized) + // This matches VoxCPM.cpp behavior which checks weight existence + // For V1 models, synthesized weights (Xavier init) should not count as present + const bool has_fusion_proj = + weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && + !weights_->assets().model_weights->is_synthesized("fusion_concat_proj.weight"); + const auto mu = has_fusion_proj + ? concat_dit_mu(projected.current_lm_dit_hidden, + projected.residual_dit_hidden) + : add_dit_mu(projected.current_lm_dit_hidden, + projected.residual_dit_hidden); const auto patch = cfm_.generate_patch( mu, prefix_cond, options.num_inference_steps, options.guidance_scale, options.seed, patch_noise_start, options.cfm_noise_file); + if (const char *patch_dump_path = std::getenv("VOXCPM_DUMP_PATCH")) { + FILE *patch_file = std::fopen(patch_dump_path, "ab"); + if (patch_file != nullptr) { + std::fwrite(patch.data(), sizeof(float), patch.size(), patch_file); + std::fclose(patch_file); + } + } patch_noise_start += static_cast(patch_elems); append_patch(result.generated_features, patch, patch_elems); ++result.generated_patches; @@ -1609,6 +1861,13 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { stop_class(projected.current_stop_logits) == 1) { break; } + if (std::getenv("VOXCPM1_LOG_STOP") != nullptr) { + const auto &sl = projected.current_stop_logits; + fprintf(stderr, "[stop_logits] pos=%lld pre stop0=%.5f stop1=%.5f\n", + static_cast(index), + sl.empty() ? 0.0 : static_cast(sl[0]), + sl.size() < 2 ? 0.0 : static_cast(sl[1])); + } const auto curr_embed = local_encoder_.encode_patch(patch); const auto next_lm = base_lm_.run_step(curr_embed).hidden; diff --git a/src/models/voxcpm2/minicpm_blocks.h b/src/models/voxcpm2/minicpm_blocks.h index 08ae4445..5a951822 100644 --- a/src/models/voxcpm2/minicpm_blocks.h +++ b/src/models/voxcpm2/minicpm_blocks.h @@ -164,6 +164,11 @@ minicpm_layer(engine::core::ModuleBuildContext &ctx, auto hidden = engine::modules::RMSNormModule( {config.hidden_size, config.rms_norm_eps, true, false}) .build(ctx, input, layer.input_norm); + if (std::getenv("VOXCPM_DUMP_NORM0") != nullptr && + ggml_nelements(hidden.tensor) == 10240) { + ggml_set_name(hidden.tensor, "dump_norm0"); + ggml_set_output(hidden.tensor); + } auto q = engine::modules::LinearModule( binding::linear_config(config.hidden_size, config.num_attention_heads * dim, false)) @@ -247,8 +252,15 @@ minicpm_transformer(engine::core::ModuleBuildContext &ctx, engine::core::TensorValue input, const engine::core::TensorValue &positions, const VoxCPM2MiniCPMWeights &weights, bool is_causal) { - for (const auto &layer : weights.layers) { - input = minicpm_layer(ctx, input, positions, layer, weights, is_causal); + for (size_t li = 0; li < weights.layers.size(); ++li) { + if (std::getenv("VOXCPM_DUMP_DECODER_LAYERS") != nullptr && li < 8) { + char name[32]; + snprintf(name, sizeof(name), "dump_layer_%zu", li); + ggml_set_name(input.tensor, name); + ggml_set_output(input.tensor); + } + input = minicpm_layer(ctx, input, positions, weights.layers[li], weights, + is_causal); } return engine::modules::RMSNormModule({weights.config.hidden_size, weights.config.rms_norm_eps, true, From 59d61857d3ae0a8f83d19d46cd830df9b49617b9 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Mon, 17 Aug 2026 16:56:39 +0200 Subject: [PATCH 03/14] =?UTF-8?q?#=20Commit:=20Fix=20VoxCPM1=20Voice=20Qua?= =?UTF-8?q?lity=20(Pure=20Noise=20=E2=86=92=20Intelligible=20Speech)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Fixed VoxCPM1 TTS producing pure noise by correcting tensor synthesis and shape validation issues. ## Changes - **`src/models/voxcpm2/assets.cpp`**: Only synthesize tensors missing from GGUF (not unconditionally). Fixed `feat_encoder.special_token` shape (1D vs 4D). Added relaxed rank handling in `set_backend_tensor()` for V1. - **`src/framework/assets/tensor_source.cpp`**: Added `relaxed_rank` parameter to `validate_expected_shape()` allowing shape mismatches when element counts match. ## Root Cause Synthesized (Xavier-initialized) tensors were used instead of learned checkpoint weights. The `is_synthesized()` check now correctly distinguishes true synthesized tensors (only `fusion_concat_proj` for V1) from loaded weights. ## Validation - VoxCPM1: 16kHz speech, RMS ~0.10-0.15 ✅ - VoxCPM2: 48kHz speech (no regression) ✅ - Embedding transpose: `[1024,73448]` → `[73448,1024]` ✅ - `has_fusion_proj=false` for V1 ✅ --- src/framework/assets/tensor_source.cpp | 32 +++++++++---- src/models/voxcpm2/assets.cpp | 64 +++++++++++++++++++------- 2 files changed, 71 insertions(+), 25 deletions(-) diff --git a/src/framework/assets/tensor_source.cpp b/src/framework/assets/tensor_source.cpp index 189ea251..49f25b8a 100644 --- a/src/framework/assets/tensor_source.cpp +++ b/src/framework/assets/tensor_source.cpp @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -71,12 +72,25 @@ core::TensorShape shape_from_dims(const std::vector & dims) { void validate_expected_shape( std::string_view name, const std::vector & actual_shape, - const std::optional> & expected_shape) { + const std::optional> & expected_shape, + bool relaxed_rank) { if (expected_shape.has_value() && actual_shape != *expected_shape) { - throw std::runtime_error("tensor shape mismatch for " + std::string(name)); + if (!relaxed_rank) { + throw std::runtime_error("tensor shape mismatch for " + std::string(name)); + } + int64_t expected_elems = 1; + for (const int64_t dim : *expected_shape) { + expected_elems *= dim; + } + int64_t actual_elems = 1; + for (const int64_t dim : actual_shape) { + actual_elems *= dim; + } + if (actual_elems != expected_elems) { + throw std::runtime_error("tensor element count mismatch for " + std::string(name)); + } } } - std::string lower_ascii(std::string_view value) { std::string out(value); for (char & ch : out) { @@ -565,7 +579,7 @@ class SafeTensorSource final : public TensorSource { if (info == nullptr) { throw std::runtime_error("missing tensor: " + std::string(name)); } - validate_expected_shape(name, info->shape, expected_shape); + validate_expected_shape(name, info->shape, expected_shape, false); const auto shape = shape_from_dims(expected_shape); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto [data, byte_size] = require_data_range(*info); @@ -594,7 +608,7 @@ class SafeTensorSource final : public TensorSource { std::string_view name, const std::optional> & expected_shape) const override { const auto tensor = require_tensor_data(name); - validate_expected_shape(name, tensor.metadata.shape, expected_shape); + validate_expected_shape(name, tensor.metadata.shape, expected_shape, false); const ggml_type type = ggml_type_for_tensor_dtype(tensor.metadata.dtype); const auto physical_shape = tensor.metadata.shape.empty() ? shape_from_dims({1}) @@ -803,7 +817,7 @@ class GgufTensorSource final : public TensorSource { TensorStorageType storage_type, const std::vector & expected_shape) const override { const auto & info = require_info(name); - validate_expected_shape(name, info.shape, expected_shape); + validate_expected_shape(name, info.shape, expected_shape, false); const auto shape = shape_from_dims(expected_shape); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto [data, byte_size] = require_data_range(info); @@ -831,7 +845,7 @@ class GgufTensorSource final : public TensorSource { std::string_view name, const std::optional> & expected_shape) const override { const auto tensor = require_tensor_data(name); - validate_expected_shape(name, tensor.metadata.shape, expected_shape); + validate_expected_shape(name, tensor.metadata.shape, expected_shape, false); const auto physical_shape = tensor.metadata.shape.empty() ? shape_from_dims({1}) : shape_from_dims(tensor.metadata.shape); @@ -1234,7 +1248,7 @@ TensorData TensorSource::require_tensor( const core::TensorShape shape = shape_from_dims(expected_shape); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto raw = require_tensor_data(name); - validate_expected_shape(name, raw.metadata.shape, expected_shape); + validate_expected_shape(name, raw.metadata.shape, expected_shape, false); if (raw_dtype_matches_ggml_type(raw.metadata.dtype, type)) { validate_raw_tensor_byte_size(name, shape, type, raw.bytes.size()); return TensorData{shape, type, raw.bytes}; @@ -1257,7 +1271,7 @@ TensorData TensorSource::require_tensor_as_shape( const core::TensorShape source_shape = shape_from_dims(expected); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto raw = require_tensor_data(name); - validate_expected_shape(name, raw.metadata.shape, expected); + validate_expected_shape(name, raw.metadata.shape, expected, false); if (raw.metadata.shape == std::vector(tensor_shape) && raw_dtype_matches_ggml_type(raw.metadata.dtype, type)) { validate_raw_tensor_byte_size(name, shape, type, raw.bytes.size()); diff --git a/src/models/voxcpm2/assets.cpp b/src/models/voxcpm2/assets.cpp index cc827af6..6f85bf2b 100644 --- a/src/models/voxcpm2/assets.cpp +++ b/src/models/voxcpm2/assets.cpp @@ -437,6 +437,23 @@ class TransformingTensorSource final : public assets::TensorSource { } } } + // V1 relaxed rank: if expected element count matches actual but shapes differ, + // fetch data without expected_shape and set manually + if (is_v1_) { + const auto source_meta = source_->require_metadata(route_it->second); + int64_t expected_elems = 1; + for (const int64_t dim : expected_shape) expected_elems *= dim; + int64_t actual_elems = 1; + for (const int64_t dim : source_meta.shape) actual_elems *= dim; + if (expected_elems == actual_elems && source_meta.shape != expected_shape) { + const auto values = source_->require_f32(route_it->second, std::nullopt); + const auto shape = make_tensor_shape(expected_shape); + const ggml_type type = engine::assets::ggml_type_for_tensor_storage( + engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); + engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, values, shape, type); + return; + } + } source_->set_backend_tensor(tensor, route_it->second, storage_type, expected_shape); } @@ -621,17 +638,24 @@ class TransformingTensorSource final : public assets::TensorSource { synthesized_tensors_["feat_encoder.diag"] = assets::TensorMetadata{"feat_encoder.diag", "F32", {feat_dim}}; - // feat_encoder.special_token (from token_embd) - synthesized_tensors_["feat_encoder.special_token"] = - assets::TensorMetadata{"feat_encoder.special_token", "F32", {1, 1, 1, encoder_hidden}}; + // feat_encoder.special_token: V1 GGUF stores as 1D [1024], model code handles reshaping + // Only synthesize if not present in GGUF + if (routes_.find("feat_encoder.special_token") == routes_.end()) { + synthesized_tensors_["feat_encoder.special_token"] = + assets::TensorMetadata{"feat_encoder.special_token", "F32", {encoder_hidden}}; + } // token_embd.extra_bias (from logit_scale or zeros) - synthesized_tensors_["token_embd.extra_bias"] = - assets::TensorMetadata{"token_embd.extra_bias", "F32", {lm_hidden}}; + if (routes_.find("token_embd.extra_bias") == routes_.end()) { + synthesized_tensors_["token_embd.extra_bias"] = + assets::TensorMetadata{"token_embd.extra_bias", "F32", {lm_hidden}}; + } // feat_encoder.merge (zeros) - synthesized_tensors_["feat_encoder.merge.weight"] = - assets::TensorMetadata{"feat_encoder.merge.weight", "F32", {encoder_hidden, feat_dim}}; + if (routes_.find("feat_encoder.merge.weight") == routes_.end()) { + synthesized_tensors_["feat_encoder.merge.weight"] = + assets::TensorMetadata{"feat_encoder.merge.weight", "F32", {encoder_hidden, feat_dim}}; + } // Identity SR-condition embeddings for V1 decoder blocks. VoxCPM1 // GGUFs contain no sr_cond_model tensors (no SR conditioning), but the @@ -644,18 +668,26 @@ class TransformingTensorSource final : public assets::TensorSource { vae.decoder_dim / (int64_t{1} << static_cast(i)); const std::string prefix = "decoder.sr_cond_model." + std::to_string(i + 2) + "."; - synthesized_tensors_[prefix + "scale_embed.weight"] = - assets::TensorMetadata{prefix + "scale_embed.weight", "F32", {1, input_channels}}; - synthesized_tensors_[prefix + "bias_embed.weight"] = - assets::TensorMetadata{prefix + "bias_embed.weight", "F32", {1, input_channels}}; + if (routes_.find(prefix + "scale_embed.weight") == routes_.end()) { + synthesized_tensors_[prefix + "scale_embed.weight"] = + assets::TensorMetadata{prefix + "scale_embed.weight", "F32", {1, input_channels}}; + } + if (routes_.find(prefix + "bias_embed.weight") == routes_.end()) { + synthesized_tensors_[prefix + "bias_embed.weight"] = + assets::TensorMetadata{prefix + "bias_embed.weight", "F32", {1, input_channels}}; + } } } // Missing projection weights for V1 (not in VoxCPM1 GGUF) - synthesized_tensors_["fusion_concat_proj.weight"] = - assets::TensorMetadata{"fusion_concat_proj.weight", "F32", {lm_hidden, lm_hidden * 2}}; - synthesized_tensors_["fusion_concat_proj.bias"] = - assets::TensorMetadata{"fusion_concat_proj.bias", "F32", {lm_hidden}}; + if (routes_.find("fusion_concat_proj.weight") == routes_.end()) { + synthesized_tensors_["fusion_concat_proj.weight"] = + assets::TensorMetadata{"fusion_concat_proj.weight", "F32", {lm_hidden, lm_hidden * 2}}; + } + if (routes_.find("fusion_concat_proj.bias") == routes_.end()) { + synthesized_tensors_["fusion_concat_proj.bias"] = + assets::TensorMetadata{"fusion_concat_proj.bias", "F32", {lm_hidden}}; + } } std::vector fold_weight_norm( @@ -840,7 +872,7 @@ void validate_weight_anchors(const VoxCPM2Assets & assets) { {config.lm.num_key_value_heads * config.lm.kv_channels, config.lm.hidden_size}); assets::require_tensor_shape(weights, "base_lm.layers.0.mlp.gate_proj.weight", {config.lm.intermediate_size, config.lm.hidden_size}); assets::require_tensor_shape(weights, "residual_lm.norm.weight", {config.lm.hidden_size}); - assets::require_tensor_shape(weights, "feat_encoder.special_token", {1, 1, 1, config.encoder.hidden_dim}); + require_vae_weight_v_shape(weights, "feat_encoder.special_token", {1, 1, 1, config.encoder.hidden_dim}, config.v1); assets::require_tensor_shape(weights, "feat_encoder.in_proj.weight", {config.encoder.hidden_dim, config.feat_dim}); assets::require_tensor_shape(weights, "feat_encoder.encoder.norm.weight", {config.encoder.hidden_dim}); assets::require_tensor_shape(weights, "feat_decoder.estimator.in_proj.weight", {config.dit.hidden_dim, config.feat_dim}); From 36567b772de50bed4c06e63c1883bb1f5cdb03ee Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Mon, 17 Aug 2026 22:19:39 +0200 Subject: [PATCH 04/14] # VoxCPM1 GGUF Self-Contained Loading MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Fix - Added GGUF metadata reading to `TensorSource` (tokenizer.ggml.*, voxcpm_*) - Created `VoxCPM1GgufTokenizer` + `load_voxcpm1_config_from_gguf()` for native GGUF loading - Added `VoxCPM2TokenizerWrapper` for dual JSON/GGUF tokenizer support - Updated `load_voxcpm2_assets()` to auto-detect/use GGUF metadata - Removed external JSON deps from `model_specs/voxcpm1.json` ## Test (ASR: sensevoice@11533) - VoxCPM1 0.5B: "This is a test run for the fix." ❌ (too fask) - VoxCPM1.5 1.5B: "I the touch for the." ❌ (too slow) ## Remaining Bugs 1. VoxCPM1 too fast (1.28s vs 2.5s) - early stop token 2. VoxCPM1.5 too slow (5.29s vs 2.5s) - arch diff --- CMakeLists.txt | 4 + .../engine/framework/assets/tensor_source.h | 29 ++ include/engine/models/voxcpm2/assets.h | 2 + include/engine/models/voxcpm2/config_gguf.h | 18 + .../engine/models/voxcpm2/tokenizer_gguf.h | 39 ++ .../engine/models/voxcpm2/tokenizer_text.h | 6 +- .../engine/models/voxcpm2/tokenizer_wrapper.h | 73 ++++ model_specs/voxcpm1.json | 26 +- src/framework/assets/tensor_source.cpp | 245 ++++++++++++ src/models/voxcpm2/assets.cpp | 30 +- src/models/voxcpm2/config_gguf.cpp | 157 ++++++++ src/models/voxcpm2/generator.cpp | 8 +- src/models/voxcpm2/tokenizer_gguf.cpp | 358 ++++++++++++++++++ src/models/voxcpm2/tokenizer_text.cpp | 1 + 14 files changed, 970 insertions(+), 26 deletions(-) create mode 100644 include/engine/models/voxcpm2/config_gguf.h create mode 100644 include/engine/models/voxcpm2/tokenizer_gguf.h create mode 100644 include/engine/models/voxcpm2/tokenizer_wrapper.h create mode 100644 src/models/voxcpm2/config_gguf.cpp create mode 100644 src/models/voxcpm2/tokenizer_gguf.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 01c8964a..40f57692 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -685,10 +685,12 @@ audiocpp_add_model(voxcpm2 SOURCES src/models/voxcpm2/assets.cpp src/models/voxcpm2/audiovae.cpp + src/models/voxcpm2/config_gguf.cpp src/models/voxcpm2/generator.cpp src/models/voxcpm2/loader.cpp src/models/voxcpm2/minicpm.cpp src/models/voxcpm2/session.cpp + src/models/voxcpm2/tokenizer_gguf.cpp src/models/voxcpm2/tokenizer_text.cpp INCLUDES engine/models/voxcpm2/loader.h @@ -700,10 +702,12 @@ audiocpp_add_model(voxcpm1 SOURCES src/models/voxcpm2/assets.cpp src/models/voxcpm2/audiovae.cpp + src/models/voxcpm2/config_gguf.cpp src/models/voxcpm2/generator.cpp src/models/voxcpm2/loader.cpp src/models/voxcpm2/minicpm.cpp src/models/voxcpm2/session.cpp + src/models/voxcpm2/tokenizer_gguf.cpp src/models/voxcpm2/tokenizer_text.cpp INCLUDES engine/models/voxcpm2/loader.h diff --git a/include/engine/framework/assets/tensor_source.h b/include/engine/framework/assets/tensor_source.h index d5c28bba..a1648a01 100644 --- a/include/engine/framework/assets/tensor_source.h +++ b/include/engine/framework/assets/tensor_source.h @@ -119,6 +119,35 @@ class TensorSource { std::initializer_list candidates) const; [[nodiscard]] virtual int64_t require_i64_scalar(std::string_view name) const = 0; [[nodiscard]] virtual bool is_synthesized(std::string_view name) const noexcept { return false; } + + // GGUF metadata access (optional, only implemented by GgufTensorSource) + [[nodiscard]] virtual std::optional optional_string(std::string_view key) const { + return std::nullopt; + } + [[nodiscard]] virtual std::optional optional_u32(std::string_view key) const { + return std::nullopt; + } + [[nodiscard]] virtual std::optional> optional_string_array(std::string_view key) const { + return std::nullopt; + } + [[nodiscard]] virtual std::optional> optional_i32_array(std::string_view key) const { + return std::nullopt; + } + [[nodiscard]] virtual std::optional> optional_f32_array(std::string_view key) const { + return std::nullopt; + } + [[nodiscard]] virtual std::string require_string(std::string_view key) const { + throw std::runtime_error("require_string not supported by this TensorSource"); + } + [[nodiscard]] virtual uint32_t require_u32(std::string_view key) const { + throw std::runtime_error("require_u32 not supported by this TensorSource"); + } + [[nodiscard]] virtual std::vector require_string_array(std::string_view key) const { + throw std::runtime_error("require_string_array not supported by this TensorSource"); + } + [[nodiscard]] virtual std::vector require_i32_array(std::string_view key) const { + throw std::runtime_error("require_i32_array not supported by this TensorSource"); + } }; [[nodiscard]] TensorStorageType parse_tensor_storage_type(std::string_view value); diff --git a/include/engine/models/voxcpm2/assets.h b/include/engine/models/voxcpm2/assets.h index 19581ed3..4e29d06a 100644 --- a/include/engine/models/voxcpm2/assets.h +++ b/include/engine/models/voxcpm2/assets.h @@ -2,6 +2,7 @@ #include "engine/framework/assets/resource_bundle.h" #include "engine/framework/assets/tensor_source.h" +#include "engine/models/voxcpm2/tokenizer_gguf.h" #include #include @@ -93,6 +94,7 @@ struct VoxCPM2Assets { VoxCPM2Config config; std::shared_ptr model_weights; std::shared_ptr audiovae_weights; + std::shared_ptr gguf_tokenizer; }; std::shared_ptr load_voxcpm2_assets(const std::filesystem::path & model_path, bool is_v1); diff --git a/include/engine/models/voxcpm2/config_gguf.h b/include/engine/models/voxcpm2/config_gguf.h new file mode 100644 index 00000000..c0867a6c --- /dev/null +++ b/include/engine/models/voxcpm2/config_gguf.h @@ -0,0 +1,18 @@ +#pragma once + +#include "engine/models/voxcpm2/assets.h" +#include "engine/framework/assets/tensor_source.h" + +#include +#include +#include + +namespace engine::models::voxcpm2 { + +// Load VoxCPM1 config from GGUF metadata +VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & source); + +// Check if GGUF has VoxCPM1 config metadata +bool has_voxcpm1_config_metadata(const engine::assets::TensorSource & source); + +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/include/engine/models/voxcpm2/tokenizer_gguf.h b/include/engine/models/voxcpm2/tokenizer_gguf.h new file mode 100644 index 00000000..d076e87a --- /dev/null +++ b/include/engine/models/voxcpm2/tokenizer_gguf.h @@ -0,0 +1,39 @@ +#pragma once + +#include "engine/models/voxcpm2/types.h" +#include "engine/framework/assets/tensor_source.h" + +#include +#include +#include + +namespace engine::models::voxcpm2 { + +// Forward declaration +struct VoxCPM2TextPrompt; + +// GGUF-native tokenizer that reads tokenizer metadata directly from GGUF +class VoxCPM1GgufTokenizer { +public: + struct Impl; + + explicit VoxCPM1GgufTokenizer(std::shared_ptr gguf_source); + + std::vector encode(const std::string & text) const; + VoxCPM2TextPrompt build_prompt(const std::string & text) const; + int32_t audio_start_token_id() const noexcept; + int32_t audio_end_token_id() const noexcept; + int32_t reference_audio_start_token_id() const noexcept; + int32_t reference_audio_end_token_id() const noexcept; + int32_t bos_token_id() const noexcept; + int32_t eos_token_id() const noexcept; + int32_t unk_token_id() const noexcept; + + // Check if the GGUF source has tokenizer metadata + static bool has_tokenizer_metadata(const engine::assets::TensorSource & source); + +private: + std::shared_ptr impl_; +}; + +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/include/engine/models/voxcpm2/tokenizer_text.h b/include/engine/models/voxcpm2/tokenizer_text.h index ae877b57..0cd3b7c8 100644 --- a/include/engine/models/voxcpm2/tokenizer_text.h +++ b/include/engine/models/voxcpm2/tokenizer_text.h @@ -1,6 +1,5 @@ #pragma once -#include "engine/models/voxcpm2/assets.h" #include "engine/models/voxcpm2/types.h" #include @@ -10,6 +9,9 @@ namespace engine::models::voxcpm2 { +// Forward declaration +struct VoxCPM2Assets; + class VoxCPM2TextTokenizer { public: struct Impl; @@ -27,4 +29,4 @@ class VoxCPM2TextTokenizer { std::shared_ptr impl_; }; -} // namespace engine::models::voxcpm2 +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/include/engine/models/voxcpm2/tokenizer_wrapper.h b/include/engine/models/voxcpm2/tokenizer_wrapper.h new file mode 100644 index 00000000..6c700cd1 --- /dev/null +++ b/include/engine/models/voxcpm2/tokenizer_wrapper.h @@ -0,0 +1,73 @@ +#pragma once + +#include "engine/models/voxcpm2/tokenizer_text.h" +#include "engine/models/voxcpm2/tokenizer_gguf.h" +#include "engine/models/voxcpm2/types.h" + +#include +#include + +namespace engine::models::voxcpm2 { + +// Wrapper that can hold either VoxCPM2TextTokenizer (JSON-based) or VoxCPM1GgufTokenizer (GGUF-based) +class VoxCPM2TokenizerWrapper { +public: + VoxCPM2TokenizerWrapper() = default; + explicit VoxCPM2TokenizerWrapper(std::shared_ptr tokenizer) + : tokenizer_(std::move(tokenizer)) {} + explicit VoxCPM2TokenizerWrapper(std::shared_ptr tokenizer) + : tokenizer_(std::move(tokenizer)) {} + + VoxCPM2TextPrompt build_prompt(const std::string & text) const { + if (std::holds_alternative>(tokenizer_)) { + return std::get>(tokenizer_)->build_prompt(text); + } else { + return std::get>(tokenizer_)->build_prompt(text); + } + } + + int32_t audio_start_token_id() const noexcept { + if (std::holds_alternative>(tokenizer_)) { + return std::get>(tokenizer_)->audio_start_token_id(); + } else { + return std::get>(tokenizer_)->audio_start_token_id(); + } + } + + int32_t audio_end_token_id() const noexcept { + if (std::holds_alternative>(tokenizer_)) { + return std::get>(tokenizer_)->audio_end_token_id(); + } else { + return std::get>(tokenizer_)->audio_end_token_id(); + } + } + + int32_t reference_audio_start_token_id() const noexcept { + if (std::holds_alternative>(tokenizer_)) { + return std::get>(tokenizer_)->reference_audio_start_token_id(); + } else { + return std::get>(tokenizer_)->reference_audio_start_token_id(); + } + } + + int32_t reference_audio_end_token_id() const noexcept { + if (std::holds_alternative>(tokenizer_)) { + return std::get>(tokenizer_)->reference_audio_end_token_id(); + } else { + return std::get>(tokenizer_)->reference_audio_end_token_id(); + } + } + + bool empty() const noexcept { + return std::holds_alternative(tokenizer_); + } + +private: + std::variant< + std::monostate, + std::shared_ptr, + std::shared_ptr + > tokenizer_; +}; + +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/model_specs/voxcpm1.json b/model_specs/voxcpm1.json index 905bd1df..e532cf7d 100644 --- a/model_specs/voxcpm1.json +++ b/model_specs/voxcpm1.json @@ -56,10 +56,7 @@ "precision": "q8_0", "target_directory": "VoxCPM1-GGUF", "files": [ - "VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf", - "VoxCPM1-GGUF/config.json", - "VoxCPM1-GGUF/tokenizer.json", - "VoxCPM1-GGUF/tokenizer_config.json" + "VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf" ], "strip_prefix": "VoxCPM1-GGUF" }, @@ -70,10 +67,7 @@ "precision": "q4_k", "target_directory": "VoxCPM1.5-GGUF", "files": [ - "VoxCPM1.5-GGUF/voxcpm1.5-q4_k-audiovae-f16.gguf", - "VoxCPM1.5-GGUF/config.json", - "VoxCPM1.5-GGUF/tokenizer.json", - "VoxCPM1.5-GGUF/tokenizer_config.json" + "VoxCPM1.5-GGUF/voxcpm1.5-q4_k-audiovae-f16.gguf" ], "strip_prefix": "VoxCPM1.5-GGUF" }, @@ -84,10 +78,7 @@ "precision": "q8_0", "target_directory": "VoxCPM1.5-GGUF", "files": [ - "VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf", - "VoxCPM1.5-GGUF/config.json", - "VoxCPM1.5-GGUF/tokenizer.json", - "VoxCPM1.5-GGUF/tokenizer_config.json" + "VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf" ], "strip_prefix": "VoxCPM1.5-GGUF" } @@ -99,13 +90,8 @@ "model": ".", "weights": "$gguf" }, - "files": { - "config": "model:config.json", - "tokenizer_config": "model:tokenizer_config.json", - "tokenizer_json": "model:tokenizer.json", - "special_tokens_map": "model:special_tokens_map.json" - }, -"tensors": { + "files": {}, + "tensors": { "weights": { "source": "weights:" }, @@ -115,4 +101,4 @@ } } ] -} +} \ No newline at end of file diff --git a/src/framework/assets/tensor_source.cpp b/src/framework/assets/tensor_source.cpp index 49f25b8a..a40509f9 100644 --- a/src/framework/assets/tensor_source.cpp +++ b/src/framework/assets/tensor_source.cpp @@ -691,6 +691,8 @@ class GgufTensorSource final : public TensorSource { public: explicit GgufTensorSource(std::filesystem::path path) : source_path_(std::filesystem::weakly_canonical(path)) { + // Read tokenizer metadata during initialization + read_metadata(path); ggml_context * tensor_context = nullptr; gguf_context * gguf = gguf_init_from_file( source_path_.string().c_str(), @@ -775,6 +777,7 @@ class GgufTensorSource final : public TensorSource { gguf_free(gguf); ggml_free(tensor_context); bytes_ = engine::io::read_binary_blob(source_path_); + read_metadata(source_path_); } const std::filesystem::path & source_path() const noexcept override { return source_path_; } @@ -894,6 +897,248 @@ class GgufTensorSource final : public TensorSource { std::vector infos_; std::unordered_map info_by_name_; mutable engine::io::BinaryBlob bytes_; + // Tokenizer metadata + std::optional tokenizer_model_; + std::optional tokenizer_pre_; + std::optional> tokenizer_tokens_; + std::optional> tokenizer_token_type_; + std::optional> tokenizer_merges_; + std::optional tokenizer_bos_token_id_; + std::optional tokenizer_eos_token_id_; + std::optional tokenizer_unknown_token_id_; + + // Config metadata (voxcpm_*) + std::unordered_map config_string_metadata_; + std::unordered_map config_u32_metadata_; + std::unordered_map> config_i32_array_metadata_; + std::unordered_map> config_f32_array_metadata_; + + void read_metadata(const std::filesystem::path & path) { + ggml_context * tensor_context = nullptr; + gguf_context * gguf = gguf_init_from_file( + path.string().c_str(), + gguf_init_params{true, &tensor_context}); + if (gguf == nullptr) { + if (tensor_context != nullptr) ggml_free(tensor_context); + return; + } + + const auto get_string = [&](const char* key) -> std::optional { + const int idx = gguf_find_key(gguf, key); + if (idx < 0 || gguf_get_kv_type(gguf, idx) != GGUF_TYPE_STRING) { + return std::nullopt; + } + const char* data = gguf_get_val_str(gguf, idx); + if (!data) return std::nullopt; + return std::string(data); + }; + + const auto get_u32 = [&](const char* key) -> std::optional { + const int idx = gguf_find_key(gguf, key); + if (idx < 0) return std::nullopt; + return gguf_get_val_u32(gguf, idx); + }; + + const auto get_f32 = [&](const char* key) -> std::optional { + const int idx = gguf_find_key(gguf, key); + if (idx < 0 || gguf_get_kv_type(gguf, idx) != GGUF_TYPE_FLOAT32) { + return std::nullopt; + } + return gguf_get_val_f32(gguf, idx); + }; + + const auto get_i32_array = [&](const char* key) -> std::optional> { + const int idx = gguf_find_key(gguf, key); + if (idx < 0) return std::nullopt; + const int32_t* data = static_cast(gguf_get_arr_data(gguf, idx)); + const size_t n = gguf_get_arr_n(gguf, idx); + if (!data && n != 0) return std::nullopt; + return std::vector(data, data + n); + }; + + const auto get_f32_array = [&](const char* key) -> std::optional> { + const int idx = gguf_find_key(gguf, key); + if (idx < 0 || gguf_get_arr_type(gguf, idx) != GGUF_TYPE_FLOAT32) { + return std::nullopt; + } + const float* data = static_cast(gguf_get_arr_data(gguf, idx)); + const size_t n = gguf_get_arr_n(gguf, idx); + if (!data && n != 0) return std::nullopt; + return std::vector(data, data + n); + }; + + const auto get_string_array = [&](const char* key) -> std::optional> { + const int idx = gguf_find_key(gguf, key); + if (idx < 0 || gguf_get_arr_type(gguf, idx) != GGUF_TYPE_STRING) { + return std::nullopt; + } + const size_t n = gguf_get_arr_n(gguf, idx); + std::vector values; + values.reserve(n); + for (size_t i = 0; i < n; ++i) { + const char* v = gguf_get_arr_str(gguf, idx, i); + values.emplace_back(v ? v : ""); + } + return values; + }; + + // Tokenizer metadata + tokenizer_model_ = get_string("tokenizer.ggml.model"); + tokenizer_pre_ = get_string("tokenizer.ggml.pre"); + tokenizer_tokens_ = get_string_array("tokenizer.ggml.tokens"); + tokenizer_token_type_ = get_i32_array("tokenizer.ggml.token_type"); + tokenizer_merges_ = get_string_array("tokenizer.ggml.merges"); + tokenizer_bos_token_id_ = get_u32("tokenizer.ggml.bos_token_id"); + tokenizer_eos_token_id_ = get_u32("tokenizer.ggml.eos_token_id"); + tokenizer_unknown_token_id_ = get_u32("tokenizer.ggml.unknown_token_id"); + + // Config metadata (voxcpm_*) - read all voxcpm_* keys + // We read known keys, but also could iterate all keys if needed + static constexpr const char* config_string_keys[] = { + "voxcpm_architecture", + "voxcpm_device", + "voxcpm_dtype", + "voxcpm_lm_config_rope_scaling_type", + "voxcpm_dit_config_cfm_config_solver", + "voxcpm_dit_config_cfm_config_t_scheduler", + }; + for (const char* key : config_string_keys) { + if (auto val = get_string(key)) { + config_string_metadata_[key] = *val; + } + } + + static constexpr const char* config_u32_keys[] = { + "voxcpm_lm_config_bos_token_id", + "voxcpm_lm_config_eos_token_id", + "voxcpm_lm_config_hidden_size", + "voxcpm_lm_config_intermediate_size", + "voxcpm_lm_config_max_position_embeddings", + "voxcpm_lm_config_num_attention_heads", + "voxcpm_lm_config_num_hidden_layers", + "voxcpm_lm_config_num_key_value_heads", + "voxcpm_lm_config_dim_model_base", + "voxcpm_lm_config_scale_emb", + "voxcpm_lm_config_rope_theta", + "voxcpm_lm_config_use_mup", + "voxcpm_lm_config_vocab_size", + "voxcpm_patch_size", + "voxcpm_feat_dim", + "voxcpm_residual_lm_num_layers", + "voxcpm_residual_lm_no_rope", + "voxcpm_scalar_quantization_latent_dim", + "voxcpm_scalar_quantization_scale", + "voxcpm_encoder_config_hidden_dim", + "voxcpm_encoder_config_ffn_dim", + "voxcpm_encoder_config_num_heads", + "voxcpm_encoder_config_num_layers", + "voxcpm_dit_config_hidden_dim", + "voxcpm_dit_config_ffn_dim", + "voxcpm_dit_config_num_heads", + "voxcpm_dit_config_num_layers", + "voxcpm_dit_config_mean_mode", + "voxcpm_audio_vae_config_encoder_dim", + "voxcpm_audio_vae_config_decoder_dim", + "voxcpm_audio_vae_config_latent_dim", + "voxcpm_audio_vae_config_sample_rate", + "voxcpm_audio_vae_config_out_sample_rate", + "voxcpm_max_length", + }; + for (const char* key : config_u32_keys) { + if (auto val = get_u32(key)) { + config_u32_metadata_[key] = *val; + } + } + + static constexpr const char* config_i32_array_keys[] = { + "voxcpm_audio_vae_config_encoder_rates", + "voxcpm_audio_vae_config_decoder_rates", + "voxcpm_audio_vae_config_sr_bin_boundaries", + }; + for (const char* key : config_i32_array_keys) { + if (auto val = get_i32_array(key)) { + config_i32_array_metadata_[key] = *val; + } + } + + static constexpr const char* config_f32_array_keys[] = { + "voxcpm_lm_config_rope_scaling_long_factor", + "voxcpm_lm_config_rope_scaling_short_factor", + }; + for (const char* key : config_f32_array_keys) { + if (auto val = get_f32_array(key)) { + config_f32_array_metadata_[key] = *val; + } + } + + gguf_free(gguf); + if (tensor_context != nullptr) ggml_free(tensor_context); + } + + // GGUF metadata access implementations + std::optional optional_string(std::string_view key) const override { + if (key == "tokenizer.ggml.model") return tokenizer_model_; + if (key == "tokenizer.ggml.pre") return tokenizer_pre_; + // Check config metadata + auto it = config_string_metadata_.find(std::string(key)); + if (it != config_string_metadata_.end()) return it->second; + return std::nullopt; + } + + std::optional optional_u32(std::string_view key) const override { + if (key == "tokenizer.ggml.bos_token_id") return tokenizer_bos_token_id_; + if (key == "tokenizer.ggml.eos_token_id") return tokenizer_eos_token_id_; + if (key == "tokenizer.ggml.unknown_token_id") return tokenizer_unknown_token_id_; + // Check config metadata + auto it = config_u32_metadata_.find(std::string(key)); + if (it != config_u32_metadata_.end()) return it->second; + return std::nullopt; + } + + std::optional> optional_string_array(std::string_view key) const override { + if (key == "tokenizer.ggml.tokens") return tokenizer_tokens_; + if (key == "tokenizer.ggml.merges") return tokenizer_merges_; + return std::nullopt; + } + + std::optional> optional_i32_array(std::string_view key) const override { + if (key == "tokenizer.ggml.token_type") return tokenizer_token_type_; + // Check config metadata + auto it = config_i32_array_metadata_.find(std::string(key)); + if (it != config_i32_array_metadata_.end()) return it->second; + return std::nullopt; + } + + std::optional> optional_f32_array(std::string_view key) const override { + // Check config metadata + auto it = config_f32_array_metadata_.find(std::string(key)); + if (it != config_f32_array_metadata_.end()) return it->second; + return std::nullopt; + } + + std::string require_string(std::string_view key) const override { + auto opt = optional_string(key); + if (opt) return *opt; + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); + } + + uint32_t require_u32(std::string_view key) const override { + auto opt = optional_u32(key); + if (opt) return *opt; + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); + } + + std::vector require_string_array(std::string_view key) const override { + auto opt = optional_string_array(key); + if (opt) return *opt; + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); + } + + std::vector require_i32_array(std::string_view key) const override { + auto opt = optional_i32_array(key); + if (opt) return *opt; + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); + } }; std::unordered_map parse_indexed_tensor_weight_map( diff --git a/src/models/voxcpm2/assets.cpp b/src/models/voxcpm2/assets.cpp index 6f85bf2b..b1fb03cb 100644 --- a/src/models/voxcpm2/assets.cpp +++ b/src/models/voxcpm2/assets.cpp @@ -1,4 +1,6 @@ #include "engine/models/voxcpm2/assets.h" +#include "engine/models/voxcpm2/tokenizer_gguf.h" +#include "engine/models/voxcpm2/config_gguf.h" #include "engine/framework/model_spec/package.h" #include "engine/framework/assets/resource_bundle.h" @@ -906,8 +908,32 @@ std::shared_ptr load_voxcpm2_assets(const std::filesystem:: out->resources = engine::model_spec::load_resource_bundle( model_path, engine::model_spec::default_spec_path(is_v1 ? "voxcpm1" : "voxcpm2")); - out->config = parse_config(out->resources); - out->config.v1 = is_v1; + + // For VoxCPM1, try to load config and tokenizer from GGUF metadata + if (is_v1) { + auto raw_model_weights = out->resources.open_tensor_source("weights"); + + // Check if GGUF has tokenizer metadata + bool has_tokenizer = VoxCPM1GgufTokenizer::has_tokenizer_metadata(*raw_model_weights); + bool has_config = has_voxcpm1_config_metadata(*raw_model_weights); + + if (has_tokenizer && has_config) { + // Load config from GGUF metadata + out->config = load_voxcpm1_config_from_gguf(*raw_model_weights); + out->config.v1 = true; + + // Create GGUF-native tokenizer + out->gguf_tokenizer = std::make_shared(raw_model_weights); + } else { + // Fall back to external files + out->config = parse_config(out->resources); + out->config.v1 = true; + } + } else { + out->config = parse_config(out->resources); + out->config.v1 = false; + } + auto raw_model_weights = out->resources.open_tensor_source("weights"); auto raw_audiovae_weights = out->resources.open_tensor_source("audiovae_weights"); if (is_v1) { diff --git a/src/models/voxcpm2/config_gguf.cpp b/src/models/voxcpm2/config_gguf.cpp new file mode 100644 index 00000000..eb961f94 --- /dev/null +++ b/src/models/voxcpm2/config_gguf.cpp @@ -0,0 +1,157 @@ +#include "engine/models/voxcpm2/config_gguf.h" + +#include "engine/framework/assets/tensor_source.h" + +#include +#include + +namespace engine::models::voxcpm2 { + +bool has_voxcpm1_config_metadata(const engine::assets::TensorSource & source) { + // Check for at least one VoxCPM1-specific metadata key + return source.optional_string("voxcpm_architecture").has_value() || + source.optional_string("voxcpm_lm_config_hidden_size").has_value() || + source.optional_u32("voxcpm_lm_config_hidden_size").has_value(); +} + +VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & source) { + VoxCPM2Config config; + config.v1 = true; + config.architecture = "voxcpm"; + + // Helper lambda to get optional i64 from GGUF metadata (via u32 or i64) + auto get_optional_i64 = [&source](const char * key) -> std::optional { + auto u32 = source.optional_u32(key); + if (u32) return static_cast(*u32); + // Try i64 scalar if it's a tensor + if (source.has_tensor(key)) { + try { + return source.require_i64_scalar(key); + } catch (...) { + // Not a scalar tensor + } + } + return std::nullopt; + }; + + // Helper lambda to get optional bool from GGUF metadata + auto get_optional_bool = [&source](const char * key) -> std::optional { + auto u32 = source.optional_u32(key); + if (u32) return *u32 != 0; + return std::nullopt; + }; + + // Helper lambda to get optional int64 array from GGUF metadata + auto get_optional_i64_array = [&source](const char * key) -> std::optional> { + auto i32_arr = source.optional_i32_array(key); + if (i32_arr) { + std::vector result; + result.reserve(i32_arr->size()); + for (int32_t v : *i32_arr) { + result.push_back(static_cast(v)); + } + return result; + } + return std::nullopt; + }; + + // Architecture + auto arch = source.optional_string("voxcpm_architecture"); + if (arch) config.architecture = *arch; + + // LM Config + config.lm.bos_token_id = get_optional_i64("voxcpm_lm_config_bos_token_id").value_or(1); + config.lm.eos_token_id = get_optional_i64("voxcpm_lm_config_eos_token_id").value_or(2); + config.lm.hidden_size = get_optional_i64("voxcpm_lm_config_hidden_size").value_or(1024); + config.lm.intermediate_size = get_optional_i64("voxcpm_lm_config_intermediate_size").value_or(4096); + config.lm.max_position_embeddings = get_optional_i64("voxcpm_lm_config_max_position_embeddings").value_or(2048); + config.lm.num_attention_heads = get_optional_i64("voxcpm_lm_config_num_attention_heads").value_or(16); + config.lm.num_hidden_layers = get_optional_i64("voxcpm_lm_config_num_hidden_layers").value_or(24); + config.lm.num_key_value_heads = get_optional_i64("voxcpm_lm_config_num_key_value_heads").value_or(16); + config.lm.kv_channels = get_optional_i64("voxcpm_lm_config_kv_channels").value_or(config.lm.hidden_size / config.lm.num_attention_heads); + config.lm.vocab_size = get_optional_i64("voxcpm_lm_config_vocab_size").value_or(73448); + config.lm.scale_emb = get_optional_i64("voxcpm_lm_config_scale_emb").value_or(1); + config.lm.dim_model_base = get_optional_i64("voxcpm_lm_config_dim_model_base").value_or(256); + config.lm.rms_norm_eps = 1e-5f; // Default, GGUF doesn't have native float + config.lm.rope_theta = 10000.0f; // Default + config.lm.scale_depth = 1.0f; // Default + config.lm.use_mup = get_optional_bool("voxcpm_lm_config_use_mup").value_or(false); + + // Rope scaling (longrope for VoxCPM1) + config.lm.rope_scaling.type = "longrope"; + // GGUF doesn't have native float arrays, use defaults + const int64_t head_dim = config.lm.hidden_size / config.lm.num_attention_heads; + const int64_t factor_size = head_dim / 2; + config.lm.rope_scaling.long_factor.assign(factor_size, 1.0f); + config.lm.rope_scaling.short_factor.assign(factor_size, 1.0f); + config.lm.rope_scaling.original_max_position_embeddings = + get_optional_i64("voxcpm_lm_config_rope_scaling_original_max_position_embeddings").value_or(2048); + + // Patch size + config.patch_size = get_optional_i64("voxcpm_patch_size").value_or(1); + + // Feature dimension + config.feat_dim = get_optional_i64("voxcpm_feat_dim").value_or(512); + + // Residual LM + config.residual_lm_num_layers = get_optional_i64("voxcpm_residual_lm_num_layers").value_or(6); + config.residual_lm_no_rope = get_optional_bool("voxcpm_residual_lm_no_rope").value_or(false); + + // Scalar quantization + config.scalar_quantization_latent_dim = get_optional_i64("voxcpm_scalar_quantization_latent_dim").value_or(8); + config.scalar_quantization_scale = get_optional_i64("voxcpm_scalar_quantization_scale").value_or(8); + + // Encoder config (local encoder) + config.encoder.hidden_dim = get_optional_i64("voxcpm_encoder_config_hidden_dim").value_or(512); + config.encoder.ffn_dim = get_optional_i64("voxcpm_encoder_config_ffn_dim").value_or(2048); + config.encoder.num_heads = get_optional_i64("voxcpm_encoder_config_num_heads").value_or(8); + config.encoder.num_layers = get_optional_i64("voxcpm_encoder_config_num_layers").value_or(4); + config.encoder.kv_channels = get_optional_i64("voxcpm_encoder_config_kv_channels").value_or(config.encoder.hidden_dim / config.encoder.num_heads); + + // DiT config (local DiT) + config.dit.hidden_dim = get_optional_i64("voxcpm_dit_config_hidden_dim").value_or(512); + config.dit.ffn_dim = get_optional_i64("voxcpm_dit_config_ffn_dim").value_or(2048); + config.dit.num_heads = get_optional_i64("voxcpm_dit_config_num_heads").value_or(8); + config.dit.num_layers = get_optional_i64("voxcpm_dit_config_num_layers").value_or(4); + config.dit.kv_channels = get_optional_i64("voxcpm_dit_config_kv_channels").value_or(config.dit.hidden_dim / config.dit.num_heads); + config.dit.mean_mode = get_optional_bool("voxcpm_dit_config_mean_mode").value_or(false); + config.dit.cfm.sigma_min = 1e-4f; // Default + config.dit.cfm.solver = "euler"; + config.dit.cfm.t_scheduler = "log-norm"; + config.dit.cfm.inference_cfg_rate = 0.5f; // Default + + // Audio VAE config + config.audio_vae.encoder_dim = get_optional_i64("voxcpm_audio_vae_config_encoder_dim").value_or(64); + config.audio_vae.encoder_rates = get_optional_i64_array("voxcpm_audio_vae_config_encoder_rates").value_or(std::vector{2, 2, 2, 2}); + config.audio_vae.latent_dim = get_optional_i64("voxcpm_audio_vae_config_latent_dim").value_or(512); + config.audio_vae.decoder_dim = get_optional_i64("voxcpm_audio_vae_config_decoder_dim").value_or(512); + config.audio_vae.decoder_rates = get_optional_i64_array("voxcpm_audio_vae_config_decoder_rates").value_or(std::vector{2, 2, 2, 2}); + config.audio_vae.sample_rate_bin_boundaries = get_optional_i64_array("voxcpm_audio_vae_config_sr_bin_boundaries").value_or(std::vector{}); + config.audio_vae.sample_rate = static_cast(get_optional_i64("voxcpm_audio_vae_config_sample_rate").value_or(16000)); + config.audio_vae.output_sample_rate = static_cast(get_optional_i64("voxcpm_audio_vae_config_out_sample_rate").value_or(16000)); + + // Max length + config.max_length = get_optional_i64("voxcpm_max_length").value_or(2048); + + // Device and dtype + config.device = source.optional_string("voxcpm_device").value_or("cpu"); + config.dtype = source.optional_string("voxcpm_dtype").value_or("fp16"); + + // Validate required fields + if (config.lm.hidden_size <= 0) { + throw std::runtime_error("voxcpm_lm_config_hidden_size must be positive"); + } + if (config.lm.vocab_size <= 0) { + throw std::runtime_error("voxcpm_lm_config_vocab_size must be positive"); + } + if (config.feat_dim != config.audio_vae.latent_dim) { + throw std::runtime_error("voxcpm_feat_dim must match voxcpm_audio_vae_config_latent_dim"); + } + if (config.residual_lm_num_layers > config.lm.num_hidden_layers) { + throw std::runtime_error("residual_lm_num_layers exceeds lm num_hidden_layers"); + } + + return config; +} + +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/src/models/voxcpm2/generator.cpp b/src/models/voxcpm2/generator.cpp index f79cf77a..0ac52dd7 100644 --- a/src/models/voxcpm2/generator.cpp +++ b/src/models/voxcpm2/generator.cpp @@ -15,6 +15,7 @@ #include "engine/models/voxcpm2/assets.h" #include "engine/models/voxcpm2/minicpm.h" #include "engine/models/voxcpm2/tokenizer_text.h" +#include "engine/models/voxcpm2/tokenizer_wrapper.h" #include #include @@ -1424,7 +1425,10 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { weights_(std::make_shared( assets_, execution_context, config.weight_context_bytes, config.weight_storage_type)), - tokenizer_(assets_), + tokenizer_(assets_->gguf_tokenizer + ? VoxCPM2TokenizerWrapper(assets_->gguf_tokenizer) + : VoxCPM2TokenizerWrapper( + std::make_shared(assets_))), text_embedding_(weights_, config.text_embedding_graph_context_bytes, config.mem_saver), prefill_(weights_, config.lm_step_graph_context_bytes, @@ -1882,7 +1886,7 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { std::shared_ptr assets_; std::shared_ptr weights_; - VoxCPM2TextTokenizer tokenizer_; + VoxCPM2TokenizerWrapper tokenizer_; VoxCPM2TextEmbeddingRuntime text_embedding_; VoxCPM2PromptPrefillRuntime prefill_; VoxCPM2MiniCPMStepRuntime base_lm_; diff --git a/src/models/voxcpm2/tokenizer_gguf.cpp b/src/models/voxcpm2/tokenizer_gguf.cpp new file mode 100644 index 00000000..e849fff9 --- /dev/null +++ b/src/models/voxcpm2/tokenizer_gguf.cpp @@ -0,0 +1,358 @@ +#include "engine/models/voxcpm2/tokenizer_gguf.h" + +#include "engine/framework/assets/tensor_source.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace engine::models::voxcpm2 { +namespace { + +// UTF-8 handling functions (copied from tokenizer_text.cpp) +uint32_t next_utf8_codepoint(std::string_view text, size_t & offset) { + if (offset >= text.size()) { + throw std::runtime_error("VoxCPM2 tokenizer UTF-8 offset is out of range"); + } + const unsigned char first = static_cast(text[offset]); + uint32_t codepoint = 0; + size_t len = 1; + if ((first & 0x80U) == 0) { + codepoint = first; + } else if ((first & 0xE0U) == 0xC0U) { + len = 2; + codepoint = first & 0x1FU; + } else if ((first & 0xF0U) == 0xE0U) { + len = 3; + codepoint = first & 0x0FU; + } else if ((first & 0xF8U) == 0xF0U) { + len = 4; + codepoint = first & 0x07U; + } else { + throw std::runtime_error("VoxCPM2 tokenizer encountered invalid UTF-8"); + } + if (offset + len > text.size()) { + throw std::runtime_error("VoxCPM2 tokenizer encountered truncated UTF-8"); + } + for (size_t i = 1; i < len; ++i) { + const unsigned char ch = static_cast(text[offset + i]); + if ((ch & 0xC0U) != 0x80U) { + throw std::runtime_error("VoxCPM2 tokenizer encountered invalid UTF-8 continuation"); + } + codepoint = (codepoint << 6U) | (ch & 0x3FU); + } + offset += len; + return codepoint; +} + +std::vector utf8_codepoints(std::string_view text) { + std::vector out; + for (size_t offset = 0; offset < text.size();) { + const size_t start = offset; + (void) next_utf8_codepoint(text, offset); + out.emplace_back(text.substr(start, offset - start)); + } + return out; +} + +std::string normalize_text(std::string_view text) { + const std::string space = "\xE2\x96\x81"; + std::string out = space; + for (char ch : text) { + if (ch == ' ') { + out += space; + } else { + out.push_back(ch); + } + } + return out; +} + +std::string byte_fallback_token(unsigned char byte) { + constexpr char kHex[] = "0123456789ABCDEF"; + std::string out = "<0x"; + out.push_back(kHex[(byte >> 4U) & 0x0FU]); + out.push_back(kHex[byte & 0x0FU]); + out.push_back('>'); + return out; +} + +std::vector bpe_initial_pieces( + std::string_view normalized_text, + const std::unordered_map & vocab) { + std::vector pieces; + for (size_t offset = 0; offset < normalized_text.size();) { + const size_t start = offset; + (void) next_utf8_codepoint(normalized_text, offset); + std::string piece(normalized_text.substr(start, offset - start)); + if (vocab.find(piece) != vocab.end()) { + pieces.push_back(std::move(piece)); + continue; + } + for (size_t i = start; i < offset; ++i) { + pieces.push_back(byte_fallback_token(static_cast(normalized_text[i]))); + } + } + return pieces; +} + +bool is_cjk_codepoint(uint32_t codepoint) { + return (codepoint >= 0x4E00 && codepoint <= 0x9FFF) || + (codepoint >= 0x3400 && codepoint <= 0x4DBF) || + (codepoint >= 0xF900 && codepoint <= 0xFAFF) || + (codepoint >= 0x20000 && codepoint <= 0x2A6DF); +} + +bool is_pure_multichar_cjk(std::string_view text) { + size_t count = 0; + for (size_t offset = 0; offset < text.size();) { + if (!is_cjk_codepoint(next_utf8_codepoint(text, offset))) { + return false; + } + ++count; + } + return count >= 2; +} + +std::string strip_sentencepiece_prefix(std::string token) { + const std::string prefix = "\xE2\x96\x81"; + size_t pos = 0; + while ((pos = token.find(prefix, pos)) != std::string::npos) { + token.erase(pos, prefix.size()); + } + return token; +} + +bool starts_with_at(std::string_view text, size_t pos, std::string_view prefix) { + return pos + prefix.size() <= text.size() && text.substr(pos, prefix.size()) == prefix; +} + +std::string pair_key(const std::string & left, const std::string & right) { + std::string key = left; + key.push_back('\0'); + key += right; + return key; +} + +} // namespace + +struct VoxCPM1GgufTokenizer::Impl { + std::unordered_map vocab; + std::unordered_map id_to_token; + std::unordered_map special_tokens; + std::unordered_map merge_ranks; + std::unordered_map> cjk_split_map; + int32_t audio_start_token_id = 101; + int32_t audio_end_token_id = 102; + int32_t reference_audio_start_token_id = 103; + int32_t reference_audio_end_token_id = 104; + int32_t bos_token_id_ = 1; + int32_t eos_token_id_ = 2; + int32_t unk_token_id_ = 3; + + std::vector bpe(std::string_view normalized_text) const { + std::vector word = bpe_initial_pieces(normalized_text, vocab); + if (word.size() <= 1) { + return word; + } + while (true) { + int32_t best_rank = std::numeric_limits::max(); + size_t best_index = word.size(); + for (size_t i = 0; i + 1 < word.size(); ++i) { + const auto it = merge_ranks.find(pair_key(word[i], word[i + 1])); + if (it != merge_ranks.end() && it->second < best_rank) { + best_rank = it->second; + best_index = i; + } + } + if (best_index == word.size()) { + break; + } + word[best_index] += word[best_index + 1]; + word.erase(word.begin() + static_cast(best_index + 1)); + if (word.size() <= 1) { + break; + } + } + return word; + } + + void append_expanded_id(std::vector & ids, int32_t id) const { + const auto split = cjk_split_map.find(id); + if (split == cjk_split_map.end()) { + ids.push_back(id); + return; + } + ids.insert(ids.end(), split->second.begin(), split->second.end()); + } +}; + +VoxCPM1GgufTokenizer::VoxCPM1GgufTokenizer(std::shared_ptr gguf_source) { + if (!gguf_source) { + throw std::runtime_error("VoxCPM1 GGUF tokenizer requires a valid GGUF tensor source"); + } + impl_ = std::make_shared(); + auto & impl = *impl_; + + // Read tokenizer metadata from GGUF directly in constructor + const std::string tokenizer_model = gguf_source->require_string("tokenizer.ggml.model"); + const std::string tokenizer_pre = gguf_source->require_string("tokenizer.ggml.pre"); + const std::vector tokens = gguf_source->require_string_array("tokenizer.ggml.tokens"); + const std::vector token_types = gguf_source->require_i32_array("tokenizer.ggml.token_type"); + const std::vector merges = gguf_source->require_string_array("tokenizer.ggml.merges"); + const uint32_t bos_id = gguf_source->require_u32("tokenizer.ggml.bos_token_id"); + const uint32_t eos_id = gguf_source->require_u32("tokenizer.ggml.eos_token_id"); + const uint32_t unk_id = gguf_source->require_u32("tokenizer.ggml.unknown_token_id"); + + if (tokenizer_model != "gpt2" || tokens.empty() || merges.empty() || token_types.size() != tokens.size()) { + throw std::runtime_error("Invalid VoxCPM1 GGUF tokenizer metadata"); + } + + constexpr int32_t kTokenTypeNormal = 1; + constexpr int32_t kTokenTypeByte = 6; + + for (size_t i = 0; i < tokens.size(); ++i) { + const int32_t id = static_cast(i); + impl.vocab.emplace(tokens[i], id); + impl.id_to_token.emplace(id, tokens[i]); + if (token_types[i] != kTokenTypeNormal && token_types[i] != kTokenTypeByte) { + impl.special_tokens.emplace(tokens[i], id); + } + } + + impl.bos_token_id_ = static_cast(bos_id); + impl.eos_token_id_ = static_cast(eos_id); + impl.unk_token_id_ = static_cast(unk_id); + + // Build merge ranks + int32_t rank = 0; + for (const std::string & merge_text : merges) { + const size_t split = merge_text.find(' '); + if (split == std::string::npos) { + ++rank; + continue; + } + const std::string left = merge_text.substr(0, split); + const std::string right = merge_text.substr(split + 1); + const auto left_it = impl.vocab.find(left); + const auto right_it = impl.vocab.find(right); + const auto merged_it = impl.vocab.find(left + right); + if (left_it != impl.vocab.end() && right_it != impl.vocab.end() && merged_it != impl.vocab.end()) { + impl.merge_ranks.emplace(pair_key(left, right), rank); + } + ++rank; + } + + if (impl.merge_ranks.empty()) { + throw std::runtime_error("VoxCPM1 GGUF tokenizer has no valid merge rules"); + } + + // Build CJK split map + for (const auto & [id, token] : impl.id_to_token) { + const std::string clean = strip_sentencepiece_prefix(token); + if (!is_pure_multichar_cjk(clean)) { + continue; + } + std::vector char_ids; + for (const auto & ch : utf8_codepoints(clean)) { + const auto it = impl.vocab.find(ch); + if (it == impl.vocab.end()) { + char_ids.clear(); + break; + } + char_ids.push_back(it->second); + } + if (!char_ids.empty()) { + impl.cjk_split_map.emplace(id, std::move(char_ids)); + } + } +} + +std::vector VoxCPM1GgufTokenizer::encode(const std::string & text) const { + std::vector ids; + for (size_t i = 0; i < text.size();) { + const auto special_it = std::find_if( + impl_->special_tokens.begin(), + impl_->special_tokens.end(), + [&](const auto & item) { return starts_with_at(text, i, item.first); }); + if (special_it != impl_->special_tokens.end()) { + impl_->append_expanded_id(ids, special_it->second); + i += special_it->first.size(); + continue; + } + + size_t next_special = text.size(); + for (const auto & [special, _] : impl_->special_tokens) { + const size_t pos = text.find(special, i); + if (pos != std::string::npos) { + next_special = std::min(next_special, pos); + } + } + const std::string normalized = normalize_text(std::string_view( + text.data() + static_cast(i), + next_special - i)); + for (const auto & bpe_token : impl_->bpe(normalized)) { + const auto vocab_it = impl_->vocab.find(bpe_token); + if (vocab_it == impl_->vocab.end()) { + throw std::runtime_error("VoxCPM1 tokenizer produced token not present in vocab: " + bpe_token); + } + impl_->append_expanded_id(ids, vocab_it->second); + } + i = next_special; + } + return ids; +} + +VoxCPM2TextPrompt VoxCPM1GgufTokenizer::build_prompt(const std::string & text) const { + if (text.empty()) { + throw std::runtime_error("VoxCPM1 requires non-empty text input"); + } + VoxCPM2TextPrompt prompt; + prompt.text = text; + prompt.input_ids = encode(text); + if (prompt.input_ids.empty()) { + throw std::runtime_error("VoxCPM1 tokenizer produced no tokens"); + } + return prompt; +} + +int32_t VoxCPM1GgufTokenizer::audio_start_token_id() const noexcept { + return impl_->audio_start_token_id; +} + +int32_t VoxCPM1GgufTokenizer::audio_end_token_id() const noexcept { + return impl_->audio_end_token_id; +} + +int32_t VoxCPM1GgufTokenizer::reference_audio_start_token_id() const noexcept { + return impl_->reference_audio_start_token_id; +} + +int32_t VoxCPM1GgufTokenizer::reference_audio_end_token_id() const noexcept { + return impl_->reference_audio_end_token_id; +} + +int32_t VoxCPM1GgufTokenizer::bos_token_id() const noexcept { + return impl_->bos_token_id_; +} + +int32_t VoxCPM1GgufTokenizer::eos_token_id() const noexcept { + return impl_->eos_token_id_; +} + +int32_t VoxCPM1GgufTokenizer::unk_token_id() const noexcept { + return impl_->unk_token_id_; +} + +bool VoxCPM1GgufTokenizer::has_tokenizer_metadata(const engine::assets::TensorSource & source) { + return source.optional_string("tokenizer.ggml.model").has_value() && + source.optional_string_array("tokenizer.ggml.tokens").has_value() && + source.optional_string_array("tokenizer.ggml.merges").has_value(); +} + +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/src/models/voxcpm2/tokenizer_text.cpp b/src/models/voxcpm2/tokenizer_text.cpp index 8f49c619..ecdc62e6 100644 --- a/src/models/voxcpm2/tokenizer_text.cpp +++ b/src/models/voxcpm2/tokenizer_text.cpp @@ -1,4 +1,5 @@ #include "engine/models/voxcpm2/tokenizer_text.h" +#include "engine/models/voxcpm2/assets.h" #include "engine/framework/io/json.h" From bc80b8b3cc3c13760f83db7473cac3665dfb0123 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Tue, 18 Aug 2026 01:52:27 +0200 Subject: [PATCH 05/14] Fix VoxCPM1 issues: sample rate (V1.5) and early stopping (V1 0.5B) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - config_gguf.cpp: output_sample_rate now falls back to sample_rate (not 16000) VoxCPM1.5 GGUF has sample_rate=44100 but no out_sample_rate → was defaulting to 16kHz - session.cpp: add V1-specific default min_tokens to prevent early stop token trigger VoxCPM1 (patch_size=2): min_tokens=20, VoxCPM1.5 (patch_size=4): min_tokens=12 Without this, stop token triggers at ~2 tokens causing 1.28s cutoff - Stop predictor weights correctly loaded via V1 relaxed rank (no transpose needed) GGUF stores [1024,2] (GGML), expected logical [2,1024] → to_ggml_dims → [1024,2] ✓ Results: VoxCPM1 (0.5B): durations scale 1.76s→4.32s with text length VoxCPM1.5 (1.5B): durations scale 2.56s→5.12s, correct 44.1kHz sample rate VoxCPM2: regression passes (48kHz, 1.28s) Files: config_gguf.cpp (+6), session.cpp (+14) --- src/models/voxcpm2/config_gguf.cpp | 8 ++++++-- src/models/voxcpm2/session.cpp | 14 ++++++++++++++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/src/models/voxcpm2/config_gguf.cpp b/src/models/voxcpm2/config_gguf.cpp index eb961f94..37255586 100644 --- a/src/models/voxcpm2/config_gguf.cpp +++ b/src/models/voxcpm2/config_gguf.cpp @@ -127,8 +127,12 @@ VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & config.audio_vae.decoder_dim = get_optional_i64("voxcpm_audio_vae_config_decoder_dim").value_or(512); config.audio_vae.decoder_rates = get_optional_i64_array("voxcpm_audio_vae_config_decoder_rates").value_or(std::vector{2, 2, 2, 2}); config.audio_vae.sample_rate_bin_boundaries = get_optional_i64_array("voxcpm_audio_vae_config_sr_bin_boundaries").value_or(std::vector{}); - config.audio_vae.sample_rate = static_cast(get_optional_i64("voxcpm_audio_vae_config_sample_rate").value_or(16000)); - config.audio_vae.output_sample_rate = static_cast(get_optional_i64("voxcpm_audio_vae_config_out_sample_rate").value_or(16000)); + auto sample_rate_opt = get_optional_i64("voxcpm_audio_vae_config_sample_rate"); + config.audio_vae.sample_rate = static_cast(sample_rate_opt.value_or(16000)); + config.audio_vae.output_sample_rate = static_cast( + get_optional_i64("voxcpm_audio_vae_config_out_sample_rate") + .value_or(sample_rate_opt.value_or(16000)) + ); // Max length config.max_length = get_optional_i64("voxcpm_max_length").value_or(2048); diff --git a/src/models/voxcpm2/session.cpp b/src/models/voxcpm2/session.cpp index 780ae8e9..21375fc4 100644 --- a/src/models/voxcpm2/session.cpp +++ b/src/models/voxcpm2/session.cpp @@ -480,9 +480,23 @@ const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( const runtime::TaskRequest &request) const { VoxCPM2GenerationOptions options; + bool min_tokens_explicit = false; if (const auto value = runtime::parse_i64_option( request.options, {"voxcpm2.min_tokens", "min_tokens"})) { options.min_tokens = *value; + min_tokens_explicit = true; + } + // Set V1-specific default min_tokens if not explicitly provided + if (!min_tokens_explicit && assets_->config.v1) { + // VoxCPM1 (0.5B) has patch_size=2, VoxCPM1.5 (1.5B) has patch_size=4 + // Use model-specific defaults for optimal speech speed + if (assets_->config.patch_size == 2) { + options.min_tokens = 20; // 20, VoxCPM1 0.5B + } else if (assets_->config.patch_size == 4) { + options.min_tokens = 12; // VoxCPM1.5 1.5B + } else { + options.min_tokens = 15; // Other V1 models + } } if (const auto value = runtime::parse_i64_option( request.options, {"max_tokens", "voxcpm2.max_tokens"})) { From 85e5859da569488cfecc4392681a32cf1670ed29 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Tue, 18 Aug 2026 12:58:34 +0200 Subject: [PATCH 06/14] **feat(voxcpm1): enable voice clone & streaming support (parity with VoxCPM2)** MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit VoxCPM1 (0.5B/1.5B) models now support voice cloning (`--voice-ref`) and streaming output (`--mode streaming`), matching the VoxCPM2 feature surface. The inference math was already shared; this unblocks the capability/option/reporting layer. **Root causes fixed (5 gaps):** - Capability advertisement: now exposes `Tts + {Offline, Streaming}` for V1 (was TTS-only) - Family identity: `family_impl()` returns `"voxcpm1"` for V1 models (was hardcoded `"voxcpm2"`) - Session options: `normalize_v1_session_options()` rewrites `voxcpm1.*` → `voxcpm2.*` keys so aliases work - Request options: added `voxcpm1.*` aliases for all params (`prompt_text`, `min_tokens`, `guidance_scale`, `retry_badcase`, etc.) - Model spec: `voxcpm1.json` adds `streaming` mode, correct sample rates (16kHz/44.1kHz) **Changes:** 7 files, +167/−32 lines - `src/models/voxcpm2/session.cpp` — option normalization, family-aware errors, request-option aliases - `src/models/voxcpm2/loader.cpp` — capability advertisement, family-labeled errors - `model_specs/voxcpm1.json` — streaming mode, tags, corrected description - `docs/tts.md` — V1 streaming/voice-clone examples, `retry_badcase=false` requirement - `tools/audiocpp_cli/audiocpp_cli_path_cases.json` — 3 new V1 path tests - `webui/configs/models_catalog.json` + `model_params.json` — V1 WebUI entries **Verified (CPU):** | Test | Result | |------|--------| | V1 offline TTS | `family=voxcpm1` ✓ | | V1 voice clone | 16kHz, 5.12s, RMS 0.115 ✓ | | V1 streaming | 40×1280 chunks, 16kHz ✓ | | V1 `voxcpm1.*` session/request options | accepted & applied ✓ | | V1 capability inspection | `modes=offline,streaming` ✓ | | V2 regression (offline/streaming) | 48kHz, parity maintained ✓ | Streaming requires `retry_badcase=false` (same as V2, pre-existing design). No V2 behavior changes. **Issue**: The audio quality is still bad --- docs/tts.md | 20 +- model_specs/voxcpm1.json | 11 +- src/models/voxcpm2/loader.cpp | 11 +- src/models/voxcpm2/session.cpp | 81 +- src/models/voxcpm2/session.cpp~ | 708 ++++++++++++++++++ .../audiocpp_cli/audiocpp_cli_path_cases.json | 68 ++ webui/configs/model_params.json | 7 + webui/configs/models_catalog.json | 1 + 8 files changed, 875 insertions(+), 32 deletions(-) create mode 100644 src/models/voxcpm2/session.cpp~ diff --git a/docs/tts.md b/docs/tts.md index e77ccd8f..92767fe9 100644 --- a/docs/tts.md +++ b/docs/tts.md @@ -402,16 +402,16 @@ audiocpp_cli --task tts --family pocket_tts --model models/pocket-tts --backend ## VoxCPM1 -VoxCPM1 supports offline TTS. It reuses the VoxCPM2 runtime tree with a GGUF tensor-adaptation layer that understands the OpenBMB folded AudioVAE weights, so the same `--family voxcpm1` path serves both the 16 kHz 0.5B model and the 44.1 kHz 1.5B variants. +VoxCPM1 supports offline and streaming TTS plus short-reference voice cloning. It reuses the VoxCPM2 runtime tree with a GGUF tensor-adaptation layer that understands the OpenBMB folded AudioVAE weights, so the same `--family voxcpm1` path serves both the 16 kHz 0.5B model and the 44.1 kHz 1.5B variants. | Field | Value | |---|---| | Family | `voxcpm1` | | Model directory | `models/VoxCPM1-GGUF` (0.5B), `models/VoxCPM1.5-GGUF` (1.5B) | | Task | `tts` | -| Modes | `offline` | +| Modes | `offline`, `streaming` | | Languages | Model auto-handles supported languages | -| Voice input | Optional reference WAV | +| Voice input | Optional reference WAV; optional transcript through `--reference-text` | | Built-in voices | Not exposed | Text to speech: @@ -426,9 +426,23 @@ audiocpp_cli --task tts --family voxcpm1 --model models/VoxCPM1-GGUF/voxcpm-0.5b audiocpp_cli --task tts --family voxcpm1 --model models/VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf --backend cpu --text "Hello from VoxCPM1." --out out.wav ``` +Voice clone: + +```bash +audiocpp_cli --task tts --family voxcpm1 --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf --backend cpu --text "Hello from VoxCPM1." --voice-ref assets/resources/b.wav --out out.wav +``` + +Streaming output: + +```bash +audiocpp_cli --task tts --family voxcpm1 --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf --backend cpu --mode streaming --text "Hello from VoxCPM1." --request-option retry_badcase=false --out out.wav +``` + | Option | Values | Default | Meaning | |---|---:|---:|---| | `--voice-ref` | WAV path | not set | Reference speaker audio. | +| `--reference-text` | text | empty string | Transcript for the reference audio (clone prompting). | +| `--mode` | `offline`, `streaming` | `offline` | Full-output or streaming run mode; streaming requires `retry_badcase=false`. | | `--session-option voxcpm1.mem_saver=true\|false` | bool | `false` | Use tighter graph workspaces and release MiniCPM/AudioVAE request graphs after completion to reduce resident VRAM. | | `--session-option voxcpm1.prompt_cache_slots=` | integer | `1` | Prompt and prompt-audio embedding cache slots. Set to `0` to disable prompt caching. | | `--max-tokens` | integer | `4096` | Maximum generated AR tokens. | diff --git a/model_specs/voxcpm1.json b/model_specs/voxcpm1.json index e532cf7d..ee9d0ebe 100644 --- a/model_specs/voxcpm1.json +++ b/model_specs/voxcpm1.json @@ -1,7 +1,7 @@ { "family": "voxcpm1", "display_name": "VoxCPM1", - "description": "OpenBMB VoxCPM 0.5B and 1.5B tokenizer-free TTS models with 24kHz output.", + "description": "OpenBMB VoxCPM 0.5B and 1.5B tokenizer-free TTS models supporting short-reference voice cloning and streaming output (16kHz for 0.5B, 44.1kHz for 1.5B).", "category": "tts", "status": "supported", "tasks": [ @@ -9,7 +9,8 @@ "clone" ], "modes": [ - "offline" + "offline", + "streaming" ], "languages": [ "zh", @@ -24,7 +25,8 @@ }, "runtime": { "tags": [ - "gguf" + "gguf", + "stream" ] }, "ui": { @@ -32,7 +34,8 @@ "tags": [ "TTS", "Clone", - "GGUF" + "GGUF", + "Stream" ], "docs": [ "docs/tts.md", diff --git a/src/models/voxcpm2/loader.cpp b/src/models/voxcpm2/loader.cpp index 33df9175..3754b0a6 100644 --- a/src/models/voxcpm2/loader.cpp +++ b/src/models/voxcpm2/loader.cpp @@ -105,7 +105,7 @@ runtime::CapabilitySet capabilities_v1(const VoxCPM2Assets &) { runtime::CapabilitySet out; out.supported_tasks = { {runtime::VoiceTaskKind::Tts, - {runtime::RunMode::Offline}}, + {runtime::RunMode::Offline, runtime::RunMode::Streaming}}, }; out.languages = {"Auto"}; out.supports_speaker_reference = true; @@ -150,7 +150,7 @@ class VoxCPM1Loader final : public runtime::IVoiceModelLoader { runtime::CapabilitySet out; out.supported_tasks = { {runtime::VoiceTaskKind::Tts, - {runtime::RunMode::Offline}}, + {runtime::RunMode::Offline, runtime::RunMode::Streaming}}, }; out.supports_speaker_reference = true; return out; @@ -219,13 +219,14 @@ std::unique_ptr VoxCPM2LoadedModel::create_task_session( const runtime::TaskSpec &task, const runtime::SessionOptions &options) const { + const std::string family_label = metadata_.family == "voxcpm1" ? "VoxCPM1" : "VoxCPM2"; if (task.mode != runtime::RunMode::Offline && task.mode != runtime::RunMode::Streaming) { - throw std::runtime_error( - "VoxCPM2 only supports offline and streaming sessions"); + throw std::runtime_error(family_label + + " only supports offline and streaming sessions"); } if (task.task != runtime::VoiceTaskKind::Tts) { - throw std::runtime_error("VoxCPM2 only supports the Tts task"); + throw std::runtime_error(family_label + " only supports the Tts task"); } if (task.mode == runtime::RunMode::Streaming) { return std::make_unique(task, options, assets_); diff --git a/src/models/voxcpm2/session.cpp b/src/models/voxcpm2/session.cpp index 21375fc4..1f8f1680 100644 --- a/src/models/voxcpm2/session.cpp +++ b/src/models/voxcpm2/session.cpp @@ -50,6 +50,24 @@ void reject_denoiser_option( } } +std::unordered_map normalize_v1_session_options( + std::unordered_map options) { + // V1 sessions share this runtime but advertise "voxcpm1.*" options; alias + // them to "voxcpm2.*" so the shared parsing below accepts both spellings. + std::unordered_map out; + out.reserve(options.size()); + for (auto &[key, value] : options) { + constexpr std::string_view kV1Prefix = "voxcpm1."; + if (key.rfind(kV1Prefix, 0) == 0) { + out[std::string("voxcpm2.") + + key.substr(kV1Prefix.size())] = std::move(value); + } else { + out[std::move(key)] = std::move(value); + } + } + return out; +} + bool audio_buffer_equal(const runtime::AudioBuffer &lhs, const runtime::AudioBuffer &rhs) { return lhs.sample_rate == rhs.sample_rate && lhs.channels == rhs.channels && @@ -67,9 +85,9 @@ bool optional_audio_equal(const std::optional &lhs, size_t prompt_cache_slots_from_options( const std::unordered_map &options) { constexpr int64_t kDefaultPromptCacheSlots = 1; - const int64_t slots = - runtime::parse_i64_option(options, {"voxcpm2.prompt_cache_slots"}) - .value_or(kDefaultPromptCacheSlots); + const int64_t slots = runtime::parse_i64_option( + options, {"voxcpm2.prompt_cache_slots", "voxcpm1.prompt_cache_slots"}) + .value_or(kDefaultPromptCacheSlots); if (slots < 0) { throw std::runtime_error("voxcpm2.prompt_cache_slots must be non-negative"); } @@ -159,12 +177,17 @@ VoxCPM2SessionBase::VoxCPM2SessionBase(runtime::TaskSpec task, if (task_.mode != runtime::RunMode::Offline && task_.mode != runtime::RunMode::Streaming) { throw std::runtime_error( - "VoxCPM2 only supports offline and streaming sessions"); + std::string(assets_->config.v1 ? "VoxCPM1" : "VoxCPM2") + + " only supports offline and streaming sessions"); } if (task_.task != runtime::VoiceTaskKind::Tts) { - throw std::runtime_error("VoxCPM2 only supports the Tts task"); + throw std::runtime_error( + std::string(assets_->config.v1 ? "VoxCPM1" : "VoxCPM2") + + " only supports the Tts task"); } + options.options = normalize_v1_session_options(std::move(options.options)); + reject_enabled_denoise(options.options, {"voxcpm2.denoise"}); reject_enabled_denoise(options.options, {"voxcpm2.load_denoiser"}); reject_denoiser_option(options.options, {"voxcpm2.denoiser"}); @@ -225,7 +248,9 @@ VoxCPM2SessionBase::VoxCPM2SessionBase(runtime::TaskSpec task, VoxCPM2SessionBase::~VoxCPM2SessionBase() = default; -std::string VoxCPM2SessionBase::family_impl() const { return "voxcpm2"; } +std::string VoxCPM2SessionBase::family_impl() const { + return assets_->config.v1 ? "voxcpm1" : "voxcpm2"; +} runtime::VoiceTaskKind VoxCPM2SessionBase::task_kind_impl() const { return task_.task; } @@ -263,6 +288,7 @@ runtime::TaskResult VoxCPM2SessionBase::run_offline_request(const runtime::TaskR const auto generation_options = generation_options_from_request(request); const auto prompt_text = runtime::find_option(request.options, {"voxcpm2.prompt_text", + "voxcpm1.prompt_text", "prompt_text", "reference_text"}) .value_or(""); std::optional reference_audio; @@ -340,6 +366,7 @@ VoxCPM2SessionBase::run_streaming_request( auto generation_options = generation_options_from_request(request); const auto prompt_text = runtime::find_option(request.options, {"voxcpm2.prompt_text", + "voxcpm1.prompt_text", "prompt_text", "reference_text"}) .value_or(""); std::optional reference_audio; @@ -482,7 +509,8 @@ VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( VoxCPM2GenerationOptions options; bool min_tokens_explicit = false; if (const auto value = runtime::parse_i64_option( - request.options, {"voxcpm2.min_tokens", "min_tokens"})) { + request.options, + {"voxcpm2.min_tokens", "voxcpm1.min_tokens", "min_tokens"})) { options.min_tokens = *value; min_tokens_explicit = true; } @@ -499,40 +527,50 @@ VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( } } if (const auto value = runtime::parse_i64_option( - request.options, {"max_tokens", "voxcpm2.max_tokens"})) { + request.options, + {"max_tokens", "voxcpm2.max_tokens", "voxcpm1.max_tokens"})) { options.max_tokens = *value; } if (const auto value = runtime::parse_i64_option( request.options, - {"num_inference_steps", "voxcpm2.num_inference_steps"})) { + {"num_inference_steps", "voxcpm2.num_inference_steps", + "voxcpm1.num_inference_steps"})) { options.num_inference_steps = *value; } if (const auto value = runtime::parse_finite_float_option( - request.options, {"guidance_scale", "voxcpm2.guidance_scale"})) { + request.options, + {"guidance_scale", "voxcpm2.guidance_scale", + "voxcpm1.guidance_scale"})) { options.guidance_scale = *value; } if (const auto match = runtime::find_option_match( - request.options, {"voxcpm2.retry_badcase", "retry_badcase"})) { + request.options, + {"voxcpm2.retry_badcase", "voxcpm1.retry_badcase", + "retry_badcase"})) { options.retry_badcase = runtime::parse_bool_option(match->value, match->key); } if (const auto value = runtime::parse_i64_option( request.options, - {"voxcpm2.retry_badcase_max_times", "retry_badcase_max_times"})) { + {"voxcpm2.retry_badcase_max_times", + "voxcpm1.retry_badcase_max_times", "retry_badcase_max_times"})) { options.retry_badcase_max_times = *value; } if (const auto value = runtime::parse_finite_float_option( - request.options, {"voxcpm2.retry_badcase_ratio_threshold", - "retry_badcase_ratio_threshold"})) { + request.options, + {"voxcpm2.retry_badcase_ratio_threshold", + "voxcpm1.retry_badcase_ratio_threshold", + "retry_badcase_ratio_threshold"})) { options.retry_badcase_ratio_threshold = *value; } - if (const auto value = runtime::parse_u32_option(request.options, - {"voxcpm2.seed", "seed"})) { + if (const auto value = runtime::parse_u32_option( + request.options, {"voxcpm2.seed", "voxcpm1.seed", "seed"})) { options.seed = *value; } options.cfm_noise_file = runtime::find_option(request.options, - {"voxcpm2.cfm_noise_file", "cfm_noise_file"}) + {"voxcpm2.cfm_noise_file", "voxcpm1.cfm_noise_file", + "cfm_noise_file"}) .value_or(""); if (options.min_tokens < 0) { throw std::runtime_error("VoxCPM2 min_tokens must be non-negative"); @@ -565,10 +603,13 @@ VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( throw std::runtime_error( "VoxCPM2 retry_badcase_ratio_threshold must be positive"); } - reject_enabled_denoise(request.options, {"voxcpm2.denoise", "denoise"}); reject_enabled_denoise(request.options, - {"voxcpm2.load_denoiser", "load_denoiser"}); - reject_denoiser_option(request.options, {"voxcpm2.denoiser", "denoiser"}); + {"voxcpm2.denoise", "voxcpm1.denoise", "denoise"}); + reject_enabled_denoise(request.options, + {"voxcpm2.load_denoiser", "voxcpm1.load_denoiser", + "load_denoiser"}); + reject_denoiser_option(request.options, + {"voxcpm2.denoiser", "voxcpm1.denoiser", "denoiser"}); return options; } diff --git a/src/models/voxcpm2/session.cpp~ b/src/models/voxcpm2/session.cpp~ new file mode 100644 index 00000000..c3cf78bf --- /dev/null +++ b/src/models/voxcpm2/session.cpp~ @@ -0,0 +1,708 @@ +#include "engine/models/voxcpm2/session.h" + +#include "engine/framework/debug/profiler.h" +#include "engine/framework/runtime/options.h" +#include "engine/framework/text/chunking.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace engine::models::voxcpm2 { +namespace { + +using Clock = std::chrono::steady_clock; +constexpr int64_t kDefaultTextChunkSize = 2048; + +std::shared_ptr +require_assets(std::shared_ptr assets) { + if (assets == nullptr) { + throw std::runtime_error("VoxCPM2 session requires assets"); + } + return assets; +} + +void reject_enabled_denoise( + const std::unordered_map &options, + std::initializer_list keys) { + const auto match = runtime::find_option_match(options, keys); + if (match.has_value() && + runtime::parse_bool_option(match->value, match->key)) { + throw std::runtime_error( + "VoxCPM2 denoise is disabled in this implementation"); + } +} + +void reject_denoiser_option( + const std::unordered_map &options, + std::initializer_list keys) { + if (runtime::find_option_match(options, keys).has_value()) { + throw std::runtime_error( + "VoxCPM2 denoise is disabled in this implementation"); + } +} + +bool audio_buffer_equal(const runtime::AudioBuffer &lhs, + const runtime::AudioBuffer &rhs) { + return lhs.sample_rate == rhs.sample_rate && lhs.channels == rhs.channels && + lhs.samples == rhs.samples; +} + +bool optional_audio_equal(const std::optional &lhs, + const std::optional &rhs) { + if (lhs.has_value() != rhs.has_value()) { + return false; + } + return !lhs.has_value() || audio_buffer_equal(*lhs, *rhs); +} + +size_t prompt_cache_slots_from_options( + const std::unordered_map &options) { + constexpr int64_t kDefaultPromptCacheSlots = 1; + const int64_t slots = + runtime::parse_i64_option(options, {"voxcpm2.prompt_cache_slots"}) + .value_or(kDefaultPromptCacheSlots); + if (slots < 0) { + throw std::runtime_error("voxcpm2.prompt_cache_slots must be non-negative"); + } + return static_cast(slots); +} + +void validate_weight_storage(engine::assets::TensorStorageType storage_type, + const char *option_name) { + if (storage_type == engine::assets::TensorStorageType::Native || + storage_type == engine::assets::TensorStorageType::F32 || + storage_type == engine::assets::TensorStorageType::F16 || + storage_type == engine::assets::TensorStorageType::BF16 || + storage_type == engine::assets::TensorStorageType::Q8_0) { + return; + } + throw std::runtime_error(std::string(option_name) + + " supports only native, f32, f16, bf16, and q8_0"); +} + +void parse_weight_type( + const std::unordered_map &options, + const char *key, engine::assets::TensorStorageType &storage_type) { + const auto it = options.find(key); + if (it == options.end()) { + return; + } + storage_type = engine::assets::parse_tensor_storage_type(it->second); + validate_weight_storage(storage_type, key); +} + +void validate_session_options( + const std::unordered_map &options) { + for (const auto &[key, value] : options) { + (void)value; + if (key.rfind("voxcpm2.", 0) != 0) { + continue; + } + if (key == "voxcpm2.weight_context_mb" || + key == "voxcpm2.text_embedding_graph_context_mb" || + key == "voxcpm2.lm_step_graph_context_mb" || + key == "voxcpm2.projection_graph_context_mb" || + key == "voxcpm2.local_encoder_graph_context_mb" || + key == "voxcpm2.dit_graph_context_mb" || + key == "voxcpm2.audiovae_weight_context_mb" || + key == "voxcpm2.audiovae_graph_context_mb" || + key == "voxcpm2.audiovae_encoder_graph_context_mb" || + key == "voxcpm2.audiovae_latent_capacity" || + key == "voxcpm2.audiovae_encoder_sample_capacity" || + key == "voxcpm2.weight_type" || + key == "voxcpm2.audiovae_weight_type" || + key == "voxcpm2.prompt_cache_slots" || + key == "voxcpm2.mem_saver" || + key == "voxcpm2.denoise" || key == "voxcpm2.load_denoiser") { + continue; + } + throw std::runtime_error("unknown VoxCPM2 session option: " + key); + } +} + +int64_t product(const std::vector &values) { + int64_t out = 1; + for (const int64_t value : values) { + if (value <= 0) { + throw std::runtime_error("VoxCPM2 AudioVAE decoder rate is invalid"); + } + out *= value; + } + return out; +} + +} // namespace + +bool VoxCPM2SessionBase::EncodedPromptCacheKeyEqual::operator()( + const EncodedPromptCacheKey &lhs, + const EncodedPromptCacheKey &rhs) const { + return lhs.prompt_text == rhs.prompt_text && + optional_audio_equal(lhs.prompt_audio, rhs.prompt_audio) && + optional_audio_equal(lhs.reference_audio, rhs.reference_audio); +} + +VoxCPM2SessionBase::VoxCPM2SessionBase(runtime::TaskSpec task, + runtime::SessionOptions options, + std::shared_ptr assets) + : RuntimeSessionBase(options), task_(task), + assets_(require_assets(std::move(assets))), + encoded_prompt_cache_(prompt_cache_slots_from_options(options.options)) { + if (task_.mode != runtime::RunMode::Offline && + task_.mode != runtime::RunMode::Streaming) { + throw std::runtime_error( + "VoxCPM2 only supports offline and streaming sessions"); + } + if (task_.task != runtime::VoiceTaskKind::Tts) { + throw std::runtime_error("VoxCPM2 only supports the Tts task"); + } + + reject_enabled_denoise(options.options, {"voxcpm2.denoise"}); + reject_enabled_denoise(options.options, {"voxcpm2.load_denoiser"}); + reject_denoiser_option(options.options, {"voxcpm2.denoiser"}); + validate_session_options(options.options); + + generator_config_.weight_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm2.weight_context_mb"}, + generator_config_.weight_context_bytes); + generator_config_.text_embedding_graph_context_bytes = + runtime::parse_size_mb_option( + options.options, {"voxcpm2.text_embedding_graph_context_mb"}, + generator_config_.text_embedding_graph_context_bytes); + generator_config_.lm_step_graph_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm2.lm_step_graph_context_mb"}, + generator_config_.lm_step_graph_context_bytes); + generator_config_.projection_graph_context_bytes = + runtime::parse_size_mb_option( + options.options, {"voxcpm2.projection_graph_context_mb"}, + generator_config_.projection_graph_context_bytes); + generator_config_.local_encoder_graph_context_bytes = + runtime::parse_size_mb_option( + options.options, {"voxcpm2.local_encoder_graph_context_mb"}, + generator_config_.local_encoder_graph_context_bytes); + generator_config_.dit_graph_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm2.dit_graph_context_mb"}, + generator_config_.dit_graph_context_bytes); + generator_config_.prompt_cache_slots = encoded_prompt_cache_.capacity(); + decoder_config_.weight_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm2.audiovae_weight_context_mb"}, + decoder_config_.weight_context_bytes); + decoder_config_.graph_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm2.audiovae_graph_context_mb"}, + decoder_config_.graph_context_bytes); + decoder_config_.encoder_graph_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm2.audiovae_encoder_graph_context_mb"}, + decoder_config_.encoder_graph_context_bytes); + decoder_config_.latent_frame_capacity = runtime::parse_positive_i64_option( + options.options, {"voxcpm2.audiovae_latent_capacity"}, + decoder_config_.latent_frame_capacity); + decoder_config_.encoder_sample_capacity = runtime::parse_positive_i64_option( + options.options, {"voxcpm2.audiovae_encoder_sample_capacity"}, + decoder_config_.encoder_sample_capacity); + parse_weight_type(options.options, "voxcpm2.weight_type", + generator_config_.weight_storage_type); + parse_weight_type(options.options, "voxcpm2.audiovae_weight_type", + decoder_config_.weight_storage_type); + if (const auto mem_saver = + runtime::find_option(options.options, {"voxcpm2.mem_saver"})) { + generator_config_.mem_saver = + runtime::parse_bool_option(*mem_saver, "voxcpm2.mem_saver"); + } + + generator_ = std::make_unique( + assets_, execution_context(), generator_config_); + decoder_ = std::make_unique( + assets_, execution_context(), decoder_config_); +} + +VoxCPM2SessionBase::~VoxCPM2SessionBase() = default; + +std::string VoxCPM2SessionBase::family_impl() const { return "voxcpm2"; } + +runtime::VoiceTaskKind VoxCPM2SessionBase::task_kind_impl() const { return task_.task; } + +runtime::RunMode VoxCPM2SessionBase::run_mode_impl() const { return task_.mode; } + +void VoxCPM2SessionBase::prepare_impl( + const runtime::SessionPreparationRequest &request) { + (void)request; + mark_prepared(); +} + +runtime::TaskResult VoxCPM2SessionBase::run_offline_request(const runtime::TaskRequest &request) { + require_prepared("VoxCPM2 run"); + if (task_.mode != runtime::RunMode::Offline) { + throw std::runtime_error("VoxCPM2 run requires an offline session"); + } + validate_request(request); + auto release_runtime_memory = [this](VoxCPM2SessionBase *self) { + if (self != nullptr) { + self->release_request_runtime_memory(); + } + }; + std::unique_ptr + release_guard(generator_config_.mem_saver ? this : nullptr, + release_runtime_memory); + + const auto wall_start = Clock::now(); + const int64_t text_chunk_size = + engine::text::parse_text_chunk_size_override(request.options).value_or(kDefaultTextChunkSize); + const auto text_chunk_mode = + engine::text::parse_text_chunk_mode_override(request.options) + .value_or(engine::text::TextChunkMode::TagAware); + const auto chunk_requests = + runtime::chunk_text_request(request, text_chunk_size, text_chunk_mode); + const auto generation_options = generation_options_from_request(request); + const auto prompt_text = + runtime::find_option(request.options, {"voxcpm2.prompt_text", + "prompt_text", "reference_text"}) + .value_or(""); + std::optional reference_audio; + if (request.voice.has_value() && request.voice->speaker.has_value() && + request.voice->speaker->audio.has_value()) { + reference_audio = *request.voice->speaker->audio; + } + const VoxCPM2EncodedPrompt *prompt = + encoded_prompt_for_request(request.audio_input, prompt_text, + reference_audio); + + runtime::TaskResult result; + double generator_ms = 0.0; + double decoder_ms = 0.0; + runtime::AudioBuffer merged_audio; + for (const auto & chunk_request : chunk_requests) { + const auto generator_start = Clock::now(); + const auto generated = generator_->generate( + chunk_request.text_input->text, prompt, generation_options); + generator_ms += engine::debug::elapsed_ms(generator_start, Clock::now()); + + const auto decoder_start = Clock::now(); + auto audio = decoder_->decode_features(generated.decode_features, + generated.decode_patches); + if (generated.decode_trim_patches > 0) { + const int64_t trim_samples = + generated.decode_trim_patches * assets_->config.patch_size * + product(assets_->config.audio_vae.decoder_rates); + if (trim_samples > static_cast(audio.samples.size())) { + throw std::runtime_error( + "VoxCPM2 decoded continuation trim exceeds audio length"); + } + audio.samples.erase( + audio.samples.begin(), + audio.samples.begin() + static_cast(trim_samples)); + } + decoder_ms += engine::debug::elapsed_ms(decoder_start, Clock::now()); + runtime::append_audio_buffer(merged_audio, audio); + } + result.audio_output = std::move(merged_audio); + + const auto wall_end = Clock::now(); + debug::trace_log_scalar("voxcpm2.text_chunk_size", text_chunk_size); + debug::trace_log_scalar("voxcpm2.text_chunk_mode", + engine::text::text_chunk_mode_name(text_chunk_mode)); + debug::trace_log_scalar("voxcpm2.text_chunk_count", + static_cast(chunk_requests.size())); + debug::timing_log_scalar("voxcpm2.generator_ms", generator_ms); + debug::timing_log_scalar("voxcpm2.audiovae_decoder_ms", decoder_ms); + debug::timing_log_scalar("session.wall_ms", + engine::debug::elapsed_ms(wall_start, wall_end)); + return result; +} + +runtime::TaskResult +VoxCPM2SessionBase::run_streaming_request( + const runtime::TaskRequest &request, + const runtime::StreamEventCallback &stream_event_sink) { + require_prepared("VoxCPM2 run_streaming"); + if (task_.mode != runtime::RunMode::Streaming) { + throw std::runtime_error( + "VoxCPM2 run_streaming requires a streaming session"); + } + validate_request(request); + auto release_runtime_memory = [this](VoxCPM2SessionBase *self) { + if (self != nullptr) { + self->release_request_runtime_memory(); + } + }; + std::unique_ptr + release_guard(generator_config_.mem_saver ? this : nullptr, + release_runtime_memory); + + const auto wall_start = Clock::now(); + auto generation_options = generation_options_from_request(request); + const auto prompt_text = + runtime::find_option(request.options, {"voxcpm2.prompt_text", + "prompt_text", "reference_text"}) + .value_or(""); + std::optional reference_audio; + if (request.voice.has_value() && request.voice->speaker.has_value() && + request.voice->speaker->audio.has_value()) { + reference_audio = *request.voice->speaker->audio; + } + const VoxCPM2EncodedPrompt *prompt = + encoded_prompt_for_request(request.audio_input, prompt_text, + reference_audio); + + runtime::TaskResult result; + runtime::AudioBuffer merged; + merged.sample_rate = assets_->config.audio_vae.output_sample_rate; + merged.channels = 1; + double decoder_ms = 0.0; + size_t emitted_chunks = 0; + auto emit_chunk = [&](const VoxCPM2StreamingChunk &chunk) { + const auto decoder_start = Clock::now(); + auto audio = decoder_->decode_features(chunk.decode_features, + chunk.decode_patches); + decoder_ms += engine::debug::elapsed_ms(decoder_start, Clock::now()); + if (emitted_chunks == 0) { + merged.sample_rate = audio.sample_rate; + merged.channels = audio.channels; + } else if (audio.sample_rate != merged.sample_rate || + audio.channels != merged.channels) { + throw std::runtime_error( + "VoxCPM2 streaming decoder chunk format changed"); + } + merged.samples.insert(merged.samples.end(), audio.samples.begin(), + audio.samples.end()); + runtime::NamedAudioBuffer named; + named.id = "chunk_" + std::to_string(emitted_chunks); + named.audio = std::move(audio); + named.meta.insert_or_assign( + "generated_patches", std::to_string(chunk.generated_patches)); + if (stream_event_sink) { + runtime::StreamEvent event; + event.named_audio_outputs.push_back(named); + stream_event_sink(event); + } + result.named_audio_outputs.push_back(std::move(named)); + ++emitted_chunks; + }; + + const auto generator_start = Clock::now(); + (void)generator_->generate_streaming(request.text_input->text, prompt, + generation_options, emit_chunk); + const auto generator_end = Clock::now(); + const double generator_with_callbacks_ms = + engine::debug::elapsed_ms(generator_start, generator_end); + + result.audio_output = std::move(merged); + + const auto wall_end = Clock::now(); + debug::timing_log_scalar( + "voxcpm2.generator_ms", + std::max(0.0, generator_with_callbacks_ms - decoder_ms)); + debug::timing_log_scalar("voxcpm2.generator_streaming_callbacks_ms", + generator_with_callbacks_ms); + debug::timing_log_scalar("voxcpm2.audiovae_decoder_ms", decoder_ms); + debug::timing_log_scalar("voxcpm2.streaming_chunks", + static_cast(emitted_chunks)); + debug::timing_log_scalar("session.wall_ms", + engine::debug::elapsed_ms(wall_start, wall_end)); + return result; +} + +void VoxCPM2SessionBase::release_request_runtime_memory() { + if (!generator_config_.mem_saver) { + return; + } + generator_->release_runtime_memory(); + decoder_->release_runtime_memory(); +} + +const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( + const std::optional &prompt_audio, + const std::string &prompt_text, + const std::optional &reference_audio) { + if (!prompt_audio.has_value() && !reference_audio.has_value()) { + return nullptr; + } + EncodedPromptCacheKey key; + key.prompt_text = prompt_text; + key.prompt_audio = prompt_audio; + key.reference_audio = reference_audio; + if (auto *cached = encoded_prompt_cache_.find(key)) { + debug::trace_log_scalar("voxcpm2.prompt_cache.hit", 1); + debug::trace_log_scalar("voxcpm2.prompt_cache.slots", + static_cast( + encoded_prompt_cache_.capacity())); + debug::trace_log_scalar("voxcpm2.prompt_cache.entries", + static_cast(encoded_prompt_cache_.size())); + debug::trace_log_scalar("voxcpm2.prompt_cache.evicted", 0); + debug::timing_log_scalar("voxcpm2.prompt_encode_ms", 0.0); + return &cached->encoded; + } + + const auto encode_start = Clock::now(); + EncodedPromptCacheEntry entry; + entry.encoded = + decoder_->encode_prompt_audio(prompt_audio, prompt_text, reference_audio); + const double encode_ms = engine::debug::elapsed_ms(encode_start); + if (encoded_prompt_cache_.capacity() == 0) { + uncached_encoded_prompt_ = std::move(entry); + debug::trace_log_scalar("voxcpm2.prompt_cache.hit", 0); + debug::trace_log_scalar("voxcpm2.prompt_cache.slots", 0); + debug::trace_log_scalar("voxcpm2.prompt_cache.entries", 0); + debug::trace_log_scalar("voxcpm2.prompt_cache.evicted", 0); + debug::timing_log_scalar("voxcpm2.prompt_encode_ms", encode_ms); + return &uncached_encoded_prompt_->encoded; + } + const bool will_evict = + encoded_prompt_cache_.size() >= encoded_prompt_cache_.capacity(); + encoded_prompt_cache_.put(std::move(key), std::move(entry)); + EncodedPromptCacheKey lookup; + lookup.prompt_text = prompt_text; + lookup.prompt_audio = prompt_audio; + lookup.reference_audio = reference_audio; + auto *cached = encoded_prompt_cache_.find(lookup); + if (cached == nullptr) { + throw std::runtime_error("VoxCPM2 prompt cache insert failed"); + } + debug::trace_log_scalar("voxcpm2.prompt_cache.hit", 0); + debug::trace_log_scalar("voxcpm2.prompt_cache.slots", + static_cast( + encoded_prompt_cache_.capacity())); + debug::trace_log_scalar("voxcpm2.prompt_cache.entries", + static_cast(encoded_prompt_cache_.size())); + debug::trace_log_scalar("voxcpm2.prompt_cache.evicted", will_evict ? 1 : 0); + debug::timing_log_scalar("voxcpm2.prompt_encode_ms", + encode_ms); + return &cached->encoded; +} + +VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( + const runtime::TaskRequest &request) const { + VoxCPM2GenerationOptions options; + bool min_tokens_explicit = false; + if (const auto value = runtime::parse_i64_option( + request.options, {"voxcpm2.min_tokens", "min_tokens"})) { + options.min_tokens = *value; + min_tokens_explicit = true; + } + // Set V1-specific default min_tokens if not explicitly provided + if (!min_tokens_explicit && assets_->config.v1) { + // VoxCPM1 (0.5B) has patch_size=2, VoxCPM1.5 (1.5B) has patch_size=4 + // Use model-specific defaults for optimal speech speed + if (assets_->config.patch_size == 2) { + options.min_tokens = 15; // 20, VoxCPM1 0.5B + } else if (assets_->config.patch_size == 4) { + options.min_tokens = 12; // VoxCPM1.5 1.5B + } else { + options.min_tokens = 15; // Other V1 models + } + } + if (const auto value = runtime::parse_i64_option( + request.options, {"max_tokens", "voxcpm2.max_tokens"})) { + options.max_tokens = *value; + } + if (const auto value = runtime::parse_i64_option( + request.options, + {"num_inference_steps", "voxcpm2.num_inference_steps"})) { + options.num_inference_steps = *value; + } + if (const auto value = runtime::parse_finite_float_option( + request.options, {"guidance_scale", "voxcpm2.guidance_scale"})) { + options.guidance_scale = *value; + } + if (const auto match = runtime::find_option_match( + request.options, {"voxcpm2.retry_badcase", "retry_badcase"})) { + options.retry_badcase = + runtime::parse_bool_option(match->value, match->key); + } + if (const auto value = runtime::parse_i64_option( + request.options, + {"voxcpm2.retry_badcase_max_times", "retry_badcase_max_times"})) { + options.retry_badcase_max_times = *value; + } + if (const auto value = runtime::parse_finite_float_option( + request.options, {"voxcpm2.retry_badcase_ratio_threshold", + "retry_badcase_ratio_threshold"})) { + options.retry_badcase_ratio_threshold = *value; + } + if (const auto value = runtime::parse_u32_option(request.options, + {"voxcpm2.seed", "seed"})) { + options.seed = *value; + } + options.cfm_noise_file = + runtime::find_option(request.options, + {"voxcpm2.cfm_noise_file", "cfm_noise_file"}) + .value_or(""); + if (options.min_tokens < 0) { + throw std::runtime_error("VoxCPM2 min_tokens must be non-negative"); + } + if (options.max_tokens < 0) { + throw std::runtime_error("VoxCPM2 max_tokens must be non-negative"); + } + if (options.max_tokens == 0) { + options.max_tokens = assets_->config.max_length; + } + if (options.min_tokens > options.max_tokens) { + throw std::runtime_error("VoxCPM2 min_tokens must not exceed max_tokens"); + } + if (options.max_tokens > assets_->config.max_length) { + throw std::runtime_error( + "VoxCPM2 max_tokens exceeds model config max_length"); + } + if (options.num_inference_steps <= 0) { + throw std::runtime_error( + "VoxCPM2 num_inference_steps must be positive"); + } + if (options.guidance_scale < 0.0F) { + throw std::runtime_error("VoxCPM2 guidance_scale must be non-negative"); + } + if (options.retry_badcase_max_times <= 0) { + throw std::runtime_error( + "VoxCPM2 retry_badcase_max_times must be positive"); + } + if (options.retry_badcase_ratio_threshold <= 0.0F) { + throw std::runtime_error( + "VoxCPM2 retry_badcase_ratio_threshold must be positive"); + } + reject_enabled_denoise(request.options, {"voxcpm2.denoise", "denoise"}); + reject_enabled_denoise(request.options, + {"voxcpm2.load_denoiser", "load_denoiser"}); + reject_denoiser_option(request.options, {"voxcpm2.denoiser", "denoiser"}); + return options; +} + +void VoxCPM2SessionBase::validate_request( + const runtime::TaskRequest &request) const { + if (!request.text_input.has_value()) { + throw std::runtime_error("VoxCPM2 requires text input"); + } + if (request.text_input->text.empty()) { + throw std::runtime_error("VoxCPM2 text input must not be empty"); + } + if (request.voice.has_value()) { + if (request.voice->style.has_value()) { + throw std::runtime_error( + "VoxCPM2 C++ session does not consume style conditions"); + } + if (request.voice->speaker.has_value()) { + const auto &speaker = *request.voice->speaker; + if (speaker.cached_voice_id.has_value()) { + throw std::runtime_error("VoxCPM2 C++ session requires speaker " + "reference audio, not a cached voice id"); + } + if (!speaker.audio.has_value()) { + throw std::runtime_error( + "VoxCPM2 C++ session speaker condition requires audio"); + } + } + } + if (!request.input_artifacts.empty()) { + throw std::runtime_error( + "VoxCPM2 C++ session does not consume input artifacts"); + } +} + +VoxCPM2OfflineSession::VoxCPM2OfflineSession( + runtime::TaskSpec task, + runtime::SessionOptions options, + std::shared_ptr assets) + : VoxCPM2SessionBase(task, std::move(options), std::move(assets)) {} + +std::string VoxCPM2OfflineSession::family() const { return family_impl(); } + +runtime::VoiceTaskKind VoxCPM2OfflineSession::task_kind() const { + return task_kind_impl(); +} + +runtime::RunMode VoxCPM2OfflineSession::run_mode() const { + return run_mode_impl(); +} + +void VoxCPM2OfflineSession::prepare( + const runtime::SessionPreparationRequest &request) { + prepare_impl(request); +} + +runtime::TaskResult +VoxCPM2OfflineSession::run(const runtime::TaskRequest &request) { + return run_offline_request(request); +} + +VoxCPM2StreamingSession::VoxCPM2StreamingSession( + runtime::TaskSpec task, + runtime::SessionOptions options, + std::shared_ptr assets) + : VoxCPM2SessionBase(task, std::move(options), std::move(assets)) {} + +std::string VoxCPM2StreamingSession::family() const { return family_impl(); } + +runtime::VoiceTaskKind VoxCPM2StreamingSession::task_kind() const { + return task_kind_impl(); +} + +runtime::RunMode VoxCPM2StreamingSession::run_mode() const { + return run_mode_impl(); +} + +void VoxCPM2StreamingSession::prepare( + const runtime::SessionPreparationRequest &request) { + prepare_impl(request); +} + +runtime::StreamingPolicy VoxCPM2StreamingSession::streaming_policy() const { + runtime::StreamingPolicy policy; + policy.input = runtime::StreamingInputKind::None; + policy.output = runtime::StreamingOutputKind::FinalResult; + return policy; +} + +void VoxCPM2StreamingSession::start_stream(const runtime::TaskRequest &request) { + reset(); + result_ = run_streaming_request(request, stream_event_sink_); + started_ = true; +} + +void VoxCPM2StreamingSession::set_stream_event_sink(runtime::StreamEventCallback sink) { + stream_event_sink_ = std::move(sink); +} + +std::optional VoxCPM2StreamingSession::next_stream_event() { + if (!started_) { + throw std::runtime_error("VoxCPM2 streaming has not been started"); + } + if (next_chunk_index_ >= result_.named_audio_outputs.size()) { + return std::nullopt; + } + const auto & named = result_.named_audio_outputs[next_chunk_index_++]; + runtime::StreamEvent event; + event.named_audio_outputs.push_back(named); + return event; +} + +runtime::TaskResult VoxCPM2StreamingSession::finish_stream() { + if (!started_) { + throw std::runtime_error("VoxCPM2 streaming has not been started"); + } + started_ = false; + next_chunk_index_ = 0; + return std::move(result_); +} + +void VoxCPM2StreamingSession::reset() { + result_ = runtime::TaskResult{}; + next_chunk_index_ = 0; + started_ = false; +} + +runtime::StreamEvent VoxCPM2StreamingSession::process_audio_chunk( + const runtime::AudioChunk &chunk) { + (void)chunk; + throw std::runtime_error("VoxCPM2 streaming does not consume audio chunks"); +} + +runtime::TaskResult VoxCPM2StreamingSession::finalize() { + return finish_stream(); +} + +} // namespace engine::models::voxcpm2 diff --git a/tools/audiocpp_cli/audiocpp_cli_path_cases.json b/tools/audiocpp_cli/audiocpp_cli_path_cases.json index 2513515c..f522a666 100644 --- a/tools/audiocpp_cli/audiocpp_cli_path_cases.json +++ b/tools/audiocpp_cli/audiocpp_cli_path_cases.json @@ -737,6 +737,74 @@ } ] }, + { + "id": "voxcpm1_tts", + "coverage": "VoxCPM1 text-to-speech path with MiniCPM generation, diffusion feature generation, and AudioVAE decode", + "family": "voxcpm1", + "model": "models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf", + "task": "tts", + "mode": "offline", + "outputs": [ + "audio" + ], + "requests": [ + { + "id": "tts", + "text": "This VoxCPM1 path test checks the text-to-speech interface through AudioCPP CLI.", + "seed": 1234, + "max_tokens": 160, + "guidance_scale": 2.0, + "num_inference_steps": 10 + } + ] + }, + { + "id": "voxcpm1_voice_clone", + "coverage": "VoxCPM1 voice clone path with reference audio encoding, MiniCPM generation, diffusion feature generation, and AudioVAE decode", + "family": "voxcpm1", + "model": "models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf", + "task": "tts", + "mode": "offline", + "outputs": [ + "audio" + ], + "requests": [ + { + "id": "clone", + "text": "This VoxCPM1 path test clones the reference speaker for a short review sentence.", + "voice_ref": "resources/sample.wav", + "seed": 1234, + "max_tokens": 160, + "guidance_scale": 2.0, + "num_inference_steps": 10 + } + ] + }, + { + "id": "voxcpm1_streaming_tts", + "coverage": "VoxCPM1 streaming text-to-speech path with MiniCPM streaming generation, diffusion feature generation, and AudioVAE chunk decode", + "family": "voxcpm1", + "model": "models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf", + "task": "tts", + "mode": "streaming", + "chunk_size": 512, + "outputs": [ + "audio" + ], + "requests": [ + { + "id": "streaming_tts", + "text": "This VoxCPM1 streaming path test checks that the CLI can emit audio chunks for a longer request while preserving a steady speaking style.", + "seed": 1234, + "max_tokens": 160, + "guidance_scale": 2.0, + "num_inference_steps": 10, + "options": { + "retry_badcase": false + } + } + ] + }, { "id": "higgs_audio_tts_voice_clone_chunked", "coverage": "Higgs Audio v3 voice clone path with framework text chunking, AR generation, and codec decode", diff --git a/webui/configs/model_params.json b/webui/configs/model_params.json index 6214ce46..01d679e6 100644 --- a/webui/configs/model_params.json +++ b/webui/configs/model_params.json @@ -28,6 +28,13 @@ {"name": "retry_badcase", "type": "bool", "label": "retry_badcase（自动重试异常输出）", "default": true} ], + "voxcpm1": [ + {"name": "num_inference_steps", "type": "number", "label": "num_inference_steps", "default": 10, "minimum": 1, "step": 1, "precision": 0, "info": "CFM/DiT 步数"}, + {"name": "guidance_scale", "type": "slider", "label": "guidance_scale", "default": 2.0, "minimum": 0.0, "maximum": 5.0, "step": 0.1}, + {"name": "min_tokens", "type": "number", "label": "min_tokens", "default": 2, "minimum": 0, "step": 1, "precision": 0}, + {"name": "retry_badcase", "type": "bool", "label": "retry_badcase（自动重试异常输出）", "default": true} + ], + "miotts": [ {"name": "temperature", "type": "slider", "label": "temperature", "default": 0.8, "minimum": 0.0, "maximum": 2.0, "step": 0.05}, {"name": "top_k", "type": "number", "label": "top_k", "default": 50, "minimum": 0, "step": 1, "precision": 0}, diff --git a/webui/configs/models_catalog.json b/webui/configs/models_catalog.json index 05cefa2e..7cecec68 100644 --- a/webui/configs/models_catalog.json +++ b/webui/configs/models_catalog.json @@ -16,6 +16,7 @@ { "id": "qwen3-tts-1.7b-custom", "display_name": "Qwen3-TTS 1.7B CustomVoice (tts)", "family": "qwen3_tts", "path": "models/Qwen3-TTS-12Hz-1.7B-CustomVoice", "task": "tts", "mode": "offline", "download_id": "qwen3_tts_1_7b_custom_voice", "min_vram_gb": 8 }, { "id": "miotts", "display_name": "MioTTS 1.7B (tts; needs MioCodec)", "family": "miotts", "path": "models/MioTTS-1.7B", "task": "tts", "mode": "offline", "download_id": "miotts_1_7b", "min_vram_gb": 8 }, { "id": "voxcpm2", "display_name": "VoxCPM2 (tts)", "family": "voxcpm2", "path": "models/VoxCPM2", "task": "tts", "mode": "offline", "download_id": "voxcpm2", "session_options": { "voxcpm2.weight_type": "q8_0" }, "min_vram_gb": 6 }, + { "id": "voxcpm1", "display_name": "VoxCPM1 0.5B (tts + clone)", "family": "voxcpm1", "path": "models/VoxCPM1-GGUF", "task": "tts", "mode": "offline", "download_id": "voxcpm1_0.5b_q8_0", "min_vram_gb": 4 }, { "id": "vibevoice", "display_name": "VibeVoice 1.5B (tts, long-form/multi-speaker)", "family": "vibevoice", "path": "models/VibeVoice-1.5B", "task": "tts", "mode": "offline", "download_id": "vibevoice_1_5b", "min_vram_gb": 7 }, { "id": "index-tts2", "display_name": "IndexTTS2 (tts 中英克隆+情感)", "display_name_en": "IndexTTS2 (tts, zh/en clone + emotion)", "family": "index_tts2", "path": "models/IndexTTS-2", "task": "tts", "mode": "offline", "download_id": "index_tts2", "min_vram_gb": 8 }, { "id": "index-tts2.5", "display_name": "IndexTTS2.5 (tts 多语种克隆+情感, GGUF Q8)", "display_name_en": "IndexTTS2.5 (tts, zh/en/ja/es/ar clone + emotion, GGUF Q8)", "family": "index_tts2", "path": "models/IndexTTS2.5-GGUF", "task": "tts", "mode": "offline", "download_id": "index_tts2_5_q8_0", "min_vram_gb": 8, From a8fef3dc6281b0bdacafaf0f1491dab9c3761daf Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Tue, 18 Aug 2026 13:58:01 +0200 Subject: [PATCH 07/14] fix(voxcpm1): load real RoPE longrope factors and align min_tokens floor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit VoxCPM1 attention used identity longrope factors and a padded stop-token floor. The GGUF's real F32 factor arrays are now read and applied (prefill, stop behavior and duration match the VoxCPM.cpp reference), and the V1 default `min_tokens` is lowered to the reference floor so short utterances are no longer padded with trailing silence. **Root causes fixed (2):** - RoPE longrope factors were hardcoded to `1.0f` in the GGUF config path ("GGUF doesn't have native float arrays" was wrong — `GgufTensorSource` already parses them); every attention computation across all four transformers (base LM, residual LM, local encoder, local DiT) used identity positional encodings - V1 default `min_tokens=20` (per patch_size) vs reference `kMinLen=2` — forced ~1.6s+ of audio and padded short utterances with trailing silence after the stop predictor fired **Changes:** 2 files, +29/−12 lines - `src/models/voxcpm2/config_gguf.cpp` — read `voxcpm_lm_config_rope_scaling_{short,long}_factor` f32 arrays via `optional_f32_array()` with size validation (`head_dim/2`), identity fallback only when the keys are absent - `src/models/voxcpm2/session.cpp` — V1 default `min_tokens = 2` (≡ reference `step > kMinLen`), keeping the `--request-option min_tokens` override **Verified (CPU, against reference `/workspace/pi/VoxCPM.cpp`):** | Test | Result | |------|--------| | Prefill lm_hidden | l2 within ~2% of reference (was diverged) | | Stop predictor ("This is a test run for the fix") | fires at pos=19 (was: never fired) | | Duration | 1.60s (ref 1.68s), trailing silence 0.13s (ref 0.44s) | | V2 regression | 48kHz output maintained ✓ | | Embedding + fusion | `[73448,1024]` transpose intact, `has_fusion_proj=false` ✓ | **Issue**: Voice clone is still not supported — `--task clon` is rejected and passing reference audio + text (`--task tts --voice-ref `) generates noise rather than cloned speech. Needs a port-audit of the VoxCPM1 reference-audio conditioning path. Full evidence in `docs/reports/2026-08-18_1128_VoxCPM1_RoPE_Longrope_Factors_Stop_Floor_Fix.md`. --- src/models/voxcpm2/config_gguf.cpp | 27 ++++++++++++++++++++++++--- src/models/voxcpm2/session.cpp | 14 +++++--------- 2 files changed, 29 insertions(+), 12 deletions(-) diff --git a/src/models/voxcpm2/config_gguf.cpp b/src/models/voxcpm2/config_gguf.cpp index 37255586..218ca747 100644 --- a/src/models/voxcpm2/config_gguf.cpp +++ b/src/models/voxcpm2/config_gguf.cpp @@ -79,11 +79,32 @@ VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & // Rope scaling (longrope for VoxCPM1) config.lm.rope_scaling.type = "longrope"; - // GGUF doesn't have native float arrays, use defaults const int64_t head_dim = config.lm.hidden_size / config.lm.num_attention_heads; const int64_t factor_size = head_dim / 2; - config.lm.rope_scaling.long_factor.assign(factor_size, 1.0f); - config.lm.rope_scaling.short_factor.assign(factor_size, 1.0f); + // The GGUF stores the real longrope factor arrays as F32 metadata arrays + // (32 values for a 64-dim head, ~1.0004 to ~49.85 for VoxCPM1). Read them + // instead of the old identity fallback: identity factors silently degrade + // every RoPE computation across all four transformers. + auto short_factor = + source.optional_f32_array("voxcpm_lm_config_rope_scaling_short_factor"); + auto long_factor = + source.optional_f32_array("voxcpm_lm_config_rope_scaling_long_factor"); + if (short_factor && + static_cast(short_factor->size()) != factor_size) { + throw std::runtime_error( + "voxcpm_lm_config_rope_scaling_short_factor must have head_dim / 2 " + "elements"); + } + if (long_factor && + static_cast(long_factor->size()) != factor_size) { + throw std::runtime_error( + "voxcpm_lm_config_rope_scaling_long_factor must have head_dim / 2 " + "elements"); + } + config.lm.rope_scaling.short_factor = + short_factor.value_or(std::vector(factor_size, 1.0f)); + config.lm.rope_scaling.long_factor = + long_factor.value_or(std::vector(factor_size, 1.0f)); config.lm.rope_scaling.original_max_position_embeddings = get_optional_i64("voxcpm_lm_config_rope_scaling_original_max_position_embeddings").value_or(2048); diff --git a/src/models/voxcpm2/session.cpp b/src/models/voxcpm2/session.cpp index 1f8f1680..8b9ea0ec 100644 --- a/src/models/voxcpm2/session.cpp +++ b/src/models/voxcpm2/session.cpp @@ -516,15 +516,11 @@ VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( } // Set V1-specific default min_tokens if not explicitly provided if (!min_tokens_explicit && assets_->config.v1) { - // VoxCPM1 (0.5B) has patch_size=2, VoxCPM1.5 (1.5B) has patch_size=4 - // Use model-specific defaults for optimal speech speed - if (assets_->config.patch_size == 2) { - options.min_tokens = 20; // 20, VoxCPM1 0.5B - } else if (assets_->config.patch_size == 4) { - options.min_tokens = 12; // VoxCPM1.5 1.5B - } else { - options.min_tokens = 15; // Other V1 models - } + // Reference VoxCPM.cpp uses kMinLen=2 (stop may fire from the 4th patch); + // the decode loop gates on `index > min_tokens`, which is the same check. + // A higher floor (e.g. 20) forces ~1.6 s of audio and pads short + // utterances with trailing silence after the stop predictor fires. + options.min_tokens = 2; } if (const auto value = runtime::parse_i64_option( request.options, From 65ea6a13f53d5511c30f1bde9664e3c2129873b9 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Wed, 19 Aug 2026 11:38:39 +0000 Subject: [PATCH 08/14] fix(voxcpm1): align voice-clone conditioning and VAE encoder with golden impl Restore working voice cloning by fixing the reference-audio conditioning and AudioVAE encoder alignment against the golden VoxCPM.cpp port: - generator: only set the CFM `prefix_cond` from prefill rows carrying audio (audio_mask). Previously the trailing text row's zero feature overwrote the reference patch, feeding the DiT a zero acoustic anchor for voice cloning (matches torch feat[:, -1] semantics) - audiovae: re-enable VAD silence trimming for prompt/reference audio (matches golden server_common.cpp:842/878), then pad to patch alignment before VAE encoding (left for prompt, right for reference) - audiovae: drop the `stride % 2` output_padding on the encoder downsample conv so causal padding matches the reference encoder - assets: declare base_lm.embed_tokens.weight as [vocab, hidden] so V1 GGUFs storing the embedding transposed ([hidden, vocab]) load correctly - audiovae: add VOXCPM_DUMP_REF_MONO / REF_FEAT / ENC_STAGE debug dumps Validation (sensevoice-small STT, continuation-mode clone with the Anna reference): 6/6 target sentences transcribe exactly; text-only TTS unchanged. Reference-only cloning (ref_start/ref_end tokens) still fails identically in the golden VoxCPM.cpp - a model-level limitation. --- src/models/voxcpm2/assets.cpp | 2 + src/models/voxcpm2/audiovae.cpp | 132 +++++++++++++++++++++++++++++-- src/models/voxcpm2/generator.cpp | 4 +- 3 files changed, 132 insertions(+), 6 deletions(-) diff --git a/src/models/voxcpm2/assets.cpp b/src/models/voxcpm2/assets.cpp index b1fb03cb..b419c58b 100644 --- a/src/models/voxcpm2/assets.cpp +++ b/src/models/voxcpm2/assets.cpp @@ -600,6 +600,8 @@ class TransformingTensorSource final : public assets::TensorSource { // feat_quant: {N, F} -> {N, F, 1} // merge: {N, D} -> {N, D, 1} // downsample/upsample: {out, in} -> {out, in, k, k} (k=3 for 3x3) + // V1 embedding: token_embd.weight [hidden, vocab] -> base_lm.embed_tokens.weight [vocab, hidden] + {"base_lm.embed_tokens.weight", {config_.lm.vocab_size, config_.lm.hidden_size}}, }; // Folded AudioVAE conv weights: v1 GGUF stores weight-norm weights diff --git a/src/models/voxcpm2/audiovae.cpp b/src/models/voxcpm2/audiovae.cpp index 2d890f5e..40893122 100644 --- a/src/models/voxcpm2/audiovae.cpp +++ b/src/models/voxcpm2/audiovae.cpp @@ -40,6 +40,78 @@ using Clock = std::chrono::steady_clock; constexpr int64_t kResidualKernel = 7; +enum class PaddingMode { Left, Right }; + +std::vector trim_audio_silence_vad(const std::vector& input, + int sample_rate, + float max_silence_ms = 100.0f, + float top_db = 30.0f) { + if (input.empty() || sample_rate <= 0) { + return input; + } + + constexpr int kFrameLength = 2048; + constexpr int kHopLength = 512; + const float ref = *std::max_element(input.begin(), input.end(), [](float a, float b) { + return std::fabs(a) < std::fabs(b); + }); + if (std::fabs(ref) <= 0.0f) { + return input; + } + + const float threshold = std::fabs(ref) * std::pow(10.0f, -top_db / 20.0f); + const size_t n = input.size(); + int first_voice_frame = -1; + int last_voice_frame = -1; + + for (size_t idx = 0, frame = 0; idx < n; idx += kHopLength, ++frame) { + const size_t frame_end = std::min(idx + static_cast(kFrameLength), n); + const size_t frame_size = frame_end - idx; + if (frame_size == 0) { + break; + } + double energy = 0.0; + for (size_t i = idx; i < frame_end; ++i) { + energy += static_cast(input[i]) * static_cast(input[i]); + } + const float rms = static_cast(std::sqrt(energy / static_cast(frame_size))); + if (rms >= threshold) { + if (first_voice_frame < 0) { + first_voice_frame = static_cast(frame); + } + last_voice_frame = static_cast(frame); + } + if (frame_end == n) { + break; + } + } + + if (first_voice_frame < 0 || last_voice_frame < 0) { + return input; + } + + const int max_silence_samples = std::max(0, static_cast(std::lround(max_silence_ms * sample_rate / 1000.0f))); + const int start = std::max(0, first_voice_frame * kHopLength - max_silence_samples); + const int end = std::min(static_cast(n), + (last_voice_frame + 1) * kHopLength + (kFrameLength - kHopLength) + max_silence_samples); + if (start >= end) { + return input; + } + return std::vector(input.begin() + start, input.begin() + end); +} + +void pad_audio_for_patch_alignment(std::vector& audio, size_t patch_len, PaddingMode mode) { + if (patch_len == 0 || audio.empty() || (audio.size() % patch_len) == 0) { + return; + } + const size_t padding = patch_len - (audio.size() % patch_len); + if (mode == PaddingMode::Left) { + audio.insert(audio.begin(), padding, 0.0f); + } else { + audio.insert(audio.end(), padding, 0.0f); + } +} + struct GgmlContextDeleter { void operator()(ggml_context *ctx) const noexcept { if (ctx != nullptr) { @@ -525,9 +597,8 @@ core::TensorValue encoder_block(core::ModuleBuildContext &ctx, hidden = residual_unit(ctx, hidden, weights.residual_units[2], 9); hidden = snake_exact(ctx, hidden, weights.snake, weights.input_channels); const int padding = static_cast((weights.stride + 1) / 2); - const int output_padding = weights.stride % 2; return causal_conv1d(ctx, hidden, weights.downsample, weights.stride, padding, - 1, output_padding); + 1); } core::TensorValue decoder_block(core::ModuleBuildContext &ctx, @@ -668,13 +739,26 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { mono = engine::audio::resample_mono_soxr_or_linear( mono, audio.sample_rate, vae.sample_rate, options); } - const int64_t patch_samples = assets_->config.patch_size * encoder_stride_; - if (patch_samples <= 0) { - throw std::runtime_error("VoxCPM2 AudioVAE patch sample size is invalid"); + // VAD trim silence (match VoxCPM.cpp server_common.cpp:842/878) + mono = trim_audio_silence_vad(mono, vae.sample_rate); + if (const char *dump_path = std::getenv("VOXCPM_DUMP_REF_MONO")) { + FILE *f = std::fopen(dump_path, "wb"); + if (f != nullptr) { + std::fwrite(mono.data(), sizeof(float), mono.size(), f); + std::fclose(f); + } } + // Patch-aligned padding (Left for prompt, Right for reference) + const int64_t patch_samples = assets_->config.patch_size * encoder_stride_; + pad_audio_for_patch_alignment(mono, static_cast(patch_samples), + left_pad ? PaddingMode::Left : PaddingMode::Right); + // Final padding to encoder_sample_capacity const int64_t sample_count = static_cast(mono.size()); const int64_t padded_samples = ((sample_count + patch_samples - 1) / patch_samples) * patch_samples; + if (patch_samples <= 0) { + throw std::runtime_error("VoxCPM2 AudioVAE patch sample size is invalid"); + } if (padded_samples > config_.encoder_sample_capacity) { throw std::runtime_error( "VoxCPM2 AudioVAE encoder sample capacity exceeded"); @@ -694,6 +778,19 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { if (status != GGML_STATUS_SUCCESS) { throw std::runtime_error("VoxCPM2 AudioVAE encoder graph compute failed"); } + if (const char *stage_path = std::getenv("VOXCPM_DUMP_ENC_STAGE")) { + const std::string dir(stage_path); + for (size_t i = 0; i < encoder_stages_.size(); ++i) { + ggml_tensor *stage = encoder_stages_[i]; + std::vector buf(static_cast(ggml_nelements(stage)), 0.0F); + ggml_backend_tensor_get(stage, buf.data(), 0, buf.size() * sizeof(float)); + FILE *f = std::fopen((dir + "/stage_" + std::to_string(i) + ".bin").c_str(), "wb"); + if (f != nullptr) { + std::fwrite(buf.data(), sizeof(float), buf.size(), f); + std::fclose(f); + } + } + } const int64_t latent_frames = padded_samples / encoder_stride_; const int64_t expected_capacity_frames = @@ -716,6 +813,14 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { full[static_cast(c * expected_capacity_frames + t)]; } } + if (const char *dump_path = std::getenv("VOXCPM_DUMP_REF_FEAT")) { + FILE *f = std::fopen(dump_path, "wb"); + if (f != nullptr) { + std::fwrite(encoded.features.data(), sizeof(float), + encoded.features.size(), f); + std::fclose(f); + } + } return encoded; } @@ -872,15 +977,31 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { core::TensorShape::from_dims({1, 1, config_.encoder_sample_capacity})); encoder_input_ = hidden.tensor; ggml_set_input(encoder_input_); + encoder_stages_.clear(); + if (std::getenv("VOXCPM_DUMP_ENC_STAGE") != nullptr) { + encoder_stages_.push_back(encoder_input_); + } hidden = causal_conv1d(ctx, hidden, weights_.encoder_first, 1, 3, 1); + if (std::getenv("VOXCPM_DUMP_ENC_STAGE") != nullptr) { + encoder_stages_.push_back(hidden.tensor); + } for (const auto &block : weights_.encoder_blocks) { hidden = encoder_block(ctx, hidden, block); + if (std::getenv("VOXCPM_DUMP_ENC_STAGE") != nullptr) { + encoder_stages_.push_back(hidden.tensor); + } } hidden = causal_conv1d(ctx, hidden, weights_.encoder_fc_mu, 1, 1, 1); encoder_output_ = hidden.tensor; ggml_set_output(encoder_output_); + for (ggml_tensor *stage : encoder_stages_) { + ggml_set_output(stage); + } encoder_graph_ = ggml_new_graph_custom(encoder_ctx_.get(), 65536, false); ggml_build_forward_expand(encoder_graph_, encoder_output_); + for (ggml_tensor *stage : encoder_stages_) { + ggml_build_forward_expand(encoder_graph_, stage); + } encoder_gallocr_ = ggml_gallocr_new( ggml_backend_get_default_buffer_type(execution_context_.backend())); if (encoder_gallocr_ == nullptr || @@ -902,6 +1023,7 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { ggml_tensor *output_ = nullptr; ggml_tensor *encoder_input_ = nullptr; ggml_tensor *encoder_output_ = nullptr; + std::vector encoder_stages_; ggml_cgraph *graph_ = nullptr; ggml_cgraph *encoder_graph_ = nullptr; ggml_gallocr_t gallocr_ = nullptr; diff --git a/src/models/voxcpm2/generator.cpp b/src/models/voxcpm2/generator.cpp index b3a353b0..f62e1b05 100644 --- a/src/models/voxcpm2/generator.cpp +++ b/src/models/voxcpm2/generator.cpp @@ -1760,7 +1760,9 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { } input_embedding = current_embed; } - prefix_cond = row.feature; + if (row.audio_mask) { + prefix_cond = row.feature; + } prefill_input.input_embeddings.insert(prefill_input.input_embeddings.end(), input_embedding.begin(), input_embedding.end()); From 00ea405ffe536f96ade3273dea7ff63feb6556f1 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Wed, 19 Aug 2026 14:48:14 +0000 Subject: [PATCH 09/14] fix(voxcpm1): route --voice-ref through prompt path so V1 cloning works MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit VoxCPM1 voice cloning via `--voice-ref ` produced non-cloned speech, while `--audio ` (plus `--reference-text`) cloned correctly. Both flags carried the same user intent, but the CLI mapped them to different request fields that the session treated as two distinct audio roles. **Root cause:** `--voice-ref` set `request.voice->speaker->audio`, which the session consumed as *reference audio*. For VoxCPM1 the reference path is wrong in two ways: - `encode_prompt_audio()` only copies `prompt_text` inside the `prompt_audio` branch, so a reference-only request dropped the reference transcript entirely (the LM never saw it). - The reference role right-pads the audio and prepends it wrapped in the `` tokens 103/104. Those belong to VoxCPM2's "reference-mode plumbing"; the V1 LM was only trained for prompt-continuation cloning (golden VoxCPM.cpp uses `--prompt-audio` + `--prompt-text`, and its V1 server never calls `encode_reference_audio`). **Fix:** in `VoxCPM2SessionBase::encoded_prompt_for_request()`, when the model is V1 and only a reference audio is supplied (no `--audio`), route it through the prompt path — the audio becomes `prompt_audio` (left-padded, after ``) and `--reference-text` becomes `prompt_text` (concatenated with the target text). V2 keeps the reference-mode path untouched. Applies to both offline and streaming runs (single shared function). Without `--reference-text` the request now fails with the golden's exact rule ("prompt audio requires prompt_text or reference_text"). **Changes:** 1 file, +24/−12 lines - `src/models/voxcpm2/session.cpp` — V1 reference→prompt routing with cache key/lookup/encode all using the effective audio roles **Verified (CPU, 0.5B Q8_0):** | Test | Result | |------|--------| | V1 `--voice-ref` + ref-text | byte-identical WAV to `--audio` + ref-text (same clone) | | V1 `--voice-ref` without ref-text | clean error (matches golden iff rule) | | V1 `--audio` regression | byte-identical output | | V2 `--voice-ref` regression | 48kHz, byte-identical to pre-fix (reference mode preserved) | | 5-voice clone batch (ana/eric/andrew/jenny/nicole) | 16kHz speech, RMS 0.06–0.08 ✓ | **Note:** V1.5 (44.1kHz) fails at load with "encoder sample capacity must be divisible by encoder stride" — pre-existing config gap (stride 1764 ∤ default capacity 240000), identical on `--audio` before this fix. --- src/models/voxcpm2/session.cpp | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/src/models/voxcpm2/session.cpp b/src/models/voxcpm2/session.cpp index 8b9ea0ec..238ef3d8 100644 --- a/src/models/voxcpm2/session.cpp +++ b/src/models/voxcpm2/session.cpp @@ -451,10 +451,22 @@ const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( if (!prompt_audio.has_value() && !reference_audio.has_value()) { return nullptr; } + // VoxCPM1 clones only via prompt-continuation mode (golden VoxCPM.cpp + // uses --prompt-audio + --prompt-text); the V2 reference-mode path wraps + // audio in tokens 103/104, which the V1 LM was never trained on. Route a + // V1 reference audio through the prompt path so --voice-ref clones like + // --audio. + std::optional effective_prompt_audio = prompt_audio; + std::optional effective_reference_audio = reference_audio; + if (assets_->config.v1 && !effective_prompt_audio.has_value() && + effective_reference_audio.has_value()) { + effective_prompt_audio = effective_reference_audio; + effective_reference_audio.reset(); + } EncodedPromptCacheKey key; key.prompt_text = prompt_text; - key.prompt_audio = prompt_audio; - key.reference_audio = reference_audio; + key.prompt_audio = effective_prompt_audio; + key.reference_audio = effective_reference_audio; if (auto *cached = encoded_prompt_cache_.find(key)) { debug::trace_log_scalar("voxcpm2.prompt_cache.hit", 1); debug::trace_log_scalar("voxcpm2.prompt_cache.slots", @@ -469,8 +481,8 @@ const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( const auto encode_start = Clock::now(); EncodedPromptCacheEntry entry; - entry.encoded = - decoder_->encode_prompt_audio(prompt_audio, prompt_text, reference_audio); + entry.encoded = decoder_->encode_prompt_audio( + effective_prompt_audio, prompt_text, effective_reference_audio); const double encode_ms = engine::debug::elapsed_ms(encode_start); if (encoded_prompt_cache_.capacity() == 0) { uncached_encoded_prompt_ = std::move(entry); @@ -486,8 +498,8 @@ const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( encoded_prompt_cache_.put(std::move(key), std::move(entry)); EncodedPromptCacheKey lookup; lookup.prompt_text = prompt_text; - lookup.prompt_audio = prompt_audio; - lookup.reference_audio = reference_audio; + lookup.prompt_audio = effective_prompt_audio; + lookup.reference_audio = effective_reference_audio; auto *cached = encoded_prompt_cache_.find(lookup); if (cached == nullptr) { throw std::runtime_error("VoxCPM2 prompt cache insert failed"); From 57b649834bd1235c726aa36ee2520a81bef6aeca Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Wed, 19 Aug 2026 16:14:27 +0000 Subject: [PATCH 10/14] refactor: release tensor_source framework changes back to upstream Move VoxCPM GGUF tokenizer/config metadata reading out of the framework TensorSource interface into a new voxcpm2 GgufMetadataReader. Revert the validate_expected_shape relaxed_rank parameter and redundant include; tensor_source.h/.cpp now differ from upstream/main by a single line (is_synthesized). --- CMakeLists.txt | 2 + .../engine/framework/assets/tensor_source.h | 31 +- include/engine/models/voxcpm2/gguf_metadata.h | 47 +++ src/framework/assets/tensor_source.cpp | 277 +----------------- src/models/voxcpm2/assets.cpp | 2 +- src/models/voxcpm2/config_gguf.cpp | 31 +- src/models/voxcpm2/gguf_metadata.cpp | 136 +++++++++ src/models/voxcpm2/tokenizer_gguf.cpp | 28 +- 8 files changed, 230 insertions(+), 324 deletions(-) create mode 100644 include/engine/models/voxcpm2/gguf_metadata.h create mode 100644 src/models/voxcpm2/gguf_metadata.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index fa888f40..5b21254e 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -707,6 +707,7 @@ audiocpp_add_model(voxcpm2 src/models/voxcpm2/audiovae.cpp src/models/voxcpm2/config_gguf.cpp src/models/voxcpm2/generator.cpp + src/models/voxcpm2/gguf_metadata.cpp src/models/voxcpm2/loader.cpp src/models/voxcpm2/minicpm.cpp src/models/voxcpm2/session.cpp @@ -724,6 +725,7 @@ audiocpp_add_model(voxcpm1 src/models/voxcpm2/audiovae.cpp src/models/voxcpm2/config_gguf.cpp src/models/voxcpm2/generator.cpp + src/models/voxcpm2/gguf_metadata.cpp src/models/voxcpm2/loader.cpp src/models/voxcpm2/minicpm.cpp src/models/voxcpm2/session.cpp diff --git a/include/engine/framework/assets/tensor_source.h b/include/engine/framework/assets/tensor_source.h index a1648a01..b8088582 100644 --- a/include/engine/framework/assets/tensor_source.h +++ b/include/engine/framework/assets/tensor_source.h @@ -118,36 +118,7 @@ class TensorSource { [[nodiscard]] std::string require_tensor_name( std::initializer_list candidates) const; [[nodiscard]] virtual int64_t require_i64_scalar(std::string_view name) const = 0; - [[nodiscard]] virtual bool is_synthesized(std::string_view name) const noexcept { return false; } - - // GGUF metadata access (optional, only implemented by GgufTensorSource) - [[nodiscard]] virtual std::optional optional_string(std::string_view key) const { - return std::nullopt; - } - [[nodiscard]] virtual std::optional optional_u32(std::string_view key) const { - return std::nullopt; - } - [[nodiscard]] virtual std::optional> optional_string_array(std::string_view key) const { - return std::nullopt; - } - [[nodiscard]] virtual std::optional> optional_i32_array(std::string_view key) const { - return std::nullopt; - } - [[nodiscard]] virtual std::optional> optional_f32_array(std::string_view key) const { - return std::nullopt; - } - [[nodiscard]] virtual std::string require_string(std::string_view key) const { - throw std::runtime_error("require_string not supported by this TensorSource"); - } - [[nodiscard]] virtual uint32_t require_u32(std::string_view key) const { - throw std::runtime_error("require_u32 not supported by this TensorSource"); - } - [[nodiscard]] virtual std::vector require_string_array(std::string_view key) const { - throw std::runtime_error("require_string_array not supported by this TensorSource"); - } - [[nodiscard]] virtual std::vector require_i32_array(std::string_view key) const { - throw std::runtime_error("require_i32_array not supported by this TensorSource"); - } + [[nodiscard]] virtual bool is_synthesized(std::string_view) const noexcept { return false; } }; [[nodiscard]] TensorStorageType parse_tensor_storage_type(std::string_view value); diff --git a/include/engine/models/voxcpm2/gguf_metadata.h b/include/engine/models/voxcpm2/gguf_metadata.h new file mode 100644 index 00000000..0f538f36 --- /dev/null +++ b/include/engine/models/voxcpm2/gguf_metadata.h @@ -0,0 +1,47 @@ +#pragma once + +#include +#include +#include +#include +#include + +struct gguf_context; + +namespace engine::assets { +class TensorSource; +} + +namespace engine::models::voxcpm2 { + +// Reads GGUF KV metadata (tokenizer.ggml.*, voxcpm_*) directly from the file +// backing a TensorSource. Only meaningful for GGUF sources: for any other +// source type valid() is false and all accessors return nullopt (optional_*) +// or throw (require_*). This keeps VoxCPM schema knowledge out of the +// framework TensorSource interface. +class GgufMetadataReader { +public: + explicit GgufMetadataReader(const engine::assets::TensorSource & source); + ~GgufMetadataReader(); + + GgufMetadataReader(const GgufMetadataReader &) = delete; + GgufMetadataReader & operator=(const GgufMetadataReader &) = delete; + + bool valid() const noexcept { return gguf_ != nullptr; } + + [[nodiscard]] std::optional optional_string(std::string_view key) const; + [[nodiscard]] std::optional optional_u32(std::string_view key) const; + [[nodiscard]] std::optional> optional_string_array(std::string_view key) const; + [[nodiscard]] std::optional> optional_i32_array(std::string_view key) const; + [[nodiscard]] std::optional> optional_f32_array(std::string_view key) const; + + [[nodiscard]] std::string require_string(std::string_view key) const; + [[nodiscard]] uint32_t require_u32(std::string_view key) const; + [[nodiscard]] std::vector require_string_array(std::string_view key) const; + [[nodiscard]] std::vector require_i32_array(std::string_view key) const; + +private: + struct gguf_context * gguf_ = nullptr; +}; + +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/src/framework/assets/tensor_source.cpp b/src/framework/assets/tensor_source.cpp index a40509f9..189ea251 100644 --- a/src/framework/assets/tensor_source.cpp +++ b/src/framework/assets/tensor_source.cpp @@ -15,7 +15,6 @@ #include #include #include -#include #include #include #include @@ -72,25 +71,12 @@ core::TensorShape shape_from_dims(const std::vector & dims) { void validate_expected_shape( std::string_view name, const std::vector & actual_shape, - const std::optional> & expected_shape, - bool relaxed_rank) { + const std::optional> & expected_shape) { if (expected_shape.has_value() && actual_shape != *expected_shape) { - if (!relaxed_rank) { - throw std::runtime_error("tensor shape mismatch for " + std::string(name)); - } - int64_t expected_elems = 1; - for (const int64_t dim : *expected_shape) { - expected_elems *= dim; - } - int64_t actual_elems = 1; - for (const int64_t dim : actual_shape) { - actual_elems *= dim; - } - if (actual_elems != expected_elems) { - throw std::runtime_error("tensor element count mismatch for " + std::string(name)); - } + throw std::runtime_error("tensor shape mismatch for " + std::string(name)); } } + std::string lower_ascii(std::string_view value) { std::string out(value); for (char & ch : out) { @@ -579,7 +565,7 @@ class SafeTensorSource final : public TensorSource { if (info == nullptr) { throw std::runtime_error("missing tensor: " + std::string(name)); } - validate_expected_shape(name, info->shape, expected_shape, false); + validate_expected_shape(name, info->shape, expected_shape); const auto shape = shape_from_dims(expected_shape); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto [data, byte_size] = require_data_range(*info); @@ -608,7 +594,7 @@ class SafeTensorSource final : public TensorSource { std::string_view name, const std::optional> & expected_shape) const override { const auto tensor = require_tensor_data(name); - validate_expected_shape(name, tensor.metadata.shape, expected_shape, false); + validate_expected_shape(name, tensor.metadata.shape, expected_shape); const ggml_type type = ggml_type_for_tensor_dtype(tensor.metadata.dtype); const auto physical_shape = tensor.metadata.shape.empty() ? shape_from_dims({1}) @@ -691,8 +677,6 @@ class GgufTensorSource final : public TensorSource { public: explicit GgufTensorSource(std::filesystem::path path) : source_path_(std::filesystem::weakly_canonical(path)) { - // Read tokenizer metadata during initialization - read_metadata(path); ggml_context * tensor_context = nullptr; gguf_context * gguf = gguf_init_from_file( source_path_.string().c_str(), @@ -777,7 +761,6 @@ class GgufTensorSource final : public TensorSource { gguf_free(gguf); ggml_free(tensor_context); bytes_ = engine::io::read_binary_blob(source_path_); - read_metadata(source_path_); } const std::filesystem::path & source_path() const noexcept override { return source_path_; } @@ -820,7 +803,7 @@ class GgufTensorSource final : public TensorSource { TensorStorageType storage_type, const std::vector & expected_shape) const override { const auto & info = require_info(name); - validate_expected_shape(name, info.shape, expected_shape, false); + validate_expected_shape(name, info.shape, expected_shape); const auto shape = shape_from_dims(expected_shape); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto [data, byte_size] = require_data_range(info); @@ -848,7 +831,7 @@ class GgufTensorSource final : public TensorSource { std::string_view name, const std::optional> & expected_shape) const override { const auto tensor = require_tensor_data(name); - validate_expected_shape(name, tensor.metadata.shape, expected_shape, false); + validate_expected_shape(name, tensor.metadata.shape, expected_shape); const auto physical_shape = tensor.metadata.shape.empty() ? shape_from_dims({1}) : shape_from_dims(tensor.metadata.shape); @@ -897,248 +880,6 @@ class GgufTensorSource final : public TensorSource { std::vector infos_; std::unordered_map info_by_name_; mutable engine::io::BinaryBlob bytes_; - // Tokenizer metadata - std::optional tokenizer_model_; - std::optional tokenizer_pre_; - std::optional> tokenizer_tokens_; - std::optional> tokenizer_token_type_; - std::optional> tokenizer_merges_; - std::optional tokenizer_bos_token_id_; - std::optional tokenizer_eos_token_id_; - std::optional tokenizer_unknown_token_id_; - - // Config metadata (voxcpm_*) - std::unordered_map config_string_metadata_; - std::unordered_map config_u32_metadata_; - std::unordered_map> config_i32_array_metadata_; - std::unordered_map> config_f32_array_metadata_; - - void read_metadata(const std::filesystem::path & path) { - ggml_context * tensor_context = nullptr; - gguf_context * gguf = gguf_init_from_file( - path.string().c_str(), - gguf_init_params{true, &tensor_context}); - if (gguf == nullptr) { - if (tensor_context != nullptr) ggml_free(tensor_context); - return; - } - - const auto get_string = [&](const char* key) -> std::optional { - const int idx = gguf_find_key(gguf, key); - if (idx < 0 || gguf_get_kv_type(gguf, idx) != GGUF_TYPE_STRING) { - return std::nullopt; - } - const char* data = gguf_get_val_str(gguf, idx); - if (!data) return std::nullopt; - return std::string(data); - }; - - const auto get_u32 = [&](const char* key) -> std::optional { - const int idx = gguf_find_key(gguf, key); - if (idx < 0) return std::nullopt; - return gguf_get_val_u32(gguf, idx); - }; - - const auto get_f32 = [&](const char* key) -> std::optional { - const int idx = gguf_find_key(gguf, key); - if (idx < 0 || gguf_get_kv_type(gguf, idx) != GGUF_TYPE_FLOAT32) { - return std::nullopt; - } - return gguf_get_val_f32(gguf, idx); - }; - - const auto get_i32_array = [&](const char* key) -> std::optional> { - const int idx = gguf_find_key(gguf, key); - if (idx < 0) return std::nullopt; - const int32_t* data = static_cast(gguf_get_arr_data(gguf, idx)); - const size_t n = gguf_get_arr_n(gguf, idx); - if (!data && n != 0) return std::nullopt; - return std::vector(data, data + n); - }; - - const auto get_f32_array = [&](const char* key) -> std::optional> { - const int idx = gguf_find_key(gguf, key); - if (idx < 0 || gguf_get_arr_type(gguf, idx) != GGUF_TYPE_FLOAT32) { - return std::nullopt; - } - const float* data = static_cast(gguf_get_arr_data(gguf, idx)); - const size_t n = gguf_get_arr_n(gguf, idx); - if (!data && n != 0) return std::nullopt; - return std::vector(data, data + n); - }; - - const auto get_string_array = [&](const char* key) -> std::optional> { - const int idx = gguf_find_key(gguf, key); - if (idx < 0 || gguf_get_arr_type(gguf, idx) != GGUF_TYPE_STRING) { - return std::nullopt; - } - const size_t n = gguf_get_arr_n(gguf, idx); - std::vector values; - values.reserve(n); - for (size_t i = 0; i < n; ++i) { - const char* v = gguf_get_arr_str(gguf, idx, i); - values.emplace_back(v ? v : ""); - } - return values; - }; - - // Tokenizer metadata - tokenizer_model_ = get_string("tokenizer.ggml.model"); - tokenizer_pre_ = get_string("tokenizer.ggml.pre"); - tokenizer_tokens_ = get_string_array("tokenizer.ggml.tokens"); - tokenizer_token_type_ = get_i32_array("tokenizer.ggml.token_type"); - tokenizer_merges_ = get_string_array("tokenizer.ggml.merges"); - tokenizer_bos_token_id_ = get_u32("tokenizer.ggml.bos_token_id"); - tokenizer_eos_token_id_ = get_u32("tokenizer.ggml.eos_token_id"); - tokenizer_unknown_token_id_ = get_u32("tokenizer.ggml.unknown_token_id"); - - // Config metadata (voxcpm_*) - read all voxcpm_* keys - // We read known keys, but also could iterate all keys if needed - static constexpr const char* config_string_keys[] = { - "voxcpm_architecture", - "voxcpm_device", - "voxcpm_dtype", - "voxcpm_lm_config_rope_scaling_type", - "voxcpm_dit_config_cfm_config_solver", - "voxcpm_dit_config_cfm_config_t_scheduler", - }; - for (const char* key : config_string_keys) { - if (auto val = get_string(key)) { - config_string_metadata_[key] = *val; - } - } - - static constexpr const char* config_u32_keys[] = { - "voxcpm_lm_config_bos_token_id", - "voxcpm_lm_config_eos_token_id", - "voxcpm_lm_config_hidden_size", - "voxcpm_lm_config_intermediate_size", - "voxcpm_lm_config_max_position_embeddings", - "voxcpm_lm_config_num_attention_heads", - "voxcpm_lm_config_num_hidden_layers", - "voxcpm_lm_config_num_key_value_heads", - "voxcpm_lm_config_dim_model_base", - "voxcpm_lm_config_scale_emb", - "voxcpm_lm_config_rope_theta", - "voxcpm_lm_config_use_mup", - "voxcpm_lm_config_vocab_size", - "voxcpm_patch_size", - "voxcpm_feat_dim", - "voxcpm_residual_lm_num_layers", - "voxcpm_residual_lm_no_rope", - "voxcpm_scalar_quantization_latent_dim", - "voxcpm_scalar_quantization_scale", - "voxcpm_encoder_config_hidden_dim", - "voxcpm_encoder_config_ffn_dim", - "voxcpm_encoder_config_num_heads", - "voxcpm_encoder_config_num_layers", - "voxcpm_dit_config_hidden_dim", - "voxcpm_dit_config_ffn_dim", - "voxcpm_dit_config_num_heads", - "voxcpm_dit_config_num_layers", - "voxcpm_dit_config_mean_mode", - "voxcpm_audio_vae_config_encoder_dim", - "voxcpm_audio_vae_config_decoder_dim", - "voxcpm_audio_vae_config_latent_dim", - "voxcpm_audio_vae_config_sample_rate", - "voxcpm_audio_vae_config_out_sample_rate", - "voxcpm_max_length", - }; - for (const char* key : config_u32_keys) { - if (auto val = get_u32(key)) { - config_u32_metadata_[key] = *val; - } - } - - static constexpr const char* config_i32_array_keys[] = { - "voxcpm_audio_vae_config_encoder_rates", - "voxcpm_audio_vae_config_decoder_rates", - "voxcpm_audio_vae_config_sr_bin_boundaries", - }; - for (const char* key : config_i32_array_keys) { - if (auto val = get_i32_array(key)) { - config_i32_array_metadata_[key] = *val; - } - } - - static constexpr const char* config_f32_array_keys[] = { - "voxcpm_lm_config_rope_scaling_long_factor", - "voxcpm_lm_config_rope_scaling_short_factor", - }; - for (const char* key : config_f32_array_keys) { - if (auto val = get_f32_array(key)) { - config_f32_array_metadata_[key] = *val; - } - } - - gguf_free(gguf); - if (tensor_context != nullptr) ggml_free(tensor_context); - } - - // GGUF metadata access implementations - std::optional optional_string(std::string_view key) const override { - if (key == "tokenizer.ggml.model") return tokenizer_model_; - if (key == "tokenizer.ggml.pre") return tokenizer_pre_; - // Check config metadata - auto it = config_string_metadata_.find(std::string(key)); - if (it != config_string_metadata_.end()) return it->second; - return std::nullopt; - } - - std::optional optional_u32(std::string_view key) const override { - if (key == "tokenizer.ggml.bos_token_id") return tokenizer_bos_token_id_; - if (key == "tokenizer.ggml.eos_token_id") return tokenizer_eos_token_id_; - if (key == "tokenizer.ggml.unknown_token_id") return tokenizer_unknown_token_id_; - // Check config metadata - auto it = config_u32_metadata_.find(std::string(key)); - if (it != config_u32_metadata_.end()) return it->second; - return std::nullopt; - } - - std::optional> optional_string_array(std::string_view key) const override { - if (key == "tokenizer.ggml.tokens") return tokenizer_tokens_; - if (key == "tokenizer.ggml.merges") return tokenizer_merges_; - return std::nullopt; - } - - std::optional> optional_i32_array(std::string_view key) const override { - if (key == "tokenizer.ggml.token_type") return tokenizer_token_type_; - // Check config metadata - auto it = config_i32_array_metadata_.find(std::string(key)); - if (it != config_i32_array_metadata_.end()) return it->second; - return std::nullopt; - } - - std::optional> optional_f32_array(std::string_view key) const override { - // Check config metadata - auto it = config_f32_array_metadata_.find(std::string(key)); - if (it != config_f32_array_metadata_.end()) return it->second; - return std::nullopt; - } - - std::string require_string(std::string_view key) const override { - auto opt = optional_string(key); - if (opt) return *opt; - throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); - } - - uint32_t require_u32(std::string_view key) const override { - auto opt = optional_u32(key); - if (opt) return *opt; - throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); - } - - std::vector require_string_array(std::string_view key) const override { - auto opt = optional_string_array(key); - if (opt) return *opt; - throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); - } - - std::vector require_i32_array(std::string_view key) const override { - auto opt = optional_i32_array(key); - if (opt) return *opt; - throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); - } }; std::unordered_map parse_indexed_tensor_weight_map( @@ -1493,7 +1234,7 @@ TensorData TensorSource::require_tensor( const core::TensorShape shape = shape_from_dims(expected_shape); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto raw = require_tensor_data(name); - validate_expected_shape(name, raw.metadata.shape, expected_shape, false); + validate_expected_shape(name, raw.metadata.shape, expected_shape); if (raw_dtype_matches_ggml_type(raw.metadata.dtype, type)) { validate_raw_tensor_byte_size(name, shape, type, raw.bytes.size()); return TensorData{shape, type, raw.bytes}; @@ -1516,7 +1257,7 @@ TensorData TensorSource::require_tensor_as_shape( const core::TensorShape source_shape = shape_from_dims(expected); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto raw = require_tensor_data(name); - validate_expected_shape(name, raw.metadata.shape, expected, false); + validate_expected_shape(name, raw.metadata.shape, expected); if (raw.metadata.shape == std::vector(tensor_shape) && raw_dtype_matches_ggml_type(raw.metadata.dtype, type)) { validate_raw_tensor_byte_size(name, shape, type, raw.bytes.size()); diff --git a/src/models/voxcpm2/assets.cpp b/src/models/voxcpm2/assets.cpp index b419c58b..a68833ae 100644 --- a/src/models/voxcpm2/assets.cpp +++ b/src/models/voxcpm2/assets.cpp @@ -720,7 +720,7 @@ class TransformingTensorSource final : public assets::TensorSource { } assets::RawTensorData reshape_tensor_data(const assets::RawTensorData & data, - const std::vector & target_shape) const { + const std::vector &) const { // For now, just return the data as-is (validation happens elsewhere) // The actual reshape happens in require_f32 return data; diff --git a/src/models/voxcpm2/config_gguf.cpp b/src/models/voxcpm2/config_gguf.cpp index 218ca747..794313e2 100644 --- a/src/models/voxcpm2/config_gguf.cpp +++ b/src/models/voxcpm2/config_gguf.cpp @@ -1,6 +1,7 @@ #include "engine/models/voxcpm2/config_gguf.h" #include "engine/framework/assets/tensor_source.h" +#include "engine/models/voxcpm2/gguf_metadata.h" #include #include @@ -8,20 +9,22 @@ namespace engine::models::voxcpm2 { bool has_voxcpm1_config_metadata(const engine::assets::TensorSource & source) { + const GgufMetadataReader metadata(source); // Check for at least one VoxCPM1-specific metadata key - return source.optional_string("voxcpm_architecture").has_value() || - source.optional_string("voxcpm_lm_config_hidden_size").has_value() || - source.optional_u32("voxcpm_lm_config_hidden_size").has_value(); + return metadata.optional_string("voxcpm_architecture").has_value() || + metadata.optional_string("voxcpm_lm_config_hidden_size").has_value() || + metadata.optional_u32("voxcpm_lm_config_hidden_size").has_value(); } VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & source) { VoxCPM2Config config; + const GgufMetadataReader metadata(source); config.v1 = true; config.architecture = "voxcpm"; // Helper lambda to get optional i64 from GGUF metadata (via u32 or i64) - auto get_optional_i64 = [&source](const char * key) -> std::optional { - auto u32 = source.optional_u32(key); + auto get_optional_i64 = [&source, &metadata](const char * key) -> std::optional { + auto u32 = metadata.optional_u32(key); if (u32) return static_cast(*u32); // Try i64 scalar if it's a tensor if (source.has_tensor(key)) { @@ -35,15 +38,15 @@ VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & }; // Helper lambda to get optional bool from GGUF metadata - auto get_optional_bool = [&source](const char * key) -> std::optional { - auto u32 = source.optional_u32(key); + auto get_optional_bool = [&metadata](const char * key) -> std::optional { + auto u32 = metadata.optional_u32(key); if (u32) return *u32 != 0; return std::nullopt; }; // Helper lambda to get optional int64 array from GGUF metadata - auto get_optional_i64_array = [&source](const char * key) -> std::optional> { - auto i32_arr = source.optional_i32_array(key); + auto get_optional_i64_array = [&metadata](const char * key) -> std::optional> { + auto i32_arr = metadata.optional_i32_array(key); if (i32_arr) { std::vector result; result.reserve(i32_arr->size()); @@ -56,7 +59,7 @@ VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & }; // Architecture - auto arch = source.optional_string("voxcpm_architecture"); + auto arch = metadata.optional_string("voxcpm_architecture"); if (arch) config.architecture = *arch; // LM Config @@ -86,9 +89,9 @@ VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & // instead of the old identity fallback: identity factors silently degrade // every RoPE computation across all four transformers. auto short_factor = - source.optional_f32_array("voxcpm_lm_config_rope_scaling_short_factor"); + metadata.optional_f32_array("voxcpm_lm_config_rope_scaling_short_factor"); auto long_factor = - source.optional_f32_array("voxcpm_lm_config_rope_scaling_long_factor"); + metadata.optional_f32_array("voxcpm_lm_config_rope_scaling_long_factor"); if (short_factor && static_cast(short_factor->size()) != factor_size) { throw std::runtime_error( @@ -159,8 +162,8 @@ VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & config.max_length = get_optional_i64("voxcpm_max_length").value_or(2048); // Device and dtype - config.device = source.optional_string("voxcpm_device").value_or("cpu"); - config.dtype = source.optional_string("voxcpm_dtype").value_or("fp16"); + config.device = metadata.optional_string("voxcpm_device").value_or("cpu"); + config.dtype = metadata.optional_string("voxcpm_dtype").value_or("fp16"); // Validate required fields if (config.lm.hidden_size <= 0) { diff --git a/src/models/voxcpm2/gguf_metadata.cpp b/src/models/voxcpm2/gguf_metadata.cpp new file mode 100644 index 00000000..13981512 --- /dev/null +++ b/src/models/voxcpm2/gguf_metadata.cpp @@ -0,0 +1,136 @@ +#include "engine/models/voxcpm2/gguf_metadata.h" + +#include "engine/framework/assets/tensor_source.h" + +#include + +#include + +namespace engine::models::voxcpm2 { + +GgufMetadataReader::GgufMetadataReader(const engine::assets::TensorSource & source) { + // Metadata-only open: no_alloc=true with no ggml context parses the GGUF + // header and KV section without ever touching tensor data. + gguf_ = gguf_init_from_file( + source.source_path().string().c_str(), + gguf_init_params{true, nullptr}); +} + +GgufMetadataReader::~GgufMetadataReader() { + if (gguf_ != nullptr) { + gguf_free(gguf_); + } +} + +std::optional GgufMetadataReader::optional_string(std::string_view key) const { + if (gguf_ == nullptr) { + return std::nullopt; + } + const int64_t idx = gguf_find_key(gguf_, std::string(key).c_str()); + if (idx < 0 || gguf_get_kv_type(gguf_, idx) != GGUF_TYPE_STRING) { + return std::nullopt; + } + const char * data = gguf_get_val_str(gguf_, idx); + if (data == nullptr) { + return std::nullopt; + } + return std::string(data); +} + +std::optional GgufMetadataReader::optional_u32(std::string_view key) const { + if (gguf_ == nullptr) { + return std::nullopt; + } + // No KV type check: VoxCPM stores boolean flags (use_mup, no_rope, + // mean_mode) as scalar values that gguf_get_val_u32 reads regardless of + // their declared scalar type. + const int64_t idx = gguf_find_key(gguf_, std::string(key).c_str()); + if (idx < 0) { + return std::nullopt; + } + return gguf_get_val_u32(gguf_, idx); +} + +std::optional> GgufMetadataReader::optional_string_array(std::string_view key) const { + if (gguf_ == nullptr) { + return std::nullopt; + } + const int64_t idx = gguf_find_key(gguf_, std::string(key).c_str()); + if (idx < 0 || gguf_get_arr_type(gguf_, idx) != GGUF_TYPE_STRING) { + return std::nullopt; + } + const size_t n = gguf_get_arr_n(gguf_, idx); + std::vector values; + values.reserve(n); + for (size_t i = 0; i < n; ++i) { + const char * v = gguf_get_arr_str(gguf_, idx, i); + values.emplace_back(v != nullptr ? v : ""); + } + return values; +} + +std::optional> GgufMetadataReader::optional_i32_array(std::string_view key) const { + if (gguf_ == nullptr) { + return std::nullopt; + } + const int64_t idx = gguf_find_key(gguf_, std::string(key).c_str()); + if (idx < 0) { + return std::nullopt; + } + const auto * data = static_cast(gguf_get_arr_data(gguf_, idx)); + const size_t n = gguf_get_arr_n(gguf_, idx); + if (data == nullptr) { + return n == 0 ? std::optional>(std::vector{}) : std::nullopt; + } + return std::vector(data, data + n); +} + +std::optional> GgufMetadataReader::optional_f32_array(std::string_view key) const { + if (gguf_ == nullptr) { + return std::nullopt; + } + const int64_t idx = gguf_find_key(gguf_, std::string(key).c_str()); + if (idx < 0 || gguf_get_arr_type(gguf_, idx) != GGUF_TYPE_FLOAT32) { + return std::nullopt; + } + const auto * data = static_cast(gguf_get_arr_data(gguf_, idx)); + const size_t n = gguf_get_arr_n(gguf_, idx); + if (data == nullptr) { + return n == 0 ? std::optional>(std::vector{}) : std::nullopt; + } + return std::vector(data, data + n); +} + +std::string GgufMetadataReader::require_string(std::string_view key) const { + auto opt = optional_string(key); + if (opt) { + return *opt; + } + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); +} + +uint32_t GgufMetadataReader::require_u32(std::string_view key) const { + auto opt = optional_u32(key); + if (opt) { + return *opt; + } + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); +} + +std::vector GgufMetadataReader::require_string_array(std::string_view key) const { + auto opt = optional_string_array(key); + if (opt) { + return *opt; + } + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); +} + +std::vector GgufMetadataReader::require_i32_array(std::string_view key) const { + auto opt = optional_i32_array(key); + if (opt) { + return *opt; + } + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); +} + +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/src/models/voxcpm2/tokenizer_gguf.cpp b/src/models/voxcpm2/tokenizer_gguf.cpp index e849fff9..f3230653 100644 --- a/src/models/voxcpm2/tokenizer_gguf.cpp +++ b/src/models/voxcpm2/tokenizer_gguf.cpp @@ -1,6 +1,7 @@ #include "engine/models/voxcpm2/tokenizer_gguf.h" #include "engine/framework/assets/tensor_source.h" +#include "engine/models/voxcpm2/gguf_metadata.h" #include #include @@ -196,18 +197,22 @@ VoxCPM1GgufTokenizer::VoxCPM1GgufTokenizer(std::shared_ptr(); auto & impl = *impl_; // Read tokenizer metadata from GGUF directly in constructor - const std::string tokenizer_model = gguf_source->require_string("tokenizer.ggml.model"); - const std::string tokenizer_pre = gguf_source->require_string("tokenizer.ggml.pre"); - const std::vector tokens = gguf_source->require_string_array("tokenizer.ggml.tokens"); - const std::vector token_types = gguf_source->require_i32_array("tokenizer.ggml.token_type"); - const std::vector merges = gguf_source->require_string_array("tokenizer.ggml.merges"); - const uint32_t bos_id = gguf_source->require_u32("tokenizer.ggml.bos_token_id"); - const uint32_t eos_id = gguf_source->require_u32("tokenizer.ggml.eos_token_id"); - const uint32_t unk_id = gguf_source->require_u32("tokenizer.ggml.unknown_token_id"); + const std::string tokenizer_model = metadata.require_string("tokenizer.ggml.model"); + const std::string tokenizer_pre = metadata.require_string("tokenizer.ggml.pre"); + const std::vector tokens = metadata.require_string_array("tokenizer.ggml.tokens"); + const std::vector token_types = metadata.require_i32_array("tokenizer.ggml.token_type"); + const std::vector merges = metadata.require_string_array("tokenizer.ggml.merges"); + const uint32_t bos_id = metadata.require_u32("tokenizer.ggml.bos_token_id"); + const uint32_t eos_id = metadata.require_u32("tokenizer.ggml.eos_token_id"); + const uint32_t unk_id = metadata.require_u32("tokenizer.ggml.unknown_token_id"); if (tokenizer_model != "gpt2" || tokens.empty() || merges.empty() || token_types.size() != tokens.size()) { throw std::runtime_error("Invalid VoxCPM1 GGUF tokenizer metadata"); @@ -350,9 +355,10 @@ int32_t VoxCPM1GgufTokenizer::unk_token_id() const noexcept { } bool VoxCPM1GgufTokenizer::has_tokenizer_metadata(const engine::assets::TensorSource & source) { - return source.optional_string("tokenizer.ggml.model").has_value() && - source.optional_string_array("tokenizer.ggml.tokens").has_value() && - source.optional_string_array("tokenizer.ggml.merges").has_value(); + const GgufMetadataReader metadata(source); + return metadata.optional_string("tokenizer.ggml.model").has_value() && + metadata.optional_string_array("tokenizer.ggml.tokens").has_value() && + metadata.optional_string_array("tokenizer.ggml.merges").has_value(); } } // namespace engine::models::voxcpm2 \ No newline at end of file From 2b65a0d59a295699c3e388be1253b9215873625d Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Wed, 19 Aug 2026 18:16:26 +0200 Subject: [PATCH 11/14] fix(voxcpm1): Add Webui support for VoxCPM v1 (0.5B) --- webui/native/dist/index.html | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/webui/native/dist/index.html b/webui/native/dist/index.html index a9f22547..6a456372 100644 --- a/webui/native/dist/index.html +++ b/webui/native/dist/index.html @@ -13,20 +13,20 @@
From 1d9c59b5e021796abe61e3fa26adc3633b360861 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Wed, 19 Aug 2026 19:26:57 +0000 Subject: [PATCH 12/14] perf(voxcpm1): release text-length-scaled VRAM after each request Only the cloned voice is cached across requests; prompt-prefill and AudioVAE encoder/decoder graphs are freed at request end and rebuilt fresh on the next request. Idle VRAM drops to ~1.4GB after generation; very long text may require up to ~3.5GB VRAM during generation. --- include/engine/models/voxcpm2/audiovae.h | 1 + include/engine/models/voxcpm2/generator.h | 1 + include/engine/models/voxcpm2/minicpm.h | 2 + src/models/voxcpm2/audiovae.cpp | 10 +- src/models/voxcpm2/generator.cpp | 151 ++++++++++++++++++---- src/models/voxcpm2/minicpm.cpp | 38 +++++- src/models/voxcpm2/session.cpp | 42 ++++-- 7 files changed, 200 insertions(+), 45 deletions(-) diff --git a/include/engine/models/voxcpm2/audiovae.h b/include/engine/models/voxcpm2/audiovae.h index 5e1b038c..d6a250aa 100644 --- a/include/engine/models/voxcpm2/audiovae.h +++ b/include/engine/models/voxcpm2/audiovae.h @@ -43,6 +43,7 @@ class VoxCPM2AudioVAEDecoderRuntime final { const std::string &prompt_text, const std::optional &reference_audio); void release_runtime_memory(); + void release_encoder_graph(); private: class Impl; diff --git a/include/engine/models/voxcpm2/generator.h b/include/engine/models/voxcpm2/generator.h index abbcf8be..cf46d955 100644 --- a/include/engine/models/voxcpm2/generator.h +++ b/include/engine/models/voxcpm2/generator.h @@ -49,6 +49,7 @@ class VoxCPM2FeatureGeneratorRuntime final { const std::function &chunk_callback = nullptr); void release_runtime_memory(); + void release_text_length_memory(); private: class Impl; diff --git a/include/engine/models/voxcpm2/minicpm.h b/include/engine/models/voxcpm2/minicpm.h index 1aa57fc3..5b40a03b 100644 --- a/include/engine/models/voxcpm2/minicpm.h +++ b/include/engine/models/voxcpm2/minicpm.h @@ -135,6 +135,7 @@ class VoxCPM2TextEmbeddingRuntime final { ~VoxCPM2TextEmbeddingRuntime(); std::vector embed_token(int32_t token_id); + void release_runtime_memory(); private: class Impl; @@ -150,6 +151,7 @@ class VoxCPM2PromptPrefillRuntime final { ~VoxCPM2PromptPrefillRuntime(); VoxCPM2PromptPrefillOutput run(const VoxCPM2PromptPrefillInput &input); + void release_runtime_memory(); private: class Impl; diff --git a/src/models/voxcpm2/audiovae.cpp b/src/models/voxcpm2/audiovae.cpp index 40893122..f59fe280 100644 --- a/src/models/voxcpm2/audiovae.cpp +++ b/src/models/voxcpm2/audiovae.cpp @@ -710,9 +710,11 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { void release_runtime_memory() { release_decoder_graph(); - release_encoder_graph(); + release_encoder_graph_impl(); } + void release_encoder_graph() { release_encoder_graph_impl(); } + private: struct EncodedFeatures { std::vector features; @@ -937,7 +939,7 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { decoder_latent_frame_capacity_ = latent_frame_capacity; } - void release_encoder_graph() { + void release_encoder_graph_impl() { if (encoder_graph_ != nullptr) { core::release_backend_graph_resources(execution_context_.backend(), encoder_graph_); @@ -1059,4 +1061,8 @@ void VoxCPM2AudioVAEDecoderRuntime::release_runtime_memory() { impl_->release_runtime_memory(); } +void VoxCPM2AudioVAEDecoderRuntime::release_encoder_graph() { + impl_->release_encoder_graph(); +} + } // namespace engine::models::voxcpm2 diff --git a/src/models/voxcpm2/generator.cpp b/src/models/voxcpm2/generator.cpp index f62e1b05..4f1e7af2 100644 --- a/src/models/voxcpm2/generator.cpp +++ b/src/models/voxcpm2/generator.cpp @@ -206,6 +206,7 @@ class VoxCPM2StepProjectionRuntime final { VoxCPM2StepProjectionOutput run(const std::vector &lm_hidden, const std::vector &residual_hidden, const std::vector ¤t_embed); + void release_runtime_memory(); private: class Impl; @@ -221,6 +222,7 @@ class VoxCPM2LocalEncoderRuntime final { std::vector encode_patch(const std::vector &patch_features) const; + void release_runtime_memory(); private: class Impl; @@ -239,6 +241,7 @@ class VoxCPM2DiTEstimatorRuntime final { const std::vector &cond, const std::vector &time_embedding, const std::vector &delta_time_embedding); + void release_runtime_memory(); private: class Impl; @@ -259,6 +262,7 @@ class VoxCPM2CFMRuntime final { uint64_t noise_start_index = 0, const std::string &noise_file = {}, float temperature = 1.0F); + void release_runtime_memory(); private: class Impl; @@ -277,15 +281,9 @@ class VoxCPM2StepProjectionRuntime::Impl { build(graph_context_bytes); } - ~Impl() { - engine::core::release_backend_graph_resources(weights_->backend(), graph_); - if (buffer_ != nullptr) { - ggml_backend_buffer_free(buffer_); - } - if (gallocr_ != nullptr) { - ggml_gallocr_free(gallocr_); - } - } + ~Impl() { release_graph(); } + + void release_runtime_memory() { release_graph(); } VoxCPM2StepProjectionOutput run(const std::vector &lm_hidden, const std::vector &residual_hidden, @@ -356,6 +354,33 @@ class VoxCPM2StepProjectionRuntime::Impl { } private: + void release_graph() { + if (graph_ != nullptr) { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + } + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + buffer_ = nullptr; + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + graph_ = nullptr; + lm_hidden_ = nullptr; + residual_hidden_ = nullptr; + current_embed_ = nullptr; + fsq_hidden_output_ = nullptr; + current_residual_input_output_ = nullptr; + residual_input_output_ = nullptr; + current_lm_dit_output_ = nullptr; + fsq_lm_dit_output_ = nullptr; + residual_dit_output_ = nullptr; + current_stop_logits_output_ = nullptr; + fsq_stop_logits_output_ = nullptr; + ctx_.reset(); + } + void build(size_t graph_context_bytes) { const auto &config = weights_->assets().config; if (graph_context_bytes == 0) { @@ -579,6 +604,10 @@ VoxCPM2StepProjectionRuntime::VoxCPM2StepProjectionRuntime( VoxCPM2StepProjectionRuntime::~VoxCPM2StepProjectionRuntime() = default; +void VoxCPM2StepProjectionRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + VoxCPM2StepProjectionOutput VoxCPM2StepProjectionRuntime::run(const std::vector &lm_hidden, const std::vector &residual_hidden, @@ -598,15 +627,9 @@ class VoxCPM2LocalEncoderRuntime::Impl { build(graph_context_bytes); } - ~Impl() { - engine::core::release_backend_graph_resources(weights_->backend(), graph_); - if (buffer_ != nullptr) { - ggml_backend_buffer_free(buffer_); - } - if (gallocr_ != nullptr) { - ggml_gallocr_free(gallocr_); - } - } + ~Impl() { release_graph(); } + + void release_runtime_memory() { release_graph(); } std::vector encode_patch(const std::vector &patch_features) const { @@ -632,6 +655,25 @@ class VoxCPM2LocalEncoderRuntime::Impl { } private: + void release_graph() { + if (graph_ != nullptr) { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + } + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + buffer_ = nullptr; + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + graph_ = nullptr; + input_ = nullptr; + positions_ = nullptr; + output_ = nullptr; + ctx_.reset(); + } + void build(size_t graph_context_bytes) { const auto &root_config = weights_->assets().config; if (graph_context_bytes == 0) { @@ -736,6 +778,10 @@ VoxCPM2LocalEncoderRuntime::VoxCPM2LocalEncoderRuntime( VoxCPM2LocalEncoderRuntime::~VoxCPM2LocalEncoderRuntime() = default; +void VoxCPM2LocalEncoderRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + std::vector VoxCPM2LocalEncoderRuntime::encode_patch( const std::vector &patch_features) const { return impl_->encode_patch(patch_features); @@ -753,15 +799,9 @@ class VoxCPM2DiTEstimatorRuntime::Impl { build(graph_context_bytes); } - ~Impl() { - engine::core::release_backend_graph_resources(weights_->backend(), graph_); - if (buffer_ != nullptr) { - ggml_backend_buffer_free(buffer_); - } - if (gallocr_ != nullptr) { - ggml_gallocr_free(gallocr_); - } - } + ~Impl() { release_graph(); } + + void release_runtime_memory() { release_graph(); } std::vector run(const std::vector &x, const std::vector &mu, @@ -937,6 +977,29 @@ class VoxCPM2DiTEstimatorRuntime::Impl { } private: + void release_graph() { + if (graph_ != nullptr) { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + } + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + buffer_ = nullptr; + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + graph_ = nullptr; + x_ = nullptr; + mu_ = nullptr; + cond_ = nullptr; + time_embedding_ = nullptr; + delta_time_embedding_ = nullptr; + positions_ = nullptr; + output_ = nullptr; + ctx_.reset(); + } + void build(size_t graph_context_bytes) { const auto &root_config = weights_->assets().config; const auto &config = root_config.dit; @@ -1164,6 +1227,10 @@ VoxCPM2DiTEstimatorRuntime::VoxCPM2DiTEstimatorRuntime( VoxCPM2DiTEstimatorRuntime::~VoxCPM2DiTEstimatorRuntime() = default; +void VoxCPM2DiTEstimatorRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + std::vector VoxCPM2DiTEstimatorRuntime::run( const std::vector &x, const std::vector &mu, const std::vector &cond, const std::vector &time_embedding, @@ -1201,6 +1268,8 @@ class VoxCPM2CFMRuntime::Impl { } } + void release_runtime_memory() { estimator_.release_runtime_memory(); } + std::vector generate_patch(const std::vector &mu, const std::vector &cond_patch, int64_t timesteps, float cfg_value, @@ -1408,6 +1477,10 @@ VoxCPM2CFMRuntime::VoxCPM2CFMRuntime( VoxCPM2CFMRuntime::~VoxCPM2CFMRuntime() = default; +void VoxCPM2CFMRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + std::vector VoxCPM2CFMRuntime::generate_patch( const std::vector &mu, const std::vector &cond_patch, int64_t timesteps, float cfg_value, uint64_t seed, @@ -1503,8 +1576,23 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { } void release_runtime_memory() { + // Release every staged graph so a session can idle at weight-only + // VRAM. Each runtime lazily rebuilds its graph on the next use. + text_embedding_.release_runtime_memory(); + prefill_.release_runtime_memory(); base_lm_.release_runtime_memory(); residual_lm_.release_runtime_memory(); + projection_.release_runtime_memory(); + cfm_.release_runtime_memory(); + local_encoder_.release_runtime_memory(); + } + + void release_text_length_memory() { + // Only the prompt-prefill graph is sized by the request text/prompt + // length; the other generator graphs have fixed-size workspaces. Drop it + // after every request so a long-lived session does not retain buffers + // that scale with text length; the next request rebuilds it fresh. + prefill_.release_runtime_memory(); } private: @@ -1795,6 +1883,11 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { residual_lm_.import_state(prefill_output.residual_state); std::vector lm_hidden = prefill_output.lm_hidden; std::vector residual_hidden = prefill_output.residual_hidden; + // The prefill graph holds the largest sequence-shaped workspace; its + // outputs have been copied to host hiddens and its KV state imported + // into the step runtimes, so nothing below references it. Drop it now + // so the token loop runs against the much smaller step graphs. + prefill_.release_runtime_memory(); VoxCPM2Result result; std::vector context_rows; @@ -1947,4 +2040,8 @@ void VoxCPM2FeatureGeneratorRuntime::release_runtime_memory() { impl_->release_runtime_memory(); } +void VoxCPM2FeatureGeneratorRuntime::release_text_length_memory() { + impl_->release_text_length_memory(); +} + } // namespace engine::models::voxcpm2 diff --git a/src/models/voxcpm2/minicpm.cpp b/src/models/voxcpm2/minicpm.cpp index e03a02e7..409aec91 100644 --- a/src/models/voxcpm2/minicpm.cpp +++ b/src/models/voxcpm2/minicpm.cpp @@ -338,15 +338,11 @@ class VoxCPM2TextEmbeddingRuntime::Impl { } ~Impl() { - engine::core::release_backend_graph_resources(weights_->backend(), graph_); - if (buffer_ != nullptr) { - ggml_backend_buffer_free(buffer_); - } - if (gallocr_ != nullptr) { - ggml_gallocr_free(gallocr_); - } + release_graph(); } + void release_runtime_memory() { release_graph(); } + std::vector embed_token(int32_t token_id) { const auto &config = weights_->assets().config.lm; if (token_id < 0 || token_id >= config.vocab_size) { @@ -368,6 +364,24 @@ class VoxCPM2TextEmbeddingRuntime::Impl { } private: + void release_graph() { + if (graph_ != nullptr) { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + } + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + buffer_ = nullptr; + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + graph_ = nullptr; + token_id_ = nullptr; + output_ = nullptr; + ctx_.reset(); + } + void build(size_t graph_context_bytes) { const auto &config = weights_->assets().config.lm; if (graph_context_bytes == 0) { @@ -450,6 +464,10 @@ std::vector VoxCPM2TextEmbeddingRuntime::embed_token(int32_t token_id) { return impl_->embed_token(token_id); } +void VoxCPM2TextEmbeddingRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + struct MiniCPMLayerWithCacheOutput { engine::core::TensorValue output; engine::core::TensorValue key; @@ -578,6 +596,8 @@ class VoxCPM2PromptPrefillRuntime::Impl { ~Impl() { release_graph(); } + void release_runtime_memory() { release_graph(); } + VoxCPM2PromptPrefillOutput run(const VoxCPM2PromptPrefillInput &input) { const auto &config = weights_->assets().config; const int64_t hidden_size = config.lm.hidden_size; @@ -909,6 +929,10 @@ VoxCPM2PromptPrefillRuntime::run(const VoxCPM2PromptPrefillInput &input) { return impl_->run(input); } +void VoxCPM2PromptPrefillRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + engine::core::TensorValue minicpm_layer_with_static_cache(engine::core::ModuleBuildContext &ctx, const engine::core::TensorValue &input, diff --git a/src/models/voxcpm2/session.cpp b/src/models/voxcpm2/session.cpp index 238ef3d8..45072ba2 100644 --- a/src/models/voxcpm2/session.cpp +++ b/src/models/voxcpm2/session.cpp @@ -274,8 +274,7 @@ runtime::TaskResult VoxCPM2SessionBase::run_offline_request(const runtime::TaskR } }; std::unique_ptr - release_guard(generator_config_.mem_saver ? this : nullptr, - release_runtime_memory); + release_guard(this, release_runtime_memory); const auto wall_start = Clock::now(); const int64_t text_chunk_size = @@ -299,17 +298,32 @@ runtime::TaskResult VoxCPM2SessionBase::run_offline_request(const runtime::TaskR const VoxCPM2EncodedPrompt *prompt = encoded_prompt_for_request(request.audio_input, prompt_text, reference_audio); + // The encoded prompt (voice-clone conditioning) is cached as host-side + // vectors; the VAE encoder graph that produced it is not needed again until + // a different voice is encoded. Drop it before the generator runs so the + // generator and decoder phases never coexist with the encoder graph. + decoder_->release_encoder_graph(); runtime::TaskResult result; double generator_ms = 0.0; double decoder_ms = 0.0; runtime::AudioBuffer merged_audio; - for (const auto & chunk_request : chunk_requests) { + const bool mem_saver = generator_config_.mem_saver; + for (size_t chunk_index = 0; chunk_index < chunk_requests.size(); + ++chunk_index) { + const auto &chunk_request = chunk_requests[chunk_index]; const auto generator_start = Clock::now(); const auto generated = generator_->generate( chunk_request.text_input->text, prompt, generation_options); generator_ms += engine::debug::elapsed_ms(generator_start, Clock::now()); + if (mem_saver && chunk_index + 1 == chunk_requests.size()) { + // Last chunk: free the generator graphs before the AudioVAE decode so + // the final decode peaks at weight + decoder graph instead of weight + + // generator + decoder. Graphs rebuild lazily on the next request. + generator_->release_runtime_memory(); + } + const auto decoder_start = Clock::now(); auto audio = decoder_->decode_features(generated.decode_features, generated.decode_patches); @@ -359,8 +373,7 @@ VoxCPM2SessionBase::run_streaming_request( } }; std::unique_ptr - release_guard(generator_config_.mem_saver ? this : nullptr, - release_runtime_memory); + release_guard(this, release_runtime_memory); const auto wall_start = Clock::now(); auto generation_options = generation_options_from_request(request); @@ -377,6 +390,10 @@ VoxCPM2SessionBase::run_streaming_request( const VoxCPM2EncodedPrompt *prompt = encoded_prompt_for_request(request.audio_input, prompt_text, reference_audio); + // Same host-side clone-conditioning cache invariant as the offline path: + // the encoder graph is only needed to produce the cached vectors, so free + // it before the streaming generation starts. + decoder_->release_encoder_graph(); runtime::TaskResult result; runtime::AudioBuffer merged; @@ -437,11 +454,18 @@ VoxCPM2SessionBase::run_streaming_request( } void VoxCPM2SessionBase::release_request_runtime_memory() { - if (!generator_config_.mem_saver) { - return; - } - generator_->release_runtime_memory(); + // Only the cloned voice is cached across requests (host-side encoded + // vectors in encoded_prompt_cache_). Every graph whose size follows the + // request text/audio length (prompt prefill, VAE encoder/decoder) is + // dropped so a long-lived server session returns to baseline VRAM and + // reallocates fresh buffers sized to the next request. + generator_->release_text_length_memory(); decoder_->release_runtime_memory(); + if (generator_config_.mem_saver) { + // mem_saver additionally drops the fixed-size generator graphs so the + // session idles at weight-only VRAM. + generator_->release_runtime_memory(); + } } const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( From 1d6576c942bf761b9deb9ac2d8126528c64c678d Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Wed, 19 Aug 2026 23:14:49 +0200 Subject: [PATCH 13/14] remove local --- docs/reports/voxcpm1_pr.md | 243 ------------------------------------- 1 file changed, 243 deletions(-) delete mode 100644 docs/reports/voxcpm1_pr.md diff --git a/docs/reports/voxcpm1_pr.md b/docs/reports/voxcpm1_pr.md deleted file mode 100644 index a5d3b70e..00000000 --- a/docs/reports/voxcpm1_pr.md +++ /dev/null @@ -1,243 +0,0 @@ -# PR: VoxCPM1 — lightweight VoxCPM TTS support (0.5B / 1.5B) - -> **Status: first porting attempt — runtime works end-to-end, output quality NOT yet acceptable** -> -> The port successfully loads and runs all three VoxCPM v1 GGUF variants (anchors pass, graphs -> execute, WAV files are produced at the correct sample rates/durations with active signal). -> **Known issue:** the generated audio is almost pure noise with only a faint trace of human -> voice. The pipeline is correct mechanically, but output quality requires further debugging -> (hypotheses and investigation plan in [Known issue](#known-issue-noisy-output)). - ---- - -## 1. Overview - -This PR adds support for the **OpenBMB VoxCPM v1** family of lightweight TTS models to -audio.cpp, reusing the existing and already-released `voxcpm2` model tree: - -| Model | Params | Output sample rate | GGUF file | -|---|---|---|---| -| VoxCPM-0.5B | 0.5B | **16 kHz** | `voxcpm-0.5b-q8_0-audiovae-f16.gguf` | -| VoxCPM-1.5B | 1.5B | **44.1 kHz** | `voxcpm1.5-q8_0.gguf` | -| VoxCPM-1.5B | 1.5B | **44.1 kHz** | `voxcpm1.5-q4_k-audiovae-f16.gguf` | - -The three models are architecturally **different variants** (they cannot share one config): - -- **0.5B:** VAE encoder 128 / decoder 1536, encoder_rates `[2,5,8,8]`, decoder_rates - `[8,8,5,2]`, patch_size 2, residual_lm 6 layers, encoder/dit 4 layers, 16 kHz, max_len 4096. -- **1.5B:** VAE encoder 64 / decoder 2048, encoder_rates `[2,3,6,7,7]`, decoder_rates - `[7,7,6,3,2]`, patch_size 4, residual_lm 8 layers, encoder/dit 8 layers, 44.1 kHz, max_len 8192. - -Since the v1 GGUFs store a different tensor convention than v2 (folded AudioVAE weights, no -`weight_v`/`weight_g` split, no `sr_cond_model` tensors, `voxcpm` architecture name), the port -wraps the v2 loader with a GGUF tensor-adaptation layer and adds `config.v1`-guarded branches -in the generator, mirroring the reference implementation (`VoxCPM.cpp`). - ---- - -## 2. Porting activities - -1. **Regenerated the 0.5B `config.json` from the GGUF metadata** — the previously shipped - sidecar was wrong on ~8 axes (patch, residual_lm/encoder/dit layer counts, VAE dims and - rates, sample rate 44.1 kHz vs the actual 16 kHz, max_len). -2. **Diagnosed the v1 GGUF conventions** (tensor dump + reference converter analysis): - - AudioVAE conv weights are stored **already folded** (weight-norm folded), with no - `weight_v`/`weight_g` decomposition and no `sr_cond_model.*` tensors. - - GGUF file dims == ggml `ne` order; the v1 GGUFs carry **no** `audiocpp.tensor_shapes` - override metadata (v2 does), so the adapter must present shapes itself. - - The 1.5B **Q8_0** file stores VAE conv weights **2D-flattened** (`{out, in·k}`, kernel - folded into dim1) while Q4_K and 0.5B store 3D `{out, in, k}` — both must load. -3. **Designed the identity-fold adapter** (see §4) so the existing `load_vae_weights` loader - works unchanged against folded v1 weights byte-for-byte. -4. **Mirrored the reference generator math** for the no-fusion (no `fusion_concat_proj`) - case: elementwise-add fusion inputs, elementwise-add dit-mu, and a real residual_lm - autoregressive step. -5. **Set up per-variant model directories** (`VoxCPM1-GGUF/` for 0.5B, `VoxCPM1.5-GGUF/` - for 1.5B) each with a config regenerated from its own GGUF metadata + tokenizer sidecars, - and updated `model_specs/voxcpm1.json` package targets accordingly. -6. **Verified end-to-end runs** for all three GGUFs on the CPU backend (see - [Validation](#6-validation-performed)). - ---- - -## 3. Changes per file - -| File | Change | -|---|---| -| `CMakeLists.txt` | Added `audiocpp_add_model(voxcpm1 ...)` reusing the 7 voxcpm2 sources; registers `engine::models::voxcpm2::make_voxcpm1_loader`. | -| `include/engine/models/voxcpm2/loader.h` | Declared `make_voxcpm1_loader()`. | -| `include/engine/models/voxcpm2/assets.h` | Added `VoxCPM2Config::v1 = false`; `load_voxcpm2_assets()` now takes `bool is_v1`. | -| `src/models/voxcpm2/loader.cpp` | Added `VoxCPM1Loader` (family `"voxcpm1"`), `load_voxcpm1_model()`, `make_voxcpm1_loader()`, `metadata_v1` / `capabilities_v1` / `cli_v1`. Offline-only TTS + speaker-reference clone, `text_prefix` policy, GGUF via `load_voxcpm2_assets(path, is_v1=true)`. | -| `src/models/voxcpm2/assets.cpp` | Added `TransformingTensorSource` v1 adapter (biggest chunk):
• v1→v2 tensor-name rename map (`token_embd.weight`→`base_lm.embed_tokens.weight`, gguf `blk.N.*`→`base_lm.layers.N.*` / `feat_encoder.encoder.layers.*` / `feat_decoder.estimator.decoder.layers.*` / `residual_lm.layers.*`, `attn_norm`→`input_layernorm`, `ffn_norm`→`post_attention_layernorm`, `attn_*`→`self_attn.*_proj`, `ffn_*`→`mlp.*_proj`, `time_mlp.*` (preserving `.linear_N`), `output_norm.weight`→`base_lm.norm.weight`, projection/fsq/stop mappings)
• **Folded weight-norm synthesis**: for every `audio_vae.*.weight` conv, `X.weight_v` → folded tensor data as-is, `X.weight_g` → per-row L2 norms (identity fold, see §4)
• Identity `decoder.sr_cond_model.{2..5}.scale_embed.weight` (ones) / `.bias_embed.weight` (zeros) since v1 GGUFs carry no SR-conditioning tensors
• Synthesized missing v1 tensors (`feat_encoder.scale_embed/bias_embed`, `feat_encoder.fc_logvar`, `feat_encoder.diag`, `feat_encoder.merge`, `token_embd.extra_bias`, `fusion_concat_proj.weight/bias`, `stop_proj.weight`, `stop_head.weight`)
• Rank-tolerant `require_f32` (accept element-count-equal, shape-different fetches — handles 2D-flattened convs and `{C,1}` alphas) + relaxed-rank VAE weight_v anchors for v1
• `has_tensor` / `require_metadata` / `require_tensor_data` folded + synthesized lookups
• **Anchor fix:** `encoder.fc_mu.weight_v` now uses computed encoder-in (`encoder_dim << #rates` = 2048), not `decoder_dim` (1536) | -| `src/models/voxcpm2/generator.cpp` | • v1 fusion guard: residual input = `AddModule(lm_hidden, current_embed)` / `AddModule(fsq, current_embed)` instead of concat+linear (matches reference `build_residual_fusion_input`)
• Added `add_dit_mu()` helper; v1 `mu` = elementwise add of `current_lm_dit_hidden + residual_dit_hidden` (matches reference `build_dit_mu`, `mu_dim = hidden·(fusion?2:1)`, v1 → hidden)
• CFM `mu` size check is now v1-aware (`hidden_dim * (v1 ? 1 : 2)`)
• v1 decode loop runs `residual_lm_.run_step(next_projected.residual_input).hidden` (the earlier `fsq_lm_dit_hidden` shortcut removed — v1 GGUFs have 6/8 residual_lm layers) | -| `src/models/voxcpm2/minicpm.cpp` | Prompt-prefill graph: v1 `residual_input` = `AddModule(lm_hidden, masked_current)` instead of concat+linear; residual_lm always runs (previously the concat path would have produced a wrong-dimension residual input for v1). | -| `model_specs/voxcpm1.json` | Package targets: `voxcpm1_0.5b_q8_0` → `VoxCPM1-GGUF`; `voxcpm1_1.5b_q4_k` and `voxcpm1_1.5b_q8_0` → `VoxCPM1.5-GGUF` (per-variant config/tokenizer). | -| `docs/tts.md` | Added VoxCPM1 section + TOC entry (usage, options, sample-rate notes). | -| `README.md` | Added `voxcpm1` row to the supported-model table. | -| `docs/reports/voxcpm1_port_status.md` | Port status log (analysis, decisions, timestamps, remaining tasks). | -| `models/VoxCPM1-GGUF/config.json` | **Regenerated** from 0.5B GGUF metadata. | -| `models/VoxCPM1.5-GGUF/config.json` | **New**, regenerated from 1.5B GGUF metadata. | -| `models/VoxCPM1.5-GGUF/tokenizer.json` (+config/special tokens) | Copied from 0.5B dir (same 73,448-vocab BPE tokenizer). | - ---- - -## 4. Key design: the identity-fold adapter - -The v1 GGUF (OpenBMB reference converter) stores AudioVAE conv weights **already folded** -(`weight = weight_g · weight_v / ‖weight_v‖`), with no `weight_v`/`weight_g` split, while -`audiovae.cpp` requests the decomposed names directly via `require_f32`. The adapter solves -this without touching the VAE loader: - -``` -X.weight_v := folded GGUF tensor data (as-is) -X.weight_g := per-row L2 norms of the folded tensor, - computed with the loader's own row grouping - (groups = expected_shape.front(), inner = elements/groups) -``` - -Because `fold_weight_norm` multiplies row `d0` by `weight_g[d0] / ‖row d0‖ = 1`, the loader -output equals the GGUF data **byte-for-byte** — an exact identity, with no layout drift -relative to the reference runtime's consumption of the same bytes. The same mechanism works -for 3D `{out, in, k}` and 2D-flattened `{out, in·k}` conversions (element counts must match; -ranks may differ, covered by rank-tolerant `require_f32` + relaxed-rank anchors). - ---- - -## 5. Usage - -### Build - -```bash -scripts/build_linux.sh --backend cpu --target audiocpp_cli -# or, with the standard full model set: -cmake -S . -B build/linux-cpu-release -DCMAKE_BUILD_TYPE=Release -cmake --build build/linux-cpu-release --target audiocpp_cli -j 8 -``` - -### Run — 0.5B (16 kHz output) - -```bash -build/linux-cpu-release/bin/audiocpp_cli \ - --task tts --family voxcpm1 \ - --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf \ - --backend cpu --text "Hello from VoxCPM1." --out out.wav -``` - -### Run — 1.5B (44.1 kHz output) - -```bash -build/linux-cpu-release/bin/audiocpp_cli \ - --task tts --family voxcpm1 \ - --model models/VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf \ - --backend cpu --text "Hello from VoxCPM1." --out out.wav -``` - -### Options - -| Option | Values | Default | Meaning | -|---|---:|---:|---| -| `--task` | `tts` | required | Task kind. | -| `--family` | `voxcpm1` | auto-detect | Selects the v1 loader. | -| `--backend` | `cpu`, `cuda`, `vulkan`, `metal`, `hip`, `best` | `best` | Backend. | -| `--voice-ref` | WAV path | not set | Reference speaker audio (clone). | -| `--max-tokens` | integer | `4096` | Maximum generated AR tokens. | -| `--num-inference-steps` | integer | `10` | Flow-matching steps. | -| `--guidance-scale` | float | `2.0` | CFG strength. | -| `--session-option voxcpm1.mem_saver=true\|false` | bool | `false` | Tighter graph workspaces + release request graphs after completion. | -| `--session-option voxcpm1.prompt_cache_slots=` | integer | `1` | Prompt/prompt-audio embedding cache slots. | -| `--text-chunk-mode` | `default`, `tag_aware`, `japanese`, `endline` | `tag_aware` | Long-form chunking mode. | - ---- - -## 6. Validation performed - -- **Load + anchors:** all three GGUFs pass `validate_weight_anchors` and - `load_vae_weights`/`load_model_weights` on CPU. This includes the 0.5B 3D convs, the 1.5B - Q4_K 3D convs, and the 1.5B Q8_0 2D-flattened convs. -- **End-to-end:** `--task tts` completes for all three models; outputs are written as WAV at - the correct sample rate (16 kHz for 0.5B, 44.1 kHz for 1.5B) with active signal and - speech-plausible duration/envelope. -- **Regression:** the released voxcpm2 path is untouched (guard style `config.v1`, v2 default - `false`); voxcpm2 was not re-benchmarked but the changed code paths are v1-gated or - v1/v2-neutral. - -> ⚠️ **Quality caveat:** "end-to-end completes" does **not** mean the output is usable yet. -> See the known issue below — the audio is predominantly noise. - ---- - -## 7. Supported modes - -| Mode | Supported | Notes | -|---|---|---| -| **Offline TTS** | ✅ implemented | Default and only advertised mode. | -| **Streaming** | ❌ not implemented for v1 | `voxcpm1` advertises offline-only. Streaming is a v2 capability; it has not been validated (or enabled) for v1. | -| **Voice clone** | ⚠️ surface present | Speaker-reference options are advertised (`--voice-ref`), but quality is gated on the same known issue as plain TTS. | - ---- - -## Known issue: noisy output - -**Symptom.** Generated v1 voices are almost pure noise with a little human voice mixed in — -the signal is dominated by broadband/noise content. This affects all three GGUFs. - -**What is confirmed working.** Model loading, tensor adaptation, anchor validation, graph -construction, graph execution, and WAV output plumbing are all correct (no crashes, no -shape/size errors, correct sample rates and durations). The failure is therefore in the -**numerics of synthesis**, i.e. the audio content itself. - -**Most likely causes (in rough priority order).** - -1. **Weight data interpretation** — the identity fold preserves bytes, but if some AudioVAE - layer's storage layout (depthwise vs pointwise handling, 2D-flattened Q8_0, transposed - decoder `{in,out,k}` conventions, per-group row ordering of `weight_g`) differs from what - `ggml_conv_1d` / `conv_transpose` expects, the VAE decoder outputs garbage while loading - still "succeeds" (element counts match). -2. **Synthesized tensor semantics** — `feat_encoder.scale_embed/bias_embed`, `merge`, - `diag`, `extra_bias`, `fusion_concat_proj`, `stop_*`, and the identity `sr_cond_model` - tensors were synthesized with plausible but unverified semantics; if any is required to be - learned/zero-`scale` (or absent entirely in the reference runtime), the feature stream - feeding the LM/CFM is wrong. -3. **Graph parity vs the reference** — fusion = add and dit-mu = add were taken from - reference `build_residual_fusion_input`/`build_dit_mu`, but adjacent details (masking, - slice indices, position ids, prompt handling, FSQ rounding, CFM conditioning inputs, - ordering of `nn.Module` sub-blocks in the residual_lm stack) may differ. -4. **Sample-rate/codec mismatch** — 0.5B output asserted 16 kHz but the reference may expect - a specific internal feature rate; patch_size/feat_dim interplay (2·64 vs 4·64) feeding the - CFM estimator could be off by a constant factor, producing frozen-then-noisy patches. -5. **Quantization path** — the 1.5B Q8_0 GGUF quantizes the VAE itself (2D-flattened); - dequantized values feed `require_f32`, but a transpose or block-order mismatch would - corrupt every activation. - -**Debugging plan (next iteration).** - -- [ ] Port a small deterministic parity harness: run the same prompt through the reference - `VoxCPM.cpp` and audio.cpp, dump intermediate tensors (lm hidden, residual hidden, - CFM mu, VAE latent, decoder output) at each major stage, and diff numerically. -- [ ] Verify `encoder.fc_mu` / `decoder.model.{0,1,N}` folded data against the Python - reference weights with a strict per-element comparison on non-quantized tensors - (f16 VAE files), including row-grouping of `weight_g`. -- [ ] Check whether the reference runtime actually instantiates `sr_cond_model` and - `feat_encoder` synthesizable blocks for v1; remove or zero-scale any block the - reference does not run. -- [ ] Experimentally force one suspected block to a no-op (e.g. sr_cond identity, merge - zeros, scale_embed 0/1) and measure whether noise level drops. -- [ ] Validate CFM mu dimension/conditioning against the reference expectation for - `patch=2` (0.5B) and `patch=4` (1.5B). -- [ ] After the numerics match, run a human listening + loudness/spectral sanity check - (the current output has a spectral envelope consistent with noise + faint voice). - ---- - -## 8. Remaining tasks - -- [x] Loader registration, tensor adaptation, generator v1 branches, configs, model spec -- [x] End-to-end execution for 0.5B Q8_0, 1.5B Q4_K, 1.5B Q8_0 -- [ ] **Fix noisy output (known issue above) — top priority** -- [ ] Numerical parity harness vs `VoxCPM.cpp` reference (stage-by-stage tensor diff) -- [ ] `tests/voxcpm1/` automated path tests mirroring `tests/voxcpm2/` -- [ ] WebUI catalog entry (`webui/configs/models_catalog.json`) -- [ ] `docs/gguf.md` support-table entry -- [ ] CUDA-backend verification + RTF measurement (expect voxcpm2-like speedups) -- [ ] Streaming support for v1 (only meaningful after numerics are fixed) -- [ ] Commit + release packaging for `audio.cpp-gguf` (0.5B and 1.5B packages) \ No newline at end of file From 5b949e78b1408528036472c76c4bd4ba78966dfc Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Thu, 20 Aug 2026 00:25:56 +0200 Subject: [PATCH 14/14] generate new index.html based on merged code --- webui/native/dist/index.html | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/webui/native/dist/index.html b/webui/native/dist/index.html index 9ce05069..b875ab6b 100644 --- a/webui/native/dist/index.html +++ b/webui/native/dist/index.html @@ -5,7 +5,7 @@ - + @@ -13,20 +13,20 @@