[Feature Request] support multiple references audios/texts

[ALameLlama]

Hey! I was looking into Fish speech and it looks like it supports having multiple references with audio/text https://docs.fish.audio/api-reference/sdk/javascript/api-reference#ttsrequest

it would be neat if it was possible to pass multiple --voice-ref and --reference-text, I don't believe all backends support this so maybe just pass through the first entry only for those?

[0xShug0]

@ALameLlama Thanks for the feature request! This is actually a missing feature in the current audio.cpp Fish Audio .
I checked the Python implementation, and it does support multiple reference audio/text pairs. It treats them as conditioning examples for one generation. I’ll find time to add this feature. The current plan is to pass multiple references through model-scoped request options, so this stays opt-in for Fish Audio and does not change the global request shape or other models’ behavior.

[handcraftedbits]

@0xShug0 I'm interested in this as well, but curious why this would be a request-scoped option? Wouldn't it make sense to be in voice_presets? Currently it's a map of voice name to a single reference, but couldn't it be extended to accept an array of references? And any model that doesn't support multiple references would result in a startup error.

[ALameLlama]

@0xShug0 I'm interested in this as well, but curious why this would be a request-scoped option? Wouldn't it make sense to be in voice_presets? Currently it's a map of voice name to a single reference, but couldn't it be extended to accept an array of references? And any model that doesn't support multiple references would result in a startup error.

I think this might be closer to my end goal, i want to be able to pass multiple references in a single request to have thr voice clone be more accurate

[0xShug0]

@handcraftedbits @ALameLlama I try to keep the generic voice layer generic. voice_ref, reference_text, and voice_presets are shared across many model families. They currently represent a single generic voice reference, and each model can decide how to use that reference.

Fish Audio’s multi-reference feature is different. It is an ordered set of paired (audio, text) conditioning examples for one generation. That pairing, ordering, and validation behavior are Fish-specific semantics.

If we put that directly into voice_ref or voice_presets, the generic server/framework layer would need to understand one-off Fish-specific semantics. That would make the common API less clean and predictable.

[handcraftedbits]

@0xShug0 yep I can respect that. If it's a request option though, will it still be possible to provide it as default_request_options? So that way it would be possible to still make a request that only includes the voice ID?

[0xShug0]

@handcraftedbits Yes. The multi reference can be added as a new fish audio request option like multi_reference_conditioning then the config

"default_request_options": {
  "fish_audio.multi_reference_conditioning": [
    {
      "audio": "/voices/alice_1.wav",
      "text": "First reference transcript."
    },
    {
      "audio": "/voices/alice_2.wav",
      "text": "Second reference transcript."
    }
  ]
}

[0xShug0]

Feature merged. Please check the fish audio doc for usage.