Voice Mixing --> Saved Voice Names

[raf-alencar]

I had an idea to begin testing out voice mixing, and create a table with the mixes I particularly liked.
I have been keeping all of this in a separate excel that I reference with the new name, so it picks up the voice mix string so I can send in my request.

3 things I thought.

1. I think more people are likely doing the same, so it would be cool to maybe have a place to work together in listing interesting voice mixtures and naming them.

2. is there a way I can officially rename the voices in my environment so that I don't have to run a separate table to link my given name to a voice mix (this would be cool if it was also done through API calls)

3. I'm sure someone knows of a tool that can give the stats of a voice. like pitch, tone etc.. if so, could you post it here? It wouldn't be perfect voice cloning, but it seems pretty feasible to build a small program that could automatically calculate the voice mix needed to obtain the desired voice outcome. like cloning, but cheaper/simpler I guess.

[radzrader]

as an FYI more than anything. i have been playing for a voice studio page for something like this

i am going to polish it up. the final version wouldnt have the upload voice (i was trying to use gemma 4 12b to try to find to make a voice, but this wasnt great)

@remsky

[RBEmerson970]

I like it! I'm looking forward to using it.

[remsky]

Hey @raf-alencar, @radzrader

I have a branch up that has some of these features. Its part of changes allowing multi-speaker to be handled inline without impacting existing. Will be merging it to master relatively soon

https://github.com/remsky/Kokoro-FastAPI/tree/feat/multi-speaker

It's currently set up to be able to pass through the OpenAI / API as an optional voice_aliases JSON param as well; it's not persistent to the server, but will have a matching Save/Import Voice Map in the Web UI.

(Also been looking at the pitch/tone/timber adjustments, but it's a little more shaky/experimental at the moment, so no promises on it.)

@radzrader (will credit you as a co-author on the PR since you had similar parallel work going)

[radzrader]

Much more refined! Like it 😁

[RBEmerson970]

I was able to duplicate the demo shown above. Got it on the first try, for that matter.

A suggestion: figure out a way to create cast pages or store document casts. EDIT: Strike that suggestion - I found save and load tags options, which achieves what I had in mind.

[RBEmerson970]

GitHub/Docker newb question: I'd like to concurrently keep a container with v0.7.1 and a container with v0.8.0-rc1. Advice? How-to?

[remsky]

@RBEmerson970 As a heads up, it is a dev branch, and will have changes/fixes being pushed still.

Easiest way would likely be to run the prebuilt image directly of the released version as the secondary. You could create a separate compose file and run it with the params below set, or just run the command itself. The main collision for concurrent use will be the web port (8880), so you'd have to map another port for it. e.g.

docker run -d --name kokoro-071 -p 8881:8880 --gpus all ghcr.io/remsky/kokoro-fastapi-gpu:v0.7.1

or for the cpu image
docker run -d --name kokoro-071 -p 8881:8880 ghcr.io/remsky/kokoro-fastapi-cpu:v0.7.1

Then it'd be in the browser at http://localhost:8881/web

[RBEmerson970]

Sorry - poor wording on my part - I don't plan to run v0.7.n and v0.8.n concurrently (yes, that could pose a port collision), only concurrently have each version in separate containers. It occurs to me I can name containers as they're being built, which AFAIK should resolve the problem of "same name, different version". Just change the container name locally. Doh.

[RBEmerson970]

Back on topic... I tested a 14K character text, injecting [voice:xxx] as needed. It worked quite well. I initially used [voice:af-alloy] in my text, and was surprised and pleased to see the tag change, in the text, from [voice:af-alloy] to [voice:Sandy] when I created that cast member. Very cool!
Overall, the voice changes are transparent (i.e., no clicks, stutters, or noise). For example,
[voice:Jessica]"I'm not really bad. I'm just drawn that way, [voice:Narrator] Jessica said as she swayed her way across the room. [voice:Roger]"Gulp" [voice:Narrator] was all Roger Rabbit could say.
works just fine.

Also, the change from a "Pause" button to "||" or "|>" is a nice touch. Any thoughts about adding scrobbling through a generated text?

[remsky]

Great! If you catch any bugs feel free to submit.

Scrobbing properly is doable, but it comes with a number of performance/stability trade-offs in the current state I don't want to dig into yet. Once some other changes go in can take another look at it, but I'll see if I can add a safer approximate version in the interim.

[RBEmerson970]

Please count this as a "the thought occurred to me in passing" suggestion rather than an outright request. I count being able to adjust voice qualities (speed, pitch, timbre) as far more important.

[RBEmerson970]

I just built v0.8.0-rc1 again (playing with trying to have v0.7.1 and v0.8.0-rc1[*]) and found the "read along" button. Nice touch!

Various voices speak at different rates. Is there hope for including speed as an attribute in a cast definition?

On occasion I've fat-fingered a cast tag and accidentally inserted it in the text. AFAIK CTRL-Z doesn't remove a voice tag insertion. Maybe it should?

---

[*]Something in the build process creates kokoro-fastapi-gpu as well as the actual container kokoro-fastapi-gpu-kokoro-tts-1. Renaming kokoro-fastapi-gpu-kokoro-tts-1 to kokoro-fastapi-gpu-kokoro-tts-Trial or kokoro-fastapi-gpu-kokoro-tts-Master fails. Any build creates kokoro-fastapi-gpu all over again, wiping out the renamed container. For now I'll build as needed (using different repo clones in different file structures); that, at least, works, if with less than maximum elegance.

[RBEmerson970]

More feedback: Hitting the "..." in a tag takes a bit of aiming. Miss, and the tag's inserted and has to be edited out (see above re: ). Overall v0.8.0-rc1 (maybe rc2???) works well.

[remsky]

Can make it a little more user friendly to help with mis-inserts at least, though ctrl-z is a little touchy to chain through. Will take a look at what's doable.

[RBEmerson970]

Agnostic about ctrl-z, just interested in a quick "forget I did that".

[RBEmerson970]

Tried the SSML mods message and tried it to check the changes in the UI. The undo button works for me. The drop-down for saving and loading casts slides off the screen on the right side (due to colliding with the undo button?)
The line per cast member looks tidy but burns a lot of window space for a short name in a long line. The < to insert a tag is OK, but why not allow clicking on the name instead? With the "Options" field behind the name, there's less risk of fat-fingering an insertion by mistake.

[remsky]

Just following up a test here, it should still be respecting inline rate tags as well e.g

[RBEmerson970]

A question about speed: I create a cast member "Fred" with speed 0.8. I insert [rate:1.1] in the text and later use "Fred", with speed 0.8. Is the resulting speed for "Fred" 0.8x1.1 or 1.1? Put another way, does [rate:n.n] override, within the spoken text, cast speed settings or modify them? (I assume the mix settings in the cast list remain unaltered)

[remsky]

It overrides it independently (not modify or multiply), so the speed would be 1.1 after that tag drops (until another voice tag is read)

edit: So in your example, the later Fred would just go back to their 0.8, and yup, it has no impact on saved cast speeds).

[RBEmerson970]

In which case I'll suggest figuring out whether not implementing "modify" (i.e., 0.8x1.1) is practical. The ax_xxxx voices, at least, are all over the map on speed. I've only used a couple of British voices, but suspect the same holds with bx_xxxx voices. So, if I throttle af_Kore to 0.8, use [rate:1.1], Kore turns into a motor-mouth. Not Good. The easiest fix is to avoid [rate:n.nnx] altogether, which I'm inclined to do anyway. Others may see things differently, though.

[remsky]

I might include the change here; does actually look like it aligns with the SSML standard.

Edit: #505 Let me know if you have a chance to try this out, seems to test well

[RBEmerson970]

Missed that change; looked at an existing cast, and saw where speed control goes. Clever!

Now, if you could only get to the point of inferring which tag goes where based on reading the text... ;D

Serious question: do you have any sense of further development for Kokoro-82M? Its emotional range is like the proverbial poor actor: "a range of A to B". And, of course, there's zero-shot TTS floating about...

[remsky]

Now, if you could only get to the point of inferring which tag goes where based on reading the text

That's doable to a point with some existing diarizations libraries, but it would add a lot of feature surface, so doubtful here, but if a stable path, spare time, or a PR appears, I'm open to it.

further development for Kokoro-82M, zero-shot TTS

Mostly just exploring and focusing on the strengths of this model. It's faster and lightweight than most, and the lower param count is part of the package. I do have a few dials and knobs that will be in the release after this to allow some level of customization to tone, pace, and tuning of the voices, but they won't be able to match the emotional range or cloning ability of the bigger models.

[RBEmerson970]

"Automagic tagging" was said in jest. I see where it would turn a pocket Swiss Army knife into something needing a flatbed truck to haul it around.

Probably the biggest emotion-related "needs attention" is response to a sentence or phrase ending in "?". More often than not, no rising inflection or other response.

On punctuation-related speech, AFAIK, an em dash doesn't produce a pause. Yes, [pause:0.25s] gives the needed pause. :)

Homonyms are also troublesome. My main use is proof-speaking my writing. If I were writing something for, say, narration, I'd adjust the words to get the right sound. For example "I red in the news that..." in place of "I read in the news that...". Some cases are harder: Does "live" get a long or short "i" - "I live near a live tree".

[RBEmerson970]

Latest build, just completed, looks great. BTW, like the new FastKoko logo/banner here.