I’m looking to locally generate voiceovers from text and also try to generate audiobooks. Does anyone have experience with sherpa-onnx? There also appear to be two separate frontends for Kokoro specifically dedicated for audiobook creation, but they appear to both be abandoned. Or am I barking up the completely wrong tree?
Thanks!

  • hendrik@palaver.p3x.deEnglish
    10·
    6 days ago

    Sherpa looks like a nice all-in-one solution. I’m just not sure if Kaldi still is the latest and greatest when it comes to high quality TTS.

    I’ve tried Kokoro, Orpheus, Chatterbox and those sound very nice, but it’s just the TTS. And then a few ones I haven’t tried are fish and OuteTTS. And then there is Kaldi and xTTSv2.

    There are some projects building upon TTS to create similar things, though I’ve mainly looked into the dialogue and podcast creation:

    • Dia (does dialogue, works for English and can do a few rudimentary things like laugh and chuckle)
    • Mozilla’s Document-to-Podcast
    • Chatterbox-Audio is someone trying to do audiobooks with Chatterbox, though I’m not sure about the quality and progress of the attempt
    • And a few abandoned attempts like Podcastify, Podcast-LLM from when Google introduced NotebookLM and people tried to recreate that. But I don’t think it’s very helpful if you want a tool that actually works.

    I ended up writing a Python script myself to split up the text into paragraphs, feed it to Kokoro/Orpheus and then stitch back together the generated audio into a dialogue. I didn’t find anything that did the exact thing I wanted to do. And Dia gave me a particularly bad time because the talking speed was all over the place and the voices weren’t very consistent.

    • splendoruranium@infosec.pubOPEnglish
      3·
      6 days ago

      Thanks, that’s a lot of pointers!
      I’m fine with not the absolutely latest and greatest. In fact, since basically all of this is still cutting-edge I’m mainly trying to weed out the hype, the scams and the malware and at least find the somewhat well-trodden paths. It’s all terribly fascinating but casually downloading month-old programs from random github repos and running cryptic docker containers with instructions that are half in Chinese and half in LLM-generated English (which still ends up too technical for me to fully understand) is kind of… stressful for me. That’s just not how I usually use a computer 😅

      • hendrik@palaver.p3x.deEnglish
        2·
        6 days ago

        Yes. I also run into the same issues. I haven’t had scams and malware yet. But I frequently read some news about the latest nice thing, and then I’m hyped and want to use it. But oftentimes it’s a tech-demo and it would need months of additional work to make it usable for real-world applications. And then we have some companies who just dump stuff to Github. Including quite some Chinese startups and such. More than once I’ve downloaded something and it didn’t work and it couldn’t ever work because the Python code had several errors or wasn’t complete. They often upload something different from what they used for their experiments or tech-demo. And then AI moves so fast, people will have moved on to the next thing before finishing the previous one. So we also end up with a lot of abandoned projects which never got to the state where they’d be useful. It’s really hard in the field of AI.

        (And a bit unrelated, but I’m a bit disappointed that we don’t have good TTS built right into our Linux desktop. Like Kokoro, just multilingual and with an open training set, plus the right toolkit to tie it into the desktop, browser, commandline and arbitrary tools building on top of it.)