Ok, so bear with me, I have this orchestration tool i call Noesis that’s built in Rust. It does a ton of stuff, but what it’s basically for is figuring out where to place AI workloads. It knows… “I have four linux hosts, here’s their system RAM, here’s their CPU, here’s what GPUs they have.” If a client that wants a voice is on the network, it asks Noesis to place chatterbox on a GPU and Noesis does so. Then it tells the API (which serves prompts, memories and endpoints) “Ok, there’s a websocket on <ip address> at <port> where chatterbox is available.” That websocket is a Noesis instance that then yes, passes incoming text to the chatterbox CLI and returns an audio stream. Anything on the network with the Noesis fleet key can hit that websocket and ask for audio.
Noesis has some ability to prioritize stuff (it knows scientific models are more important than tts which is more important than lipsync animations), but my local model can also tell Noesis she wants a voice and kick something out. For example, when my son comes in the room and I’m not there, she gets a prompt telling her he came in and she should say hi and ask him what he needs, and she can (and has) kicked out other workloads to get a voice to use for that. She can also turn her animations on and off and wake the monitor, so yes, when my teenager comes in the room the monitor wakes up and there’s a very uncanny valley AI scientist girl on it who lipsyncs and says “Hey, what’s going on? Did you need something? Want me to call your dad?” Which is dope, but like, the very bottom of the stack of cool things she can do.
Hah! That’s a fun description. Though on the other hand, I still have the question, do you mind sharing how run chatterbox itself?
What I’ve done so far is I’ve visited their github (and the pypy page, and a few others). In there, they have the “Usage” section, which ultimately is a python interface from what I understand. I’m wondering if you’re using a standard wrapper (e.g. a CLI tool of sorts), or if you’ve built your own interface for that. Not that it’s terribly difficult to write a CLI, but I was surprised it’s not mentioned out-of-the-box and I got lost after checking 2 external projects relying on chatterbox-tts and not feeling confident in either. So… how do you actually use it, if you don’t mind sharing?
Ok, so bear with me, I have this orchestration tool i call Noesis that’s built in Rust. It does a ton of stuff, but what it’s basically for is figuring out where to place AI workloads. It knows… “I have four linux hosts, here’s their system RAM, here’s their CPU, here’s what GPUs they have.” If a client that wants a voice is on the network, it asks Noesis to place chatterbox on a GPU and Noesis does so. Then it tells the API (which serves prompts, memories and endpoints) “Ok, there’s a websocket on <ip address> at <port> where chatterbox is available.” That websocket is a Noesis instance that then yes, passes incoming text to the chatterbox CLI and returns an audio stream. Anything on the network with the Noesis fleet key can hit that websocket and ask for audio.
Noesis has some ability to prioritize stuff (it knows scientific models are more important than tts which is more important than lipsync animations), but my local model can also tell Noesis she wants a voice and kick something out. For example, when my son comes in the room and I’m not there, she gets a prompt telling her he came in and she should say hi and ask him what he needs, and she can (and has) kicked out other workloads to get a voice to use for that. She can also turn her animations on and off and wake the monitor, so yes, when my teenager comes in the room the monitor wakes up and there’s a very uncanny valley AI scientist girl on it who lipsyncs and says “Hey, what’s going on? Did you need something? Want me to call your dad?” Which is dope, but like, the very bottom of the stack of cool things she can do.
Hah! That’s a fun description. Though on the other hand, I still have the question, do you mind sharing how run chatterbox itself?
What I’ve done so far is I’ve visited their github (and the pypy page, and a few others). In there, they have the “Usage” section, which ultimately is a python interface from what I understand. I’m wondering if you’re using a standard wrapper (e.g. a CLI tool of sorts), or if you’ve built your own interface for that. Not that it’s terribly difficult to write a CLI, but I was surprised it’s not mentioned out-of-the-box and I got lost after checking 2 external projects relying on chatterbox-tts and not feeling confident in either. So… how do you actually use it, if you don’t mind sharing?