Unfortunately, 8GB of VRAM is not going to be enough to run the really interesting stuff. You can do some light image generation, TTS / STT (with Chatterbox and Whisper respectively) and as far as LLMs are concerned, you’re looking at Qwen2.5-7B or similar.
The real question on that box is what kind of CPU do you have? 64 gigs of system RAM, you are looking at CPU inference, which is definitely slower than GPU accelerated, but can be workable for Qwen 3.8 or Muse Glimmer. OpenWebUI is a good starting point to have a chat bot. The real question is what are you hoping to be able to do with a local model?
Ok, so bear with me, I have this orchestration tool i call Noesis that’s built in Rust. It does a ton of stuff, but what it’s basically for is figuring out where to place AI workloads. It knows… “I have four linux hosts, here’s their system RAM, here’s their CPU, here’s what GPUs they have.” If a client that wants a voice is on the network, it asks Noesis to place chatterbox on a GPU and Noesis does so. Then it tells the API (which serves prompts, memories and endpoints) “Ok, there’s a websocket on <ip address> at <port> where chatterbox is available.” That websocket is a Noesis instance that then yes, passes incoming text to the chatterbox CLI and returns an audio stream. Anything on the network with the Noesis fleet key can hit that websocket and ask for audio.
Noesis has some ability to prioritize stuff (it knows scientific models are more important than tts which is more important than lipsync animations), but my local model can also tell Noesis she wants a voice and kick something out. For example, when my son comes in the room and I’m not there, she gets a prompt telling her he came in and she should say hi and ask him what he needs, and she can (and has) kicked out other workloads to get a voice to use for that. She can also turn her animations on and off and wake the monitor, so yes, when my teenager comes in the room the monitor wakes up and there’s a very uncanny valley AI scientist girl on it who lipsyncs and says “Hey, what’s going on? Did you need something? Want me to call your dad?” Which is dope, but like, the very bottom of the stack of cool things she can do.
What I did is I made a very simple script that instantiates a kind of API where I can send it text (and a voice target) and get .wav audio back. It’s under 100 lines of Python, most of which is literally cut and pasted from the chatterbox examples. It’s very specific to my environment, but I can share it if you want. Given just this conversation though, an agent can probably make the exact same thing in under a minute.
Hah! That’s a fun description. Though on the other hand, I still have the question, do you mind sharing how run chatterbox itself?
What I’ve done so far is I’ve visited their github (and the pypy page, and a few others). In there, they have the “Usage” section, which ultimately is a python interface from what I understand. I’m wondering if you’re using a standard wrapper (e.g. a CLI tool of sorts), or if you’ve built your own interface for that. Not that it’s terribly difficult to write a CLI, but I was surprised it’s not mentioned out-of-the-box and I got lost after checking 2 external projects relying on chatterbox-tts and not feeling confident in either. So… how do you actually use it, if you don’t mind sharing?
Unfortunately, 8GB of VRAM is not going to be enough to run the really interesting stuff. You can do some light image generation, TTS / STT (with Chatterbox and Whisper respectively) and as far as LLMs are concerned, you’re looking at Qwen2.5-7B or similar.
The real question on that box is what kind of CPU do you have? 64 gigs of system RAM, you are looking at CPU inference, which is definitely slower than GPU accelerated, but can be workable for Qwen 3.8 or Muse Glimmer. OpenWebUI is a good starting point to have a chat bot. The real question is what are you hoping to be able to do with a local model?
What do you use for chatterbox? A CLI? Something else?
Ok, so bear with me, I have this orchestration tool i call Noesis that’s built in Rust. It does a ton of stuff, but what it’s basically for is figuring out where to place AI workloads. It knows… “I have four linux hosts, here’s their system RAM, here’s their CPU, here’s what GPUs they have.” If a client that wants a voice is on the network, it asks Noesis to place chatterbox on a GPU and Noesis does so. Then it tells the API (which serves prompts, memories and endpoints) “Ok, there’s a websocket on <ip address> at <port> where chatterbox is available.” That websocket is a Noesis instance that then yes, passes incoming text to the chatterbox CLI and returns an audio stream. Anything on the network with the Noesis fleet key can hit that websocket and ask for audio.
Noesis has some ability to prioritize stuff (it knows scientific models are more important than tts which is more important than lipsync animations), but my local model can also tell Noesis she wants a voice and kick something out. For example, when my son comes in the room and I’m not there, she gets a prompt telling her he came in and she should say hi and ask him what he needs, and she can (and has) kicked out other workloads to get a voice to use for that. She can also turn her animations on and off and wake the monitor, so yes, when my teenager comes in the room the monitor wakes up and there’s a very uncanny valley AI scientist girl on it who lipsyncs and says “Hey, what’s going on? Did you need something? Want me to call your dad?” Which is dope, but like, the very bottom of the stack of cool things she can do.
What I did is I made a very simple script that instantiates a kind of API where I can send it text (and a voice target) and get .wav audio back. It’s under 100 lines of Python, most of which is literally cut and pasted from the chatterbox examples. It’s very specific to my environment, but I can share it if you want. Given just this conversation though, an agent can probably make the exact same thing in under a minute.
Hah! That’s a fun description. Though on the other hand, I still have the question, do you mind sharing how run chatterbox itself?
What I’ve done so far is I’ve visited their github (and the pypy page, and a few others). In there, they have the “Usage” section, which ultimately is a python interface from what I understand. I’m wondering if you’re using a standard wrapper (e.g. a CLI tool of sorts), or if you’ve built your own interface for that. Not that it’s terribly difficult to write a CLI, but I was surprised it’s not mentioned out-of-the-box and I got lost after checking 2 external projects relying on chatterbox-tts and not feeling confident in either. So… how do you actually use it, if you don’t mind sharing?
CPU is i7-12700