I’m new to local llms, i’m running Debian on this machine. Where should I start and what model(s) could I get testing on it?

  • Spite_By_Design_LLC@lemmy.dbzer0.comEnglish
    31·
    5 days ago

    Depends what your interests are! I can get around 35ish tok/s TG at - c 48,000 of Qwen 3.6-35B-A3B (Q4_K_M) using polystrata on a similar rig (you’ve got an A1000, I have a T1000).

    Which means absolutely nothing to you right now but suffice to say - Polystrata + Qwen 35B is something to try. It’s among the “smartest” you can run on your rig.

    Personally, for every day use, I like Ling 3.0 Tiny at Q6 or SharpMiniCPM5-2B at Q8. They are stupidly fast, and supported by MCP tools and a bridge I wrote, work as excellent “hey, I’m thinking of X. Can you tell me about it? And what about Y and Z?” or “How does this read?”.

    I pull out my phone, send them a text message and they text me back.

    Think of it like a Google in your pocket, except off line (unless you tell it to go online). Which technically I suppose makes it agentic lol.

    (I can show you some examples if you’d like. I keep a set list of 8 tasks I use for benchmarking as I tweak things, as well as some off the wall / out of band queries)

    Being that Ling or CPM only take up a sliver of the GPU, I can run them concurrently with other models (like an image generator, text to speech, vision etc) to create a really useful offline house brain, that can roam across my local documents, offline wikipedia (Kiwix) and the like.

    It’s the combination of things that results in a 1+1=3

    Having said that, my AI ladder is

    • 90% Ling or minicpm5
    • 9% Qwen 3.6-35B
    • 1% GLM5.3 Flash via open router… which costs fractions of a cent for what I need (there are tricks to this but it’s probably out of scope to info dump it on you at this stage).

    For now, I keep a $20 ChatGPT Plus (separate pools for chat and Codex have been so good) but I am ditching that asap, with the changes coming this month.

    HTH. Write back if you need anything else. It’s nice to see someone else rocking an A1000 / T1000. Your rig should be significantly faster than mine… which is why the A1000 is still stupidly expensive for a 50W card.

    EDIT: just saw you said “CPU is i7-12700”. With that RAM, that CPU and that GPU, I’d guess you’ll hit 50-60 tok /s with Qwen 3.6-35B using polystrata once you get the GPU / CPU split right.

    In other words, congrats and go fuck yourself :) You have a very nice little inference box. And yes, it can do your calendar alerts thing.

  • The Bard in GreenAEnglish
    21·
    6 days ago

    Unfortunately, 8GB of VRAM is not going to be enough to run the really interesting stuff. You can do some light image generation, TTS / STT (with Chatterbox and Whisper respectively) and as far as LLMs are concerned, you’re looking at Qwen2.5-7B or similar.

    The real question on that box is what kind of CPU do you have? 64 gigs of system RAM, you are looking at CPU inference, which is definitely slower than GPU accelerated, but can be workable for Qwen 3.8 or Muse Glimmer. OpenWebUI is a good starting point to have a chat bot. The real question is what are you hoping to be able to do with a local model?

      • The Bard in GreenAEnglish
        1·
        2 days ago

        Ok, so bear with me, I have this orchestration tool i call Noesis that’s built in Rust. It does a ton of stuff, but what it’s basically for is figuring out where to place AI workloads. It knows… “I have four linux hosts, here’s their system RAM, here’s their CPU, here’s what GPUs they have.” If a client that wants a voice is on the network, it asks Noesis to place chatterbox on a GPU and Noesis does so. Then it tells the API (which serves prompts, memories and endpoints) “Ok, there’s a websocket on <ip address> at <port> where chatterbox is available.” That websocket is a Noesis instance that then yes, passes incoming text to the chatterbox CLI and returns an audio stream. Anything on the network with the Noesis fleet key can hit that websocket and ask for audio.

        Noesis has some ability to prioritize stuff (it knows scientific models are more important than tts which is more important than lipsync animations), but my local model can also tell Noesis she wants a voice and kick something out. For example, when my son comes in the room and I’m not there, she gets a prompt telling her he came in and she should say hi and ask him what he needs, and she can (and has) kicked out other workloads to get a voice to use for that. She can also turn her animations on and off and wake the monitor, so yes, when my teenager comes in the room the monitor wakes up and there’s a very uncanny valley AI scientist girl on it who lipsyncs and says “Hey, what’s going on? Did you need something? Want me to call your dad?” Which is dope, but like, the very bottom of the stack of cool things she can do.

        • The Bard in GreenAEnglish
          1·
          2 days ago

          What I did is I made a very simple script that instantiates a kind of API where I can send it text (and a voice target) and get .wav audio back. It’s under 100 lines of Python, most of which is literally cut and pasted from the chatterbox examples. It’s very specific to my environment, but I can share it if you want. Given just this conversation though, an agent can probably make the exact same thing in under a minute.

        • vas@lemmy.mlEnglish
          1·
          2 days ago

          Hah! That’s a fun description. Though on the other hand, I still have the question, do you mind sharing how run chatterbox itself?

          What I’ve done so far is I’ve visited their github (and the pypy page, and a few others). In there, they have the “Usage” section, which ultimately is a python interface from what I understand. I’m wondering if you’re using a standard wrapper (e.g. a CLI tool of sorts), or if you’ve built your own interface for that. Not that it’s terribly difficult to write a CLI, but I was surprised it’s not mentioned out-of-the-box and I got lost after checking 2 external projects relying on chatterbox-tts and not feeling confident in either. So… how do you actually use it, if you don’t mind sharing?