• 2 Posts
  • 17 Comments
Joined 6 days ago
Cake day: October 31st, 2026
  • Spite_By_Design_LLC@lemmy.dbzer0.comOPtoLocalLLaMA@sh.itjust.works•British fintech Revolut slashes AI coding costs 10x by cutting out Claude and going self-hostedEnglish
    2·
    18 hours ago

    Agreed. Economies of scale are about the only way this makes sense.

    I was just working thru the use case for purchasing a 64gb Strix (vs the typical coding I might do using OR or Open Code Go). The ROI is somewhere in the order of 7-10 years for me. So, on a purely dollar cost basis, local inference is $$$. OTOH, from a sovereignty and “own your own code” basis…there’s something to be said for it.

    Probably I will run a few tests using OR with a few outstanding tickets of mine, to kick the tyres of Qwen3.8-27B. I’ve tried Qwen3.6-35B and frankly it’s…not very good at code gen or patching.

    To boot, code gen isn’t the only thing I use LLM for, so I suppose it depends on how bloody minded I want to be about future use case (I do actually have one).

  • Spite_By_Design_LLC@lemmy.dbzer0.comOPtoLocalLLaMA@sh.itjust.works•British fintech Revolut slashes AI coding costs 10x by cutting out Claude and going self-hostedEnglish
    1·
    23 hours ago

    BRB, just getting some scratch lotto tickets.

    14.3 kw. That’s fucking obscene.

    And by all rights, the Fintech datacentre is what…small? medium?

    GLM again:

    One DGX B200 flat-out: 14.3 kW × 24h = ~343 kWh/day ≈ what a typical UK house uses in a month

    Datacentre size: a full rack of AI gear like this is ~50–100kW. Classification-wise:

    Your average office server closet: 5–20 kW
Small datacentre: <1 MW
Mid-size / enterprise: 1–10 MW
Hyperscale (AWS/Google/Microsoft): 50–500+ MW per campus

    Revolut’s 56 GPUs ≈ one dense rack or two — well under 1 MW, so firmly “small,” and that’s exactly why they colocate/rent space (“a US-based service”) rather than owning a facility. Self-hosting the GPUs doesn’t mean self-hosting the building. You don’t put 100kW of fridges in your London office — you rent rack space in a proper colo where the power and cooling already exist.

    Enjoy the lotto. If you win, you can buy an eighth of a DGX.

    Fuuuuuuuuuck

  • Spite_By_Design_LLC@lemmy.dbzer0.comOPtoLocalLLaMA@sh.itjust.works•British fintech Revolut slashes AI coding costs 10x by cutting out Claude and going self-hostedEnglish
    2·
    23 hours ago

    In the spirit of things, I asked GLM5.3 what that set up actually looks like. Ouch.

    Fetched that page for real. Specs (from gpuworld’s DGX B200 datasheet page):

    Dimensions: 17.5" W × 87.5" D × 42.9" H (cm: 44.5 × 222.3 × 109.0)
Weight: 275 lb (124.7 kg)
Rack height: 10U equivalent
Power: up to 14.3 kW max per system

    In real-world terms:

    Height: 10U = 10 rack units = 10 × 1.75" = 17.5" tall — about the height of a bar fridge standing on its side. But that's the chassis; you'll also want ~2U of clearance for airflow/management, so plan for ~1.5U of breathing room above it in a rack.
Depth: 87.5cm = just under a metre — so it'll stick out of a standard 1m-deep server rack unless you've got deep racks (or you let the back panel hang open). Standard 42U racks are typically 1.1–1.2m deep to handle this.
Weight: 275 lb / 125kg. That's a hefty man's weight — two people minimum to lift, ideally four or a rack lift. Standard datacentre floor loading is usually fine at this density but worth checking if you're stacking multiple in a rack.

    Rack math for 7 units: 70U of just these systems — so a single 42U rack can’t hold them all (you’d need 2 racks), and at ~14kW each you’re looking at ~100kW total draw — that’s serious power/cooling, way beyond a standard office closet.

    Short version: think “fridge-sized box, half a tonne with rails, eats 14kW” — not a pizza box.

  • Spite_By_Design_LLC@lemmy.dbzer0.comtoLocalLLaMA@sh.itjust.works•I have a RTX A1000 (8GB) with 64GB DRR5, what would you recommend for a newbie?English
    3·
    4 days ago

    Depends what your interests are! I can get around 35ish tok/s TG at - c 48,000 of Qwen 3.6-35B-A3B (Q4_K_M) using polystrata on a similar rig (you’ve got an A1000, I have a T1000).

    Which means absolutely nothing to you right now but suffice to say - Polystrata + Qwen 35B is something to try. It’s among the “smartest” you can run on your rig.

    Personally, for every day use, I like Ling 3.0 Tiny at Q6 or SharpMiniCPM5-2B at Q8. They are stupidly fast, and supported by MCP tools and a bridge I wrote, work as excellent “hey, I’m thinking of X. Can you tell me about it? And what about Y and Z?” or “How does this read?”.

    I pull out my phone, send them a text message and they text me back.

    Think of it like a Google in your pocket, except off line (unless you tell it to go online). Which technically I suppose makes it agentic lol.

    (I can show you some examples if you’d like. I keep a set list of 8 tasks I use for benchmarking as I tweak things, as well as some off the wall / out of band queries)

    Being that Ling or CPM only take up a sliver of the GPU, I can run them concurrently with other models (like an image generator, text to speech, vision etc) to create a really useful offline house brain, that can roam across my local documents, offline wikipedia (Kiwix) and the like.

    It’s the combination of things that results in a 1+1=3

    Having said that, my AI ladder is

    • 90% Ling or minicpm5
    • 9% Qwen 3.6-35B
    • 1% GLM5.3 Flash via open router… which costs fractions of a cent for what I need (there are tricks to this but it’s probably out of scope to info dump it on you at this stage).

    For now, I keep a $20 ChatGPT Plus (separate pools for chat and Codex have been so good) but I am ditching that asap, with the changes coming this month.

    HTH. Write back if you need anything else. It’s nice to see someone else rocking an A1000 / T1000. Your rig should be significantly faster than mine… which is why the A1000 is still stupidly expensive for a 50W card.

    EDIT: just saw you said “CPU is i7-12700”. With that RAM, that CPU and that GPU, I’d guess you’ll hit 50-60 tok /s with Qwen 3.6-35B using polystrata once you get the GPU / CPU split right.

    In other words, congrats and go fuck yourself :) You have a very nice little inference box. And yes, it can do your calendar alerts thing.