For me it's be Strix Halo, 128gb machine, especially running Qwen models. Except when I bought it, it was $1,900, now it's $4,600 for the same box. (Wow that's insane)
For tinkering and learning, it's been great. Tie it into something like Hermes and you have a pretty powerful AI assistant in a box. And when you need to step up your model, you just do something like OpenRouter and it makes it pretty easy.
I have been looking for a good local setup for a while now. Qwen 3.8 27b is really good for a dense model of this size IMO. I already had an RTX 4090 and I forked ninfer [0] with the obsession to squeeze everything out of this card for this model. Results: 149 tok/s decode speed (aggregate with concurrency about ~270 tok/s) with prefill speeds faster than 2500 tok/s. And all of this with full 262k albeit quantized context. Fast prefill speed is really important when launching multiple clients such as opencode or pi at the same time and especially if they launch subagents. This is why I also implemented a caching tier so computed contexts can be faster loaded from RAM (or disk). Speeds feel almost like with official SOTA openai or anthropic models.
Im currently measuring a pareto front in J/tok in order to set power limits of this card without sacrificing too much performance. Since we are talking about full power draw of ~480W which is fine during the day (with solar panels) but during night when the sun doesn't shine (even with a battery) I'd like to limit this a little bit.
Sure, just consumer grade products: an ~8yr old 16 core AMD Threadripper CPU + 64 GB RAM, Samsung NVMe drive(s) and obviously an nvidia RTX 4090 (watercooled). Only the GPU is important though.
