LocalLLaMA
Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.
Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.
As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.
Rules:
Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.
Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.
Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.
Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.
view the rest of the comments
Honestly I don’t use Qwen3 instruct unless it’s for code or “logic.” Even the 32B is soo dry and focused on that, and countering it with sampling dumbs it down.
Not sure if it’s too big, but I have been super impressed with Jamba 52B. It knows tons of fiction trivia and writing styles for such a “small” model, though I haven’t tried to manipulate its prompt for writing yet. And it’s an MoE model like A3B.
Interesting. I hadn't heard of this one before, but the design sounds innovative. The biggest I run is 35B or 7x8B, but I'll have to try and check it out.
Jamba is a killer model flying under the radar, though it does have a quirk I more recently discovered: no prompt caching in llama.cpp (yet).
If you have a 24GB GPU you can cram Nemotron 49B in it with no offloading, including the new reasoning version. It’s a monster at STEM stuff, and I can upload my special quantization (3bpw, with 4bpw KV heads, exllamav3) if you ask.
Qwen 30B coder is ridiculously fast for how smart it is at coding, just came out today…
TBH the last week or two has been nuts with new releases.
But FYI if you are looking for pure prose quality, I still use EVA Gutenberg 32B (based on Qwen 2.5 base) and Jonboro's brand new QWQ 32B fine tune, as new models have not surpassed them IMO. But for creative writing, I tend to write novel style instead of multi turn, so YMMV.
I only have a 16GB card, and my CPU is new enough that it's better to offload some layers of all but 7-8B models, so I haven't tried exllama, but you're making me think I should, if only for comparison.
I like Qwen 2.5 based models in the 14B size range, but I don't think I tried the bigger ones. I tried the QWQ and didn't really like it, but I haven't seen this new one. You've given me a whole list of things to try, so thanks.
Is it 3000 series or newer?
If so, with exllamav3, you can squeeze 32Bs in that 16GB card with relatively little loss. For instance: https://huggingface.co/turboderp/EXAONE-4.0-32B-exl3/tree/3.0bpw
The 3bpw weights are 13 GB, say another 1.5GB for some q5_q4 context, and you are looking at 14.5GB-15GB or so. It will be tight, but it will be leagues smarter than 14Bs.
24B Mistral models will fit much more easily. No need to CPU offload those on a 16GB card, you just need to be careful with your settings.