rkd

joined 5 days ago
sorted by: new top controversial old
So image generation is where it's at? in c/localllama@sh.itjust.works
[–] rkd@sh.itjust.works 1 points 12 hours ago

That's a good point, but it seems that there are several ways to make models fit in smaller memory hardware. But there aren't many options to compensate for not having the ML data types that allows NVIDIA to be like 8x faster sometimes.

So image generation is where it's at? in c/localllama@sh.itjust.works
[–] rkd@sh.itjust.works 1 points 12 hours ago

For image generation, you don't need that much memory. That's the trade-off, I believe. Get NVIDIA with 16GB VRAM to run Flux and have something like 96GB of RAM for GPT OSS 120b. Or you give up on fast image generation and just do AMD Max+ 395 like you said or Apple Silicon.

So image generation is where it's at? in c/localllama@sh.itjust.works
[–] rkd@sh.itjust.works 3 points 23 hours ago (4 children)

I'm aware of it, seems cool. But I don't think AMD fully supports the ML data types that can be used in diffusion and therefore it's slower than NVIDIA.

13
So image generation is where it's at? (sh.itjust.works)
 

Total noob to this space, correct me if I'm wrong. I'm looking at getting new hardware for inference and I'm open to AMD, NVIDIA or even Apple Silicon.

It feels like consumer hardware comparatively gives you more value generating images than trying to run chatbots. Like, the models you can run at home are just dumb to talk to. But they can generate images of comparable quality to online services if you're willing to wait a bit longer.

Like, GPT OSS 120b, assuming you can spare 80GB of memory, is still not GPT 5. But Flux Shnell is still Flux Shnel, right? So if diffusion is the thing, NVIDIA wins right now.

Other options might even be better for other uses, but chatbots are comparatively hard to justify. Maybe for more specific cases like code completion with zero latency or building a voice assistant, I guess.

Am I too off the mark?

It's uh drugs, yea that's it! in c/localllama@sh.itjust.works
[–] rkd@sh.itjust.works 12 points 3 days ago (2 children)

it's most likely math

Nintendo-owned titles excluded from Japan’s biggest speedrunning event after organizers were told they had to apply for permission for each game in c/games@sh.itjust.works
[–] rkd@sh.itjust.works 11 points 3 days ago

Congratulations Nintendo, you played yourself.