It should work in any generic cuda container, but yeah it’s more of a hobbyist engine. Honestly I just run it raw since it’s dependency free, except for system CUDA.
Vllm absolutely cannot CPU offload AFAIK, but small models will fit in your vram with room to spare.
I have not had to really mess with CachyOS for over a year, while “stable” distros were a nightmare for me.
…Yeah it just depends what you’re trying to get your system to do. Arch can range from incredibly hazardous to “it just works” depending on the person and thing, and so can Mint. I think most distros should be viewed that way.
I have trouble looking at my life when things are bad, TBH, especially dull/isolated kind of bad. It is not pleasant to write.
Writing some fiction for the sake of writing (and weaving personal issues in) has actually lead to some insights, though. And maybe some feeling/reality processing going forward. And happiness.
And when things get better, I want to journal (which I have failed to make time for in the past).
We will eventually have to change our economic system and adapt one with a much much lower consumption rate, figure ways to limit our population growth, or more than likely both.
Of course. 100% agree with this, even if better technology helps. It will have to be pretty soon.
But in the very short term? This is going to be a disaster, and the human population is shooting itself in the foot by not accepting immigration from extreme birth rate countries (where overpopulation is indeed an issue).
I get the joke, but how does onlyfans slide by?
WTF? Do they have dirt on finance execs or something?
…Actually that would make a lot of sense…
Thanks, I nearly choked on my drink imagining that…
Thanks for the TED talk (really)
You have to “unlock” them with a lot of tweaks. And to be clear, I’m just saying they’re better than Windows. Ugh, trying to compile anything on Windows…
Hardware wise, they’re far better for local code assistants, too, with the exception of a few exotic AMD laptops just now coming out.
I cannot concentrate with music playing.
Literally anything else? Like a truck crashing through the window or someone literally trying to engage me? Zero lost focus, to the extent I was initially diagnosed with a hearing disorder.
Not that that helps get stuff done…
You should watch the video. I used to be an overpopulation doomer, and it completely changed my perspective, especially when I dug into papers backing it up.
I dunno what you are getting at, because massive immigration has been an economic miracle for the USA. It’s ballooned the population, drained brains from other countries, and bolstered entire industries.
You know what gives industrialized countries a good quality of life? A young population with a high birth rate, so that the working population isn’t overburdened taking care of retirees, as Korea and Japan are existentially facing now. And low wealth inequality, which is a definite unrelated problem in the US.
The USA has a low birthrate. And immigration has completely made up for it because they skew young.
Take that away?
http://www.youtube.com/watch?v=Ufmu1WD2TSk
From a purely economic quality of life perspective, setting all morals aside, the US should pull in as many immigrants as it can. We're basically screwed now.
Meanwhile, the fallacy of immigration hurting wages is basically the South Park episode “Goobacks”
This is one of the "smartest" models you can fit on a 24GB GPU now, with no offloading and very little quantization loss. It feels big and insightful, like a better (albeit dry) Llama 3.3 70B with thinking, and with more STEM world knowledge than QwQ 32B, but comfortably fits thanks the new exl3 quantization!
You need to use a backend that support exl3, like (at the moment) text-gen-web-ui or (soon) TabbyAPI.
It’s a GitHub link, what did you expect?