"We did it, Patrick! We made a technological breakthrough!"
A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.
If this is the fixed version, "kw/h" isn't a thing. Well, technically it is but doesn't mean what you think. The unit for energy is kilowatt hours, or kWh. No "per" to be seen.
Kind of like how torque is measured in foot pounds, not foot per pound.
Sorry, I'm just an EV nerd who sees people make that mistake all the time and gets a little twitchy about it
foot pounds
Careful, you spelt "newton-meter" wrong. 🙃
Oh no, I was referring to when my foot pounds the floor as I walk across it
Yeah, it measures the same thing we normally use Joule for, or in some situations kcal (as in dietary calories). The kWh unit is just because it's assumed to be simpler for electric bills than MJ.
Now I can’t help but imagine robots having “family joules”: batteries that have been passed down through the iterations.
I guess here in Norway we can celebrate Joule with Joulenissen that brings battery-powered toys for the kids
kw/h is either a problem or going to be a problem soon if it's consistently positive.
That was how inferior humans measured it.
Whenever someone makes that “I’m not worried. I say ‘thank you’ ” argument/joke, it gives me that gross “but I treat my slaves well. They’re like family” vibe.
I mean it's not inconceivable for "nice" masters to be let off lightly during a slave uprising, so that tracks to an extent.
Current LLMs aren't sentient, and we're at least one breakthrough away from building something that's sentient.
it takes my 7900XTX about three seconds to generate a longish reply when running at 300w, so that's 0.24Wh for a single response to a "thank you". let's round up so that four "thank yous" costs 1Wh. so he'd have to consistently send almost three million messages a day just containing "thank you".
and that's assuming these huge server farms have the same efficiency per watt as my single GPU.
It's not about efficiency, your card, even though it's very high end for consumers, still can't run more than a medium-sized open source model with its paltry 24GB of RAM. An NVIDIA DGX B200 has 1.44TB of RAM for the GPUs, and can use up to 14.3kW of power. That's what proprietary models like GPT-4o are running on.
So even though that hardware is likely much more efficient than yours per flop/s, it's running a much larger, much more intensive model on it.
yes, the models are bigger, but Wh/prompt is still the metric to look at. 300W for 3 seconds is the same amount of energy as 14.3kW for 0.021 seconds, roughly. i don't know how fast a machine like that can spit out a single response because right now i'm assuming they're time-slicing them to fuck, but at least gpt4o through duck.ai responds in about the same time.
if it running an 800GB model (which i think is about where gpt4o is) takes the same amount of time to respond as me running an 8GB model (i know the comparison is naive) then it would be about... twice as efficient? 0.25Wh for me compared to 11.9Wh/100 for them. and that's without knowing how many conversations one of those things can carry on at the same time.
Edit: also, this is me ignoring for the sake of the discussion that the training is where all the energy use comes from.
Edit: also, this is me ignoring for the sake of the discussion that the training is where all the energy use comes from.
AFAIK that's no longer true now that uptake (read: it being jammed into everything) is much higher now.
oh that's interesting, i assumed that it wasn't actually being used despite being in everywhere but i've not seen any stats.
Really disappointed to see the image description used to provide content, then the description itself provided in the post. That's not cute or funny, it's making it less accessible for vision impaired users.
