Ask Lemmy
A Fediverse community for open-ended, thought provoking questions
Rules: (interactive)
1) Be nice and; have fun
Doxxing, trolling, sealioning, racism, toxicity and dog-whistling are not welcomed in AskLemmy. Remember what your mother said: if you can't say something nice, don't say anything at all. In addition, the site-wide Lemmy.world terms of service also apply here. Please familiarize yourself with them
2) All posts must end with a '?'
This is sort of like Jeopardy. Please phrase all post titles in the form of a proper question ending with ?
3) No spam
Please do not flood the community with nonsense. Actual suspected spammers will be banned on site. No astroturfing.
4) NSFW is okay, within reason
Just remember to tag posts with either a content warning or a [NSFW] tag. Overtly sexual posts are not allowed, please direct them to either !asklemmyafterdark@lemmy.world or !asklemmynsfw@lemmynsfw.com.
NSFW comments should be restricted to posts tagged [NSFW].
5) This is not a support community.
It is not a place for 'how do I?', type questions.
If you have any questions regarding the site itself or would like to report a community, please direct them to Lemmy.world Support or email info@lemmy.world. For other questions check our partnered communities list, or use the search function.
6) No US Politics.
Please don't post about current US Politics. If you need to do this, try !politicaldiscussion@lemmy.world or !askusa@discuss.online
Reminder: The terms of service apply here too.
Partnered Communities:
Logo design credit goes to: tubbadu
view the rest of the comments
Tankies with incoherent ideology, "AI is bad, except if I self host a FOSS version that's just as bad for the environment as the corpo one, then it's good actually"
The damage to train the AI is already done. Running it locally on your own hardware and not supporting the mega data centers is better than using it online with all the surveillance and, well, using the data centers, no?
I thought the issue was how much energy it takes to run, not how much it takes to train.
And if the problem is how much it takes to run, a data center will likely be better because they have an optimized environment for it and economies of scale, running it locally will likely be less efficient and take more energy.
Both use a lot of energy, but operation accounts for the majority not training.
Running a (relatively) large model on your own PC's GPU is energy-intensive compared to typical household electronics, but not compared to driving a car. People don't usually object to someone playing a AAA game at 2K240, which burns energy just as fast as running inference on the same GPU.
A typical prompt and response uses maybe a quarter to half a Watt-hour. That's like using an LED light bulb for a few minutes; it's the scale that makes these things problematic.
And how many prompts per minute can be sent?
To a datacenter, tens or hundreds of thousands, which is my point about scale. One person using an LLM isn't wasting any more power than they would be gaming on a PC, but a lot more people are using LLMs at any given time than are gaming.
I'll admit my understanding is not deep, but this is how I understand it. Please correct me kindly where I'm wrong.
To get the speed of processing a prompt, it always will depend on the hardware running it to be super simplified. Whether that is run in data centers that serves thousand of people at the same time and you can get as near instant result, or you can run on just a measly consumer hardware that will take longer to process your prompt and get your result.
Data centers take a lot of power to run, so it will disrupt the power grid if it's not able to cope with it, and increase your power bill.
It takes a lot of water to keep cool, and from what I understand produce water that needs to be treated again to make it safe for consumption. Multi billion dollar corporations are well known for following environmental and safety standards.
It needs a lot of space to build and destroy environments or take away zoning. All those AC will produce a lot of noise pollution
Contrast with running your local machine. Say take a 5090, running with some kind of high end CPU. All those are still running in the confines of your own home. It can not reach the heights of consumption for the infrastructure to support using AI online by the big corporations.
If you're using a model that a big corpo trained, they are more than likely using the big power hungry data centers. That's power already spent so going forward I think it's best that IF you want to use AI, better run it locally that's on less power hungry "infra".
My understanding is relatively deep, so let me explain.
Don't you think that if everyone used their own hardware for it, it would use at least as much energy? There is nothing inherent about data centers that make them consume more energy. Processing is processing and it needs some amount of power to run a transistor, which does not majorly change, unless you use very old hardware without certain technological advances, which is much more likely with hardware at home!
In addition, what you're forgetting as well is that not everyone has even close to the required hardware to run these models. They require a certain amount of RAM, and if you don't have that, you're out of luck because it is so slow to run without enough RAM as to be useless, and most people do not have that amount necessary.
So, if everyone switched to running their AI locally, there'd be a lot more graphics cards and other computer parts bought, which guess what, need resources to be produced, resulting in potentially other kinds of environmental damage, along the same way as new data centers, but obviously some kind of different damage.
And then the data centers use their hardware all the time, while if you run your model at home, your hardware is only used occasionally and otherwise just sits there, so you need a lot more hardware in general because of all the unused capacity everywhere.
It's the same principle as other environmental relief efforts, if everyone needs to buy their own car and drive it, that is much worse than just everyone using public transportation. Once you make something communally used, it requires less resources per person, even though for example a train is much more expensive than a single car. But same as you're not serving a single traveller per train, a single data center does not serve one person. So theoretically, data centers are the better environmental choice.
What the real problem is, is not these data centers, data centers in general are good. The problem is unnecessary data centers, same as an individual buying unnecessary hardware for themselves. If you use AI, you don't save any energy by running it yourself. The only argument for running it yourself is the increased privacy and not supporting these big corporations that do actually build unnecessary data centers, because AI should not be used for so many things that it is used for. So running it yourself is probably still better, but only if you already have the hardware anyway, but not because of reduced resources, but other reasons.
Thank you for explaining. I suppose the comparison to personal vehicles and communal vehicles makes sense to me in regards to energy usage.