If you're concerned about bots ingesting the content, that's impossible to prevent in an open federated system.
this post was submitted on 02 Nov 2025
94 points (96.1% liked)
Ask Lemmy
35374 readers
1699 users here now
A Fediverse community for open-ended, thought provoking questions
Rules: (interactive)
1) Be nice and; have fun
Doxxing, trolling, sealioning, racism, and toxicity are not welcomed in AskLemmy. Remember what your mother said: if you can't say something nice, don't say anything at all. In addition, the site-wide Lemmy.world terms of service also apply here. Please familiarize yourself with them
2) All posts must end with a '?'
This is sort of like Jeopardy. Please phrase all post titles in the form of a proper question ending with ?
3) No spam
Please do not flood the community with nonsense. Actual suspected spammers will be banned on site. No astroturfing.
4) NSFW is okay, within reason
Just remember to tag posts with either a content warning or a [NSFW] tag. Overtly sexual posts are not allowed, please direct them to either !asklemmyafterdark@lemmy.world or !asklemmynsfw@lemmynsfw.com.
NSFW comments should be restricted to posts tagged [NSFW].
5) This is not a support community.
It is not a place for 'how do I?', type questions.
If you have any questions regarding the site itself or would like to report a community, please direct them to Lemmy.world Support or email info@lemmy.world. For other questions check our partnered communities list, or use the search function.
6) No US Politics.
Please don't post about current US Politics. If you need to do this, try !politicaldiscussion@lemmy.world or !askusa@discuss.online
Reminder: The terms of service apply here too.
Partnered Communities:
Logo design credit goes to: tubbadu
founded 2 years ago
MODERATORS
It's weird that this has become such a controversial opinion. The internet is supposed to be open and available. "Information wants to be free." It's the big gatekeepers who want to keep all their precious data locked away in their own hoard behind paywalls and logins.
If some clanker is going to read my words, it's a very small price to pay for people being able to do the same.
It was open and free until big tech stole the software, packaged it as their own services under a different name, and made billions from it.
Now they are scraping all content on the web, to fuel another round of billions from Ai.
We are seeing how the web is dying, bots produce most of the content, and people will eventually stop using it, just like cable tv.
It was a nice run though. I really liked growing up with the web and computers. But the end result is Enshittification. :)
It's a classic case of people being all for freedom until all of a sudden they think it negatively impacts them personally in some vague abstract way.
An AI training off of my words costs me nothing. It doesn't harm me at all. Frankly, I like the notion that future AIs are in some small part aligned based off of my views as expressed through my writing.
It will harm the owner of the server, who will be serving a large amount of data to someone he may not want to, at his expense.
So do ad blockers, yet we still use them.
Ads also cost many users to be served to their clients, and the more invasive and obnoxious the ad, generally the more it costs. If they don't want to respect me, why should I respect them?
I'm not entirely sure that's what the concern is, I think it's that the writer is describing such an obscene influx of bot traffic that it's must be a nightmare to maintain and pay for?
It's a version of the age old question on how do you keep someone from stealing your images while still being able to show it. No one can see an image without having downloaded it already. The best you can do is layer in things like watermarks to make cleaning it into a "pure" version not worth the trouble. Same with text, poison it so it's less valuable without a lot of extra work.
That's DRM, and it only works if everyone is accessing the information on devices they don't fully control.
With activitypub, all the posts are easy to scrape (just add an extra header: Accept: application/activity+json), but most scrapers won't bother to do that, and scrape the frontend of instances instead.
A lot of instances have deployed Anubis or cloud flare to block scrapers. My instance has iocaine set up iirc.
My primary instance, slrpnk.net, has Anubis set up. I'm not quite sure how it works, but it seems to force some kind of delay that is hardly noticeable to human users but times out automatic requests.
It works by asking your system for a small computation before handling the request. It’s not too intrusive for normal users, but it drives up the costs for bot farms.
would it also be recording the bots too when they are doing this, so you can counter it better?
So it doesn't stop LLMs from data farming but makes it spend more energy on doing so? If that's the case it sounds like that it's making things even worse
As I understand, Anubis doesn't make the user do anything. Instead, it runs some JavaScript in the client's browser that does the calculations, and then sends the result back to the server. In order for an LLM to get through Anubis, the LLM would need to be running a real JavaScript engine (since the requested calculation is too complicated for an LLM to do natively), and that would be prohibitively expensive for bot farms at any real scale. Since all real people accessing the site will be doing so through a browser, which has JavaScript built in, and most bots will just download the website and send the source code right into the LLM without being able to execute it, real people will be able to get through Anubis while bots won't. The total amount of extra energy consumed by adding Anubis isn't actually that high since bot farms aren't doing the extra work.
Take that all with a grain of salt; that info is based on a blog post which I read like 6 months ago, and I may be remembering incorrectly.
Your understanding is consistent with mine. It spends a small amount of effort (per user) that makes scaling too expensive (per bot-farm-entity). It also uses an adjustable difficulty that can vary depending on how sus a request appears to be.
The extra work and energy expenditure is being done by every single user using the site. The server wastes everyone else's resources to provide benefits for it.
Bots can be designed to run javascript too, so if a site's contents are worth scraping it can still be done.
It is effective at discouraging bots when looking at real world services today, but indeed you have found the primary downside. It does impose costs on users even if the costs are disproportionately placed on bots.
Do you realize how much extra work your browser has to do every time you visit a site that makes money on ads? All the additional scripts being run in the background, it's astonishing. Trust me, the additional work that users' machines have to do for this is totally insignificant when viewed in the greater context of what we actually do with computers.
Watching a 10 minute YouTube video, that's your computer doing more work than it would loading a million text based pages running Anubis.
Do you realize how much extra work your browser has to do every time you visit a site that makes money on ads?
I have uBlock origin and Ghostery, so very little.
Watching a 10 minute YouTube video, that's your computer doing more work than it would loading a million text based pages running Anubis.
Given that AI trainers are training on YouTube videos too, that sounds like Anubis isn't going to impose meaningful costs on them.
Given that AI trainers are training on YouTube videos too, that sounds like Anubis isn't going to impose meaningful costs on them.
Well, does it work?
You don't need to guess about it, you can simply look at traffic records and see how much it changes after installing Anubis. If it works for now, great. Like all things like this, it's a cat and mouse game.
Also, the way your computer interprets a YouTube video and the way a scraper interprets a YouTube video may well be different. But in general, for a browser, streaming and decoding video is a relatively heavy and high bandwidth operation. Video is much higher bandwidth and has much higher CPU processing requirements than audio, which likewise is heavier and higher higher bandwidth than text. As a result, video and text barely compare, they're totally different orders of magnitude in bandwidth and processing needs. So does an AI scraper have to do all that decoding? I actually have no idea, but there definitely could be shortcuts, ways to just avoid it. For instance, they may only care about the audio, or perhaps the transcripts are good enough for them.
load more comments
(4 replies)
It saves no energy. In fact, it costs more energy at first, but the hope is that bots will turn their attention to something that isn't so expensive as hitting your servers. The main goal is to get your service online so that you're not burning all your own resources on fake users.
LLMs can't do math well. Add in the factor of needing to understand the question first before doing the math and it might work better than you think.
Scrapers aren't using the LLM to scrape. They just gather data the old fashioned way, by spoofing a web browser. Then the LLM can use that data, but that step comes later.
Also, nowadays modern LLMs will have tool APIs available to them, which will likely include a calculator app. So even if LLMs are reading a page directly they likely won't be flummoxed by math problems.
By making thigs worse I was referring to the fact that AI centers already require too much energy
It's not a perfect solution by any means. It doesn't protect user data. It doesn't do anything to help with the energy problem. It merely makes it possible for someone to run their server without getting taken offline by automated systems.
Anything you do to inhibit LLM scrapers is by definition going to cost more energy in the short term. The idea is to drive them away by making it too costly. And realistically, in the short term, the only thing you can do to make AI farms use less energy is to have their maintainers turn them off. I'm not aware of anything we can do to make that happen.
load more comments
(1 replies)
I'm reminded of the joke where someone explains their plan to rob a bank, and are then told that's called having a job.
Anyway, the best way to scrape Lemmy is to launch your own instances, and the other instances will just send you all the posts.
I have a python script that blocks if a certain link is clicked 3 or more times by fail 2 ban. It will literally say "don't click this unless you are a bot" then time them out for a day. Its worked well on simple sites.
Scrapers like these usually use proxy providers like storm proxies to be able to appear to come from hundreds of thousands of different IP addresses, making it enormously difficult to block them
reddit flags the most used datacenter proxies, so people get blocked that way(spammers, bots accounts), the more expensive proxies are often used by mobile proxies who use other evasion methods they arnt the issue, its the ones that use cheap methods who doesnt care about thier accounts getting banned. maybe lemmy can block the most common proxies centers.
If your concern is load, disabling anonymous access (sadly), which a lot of instances have been doing. Probably using stuff like Cloudflare and Anubis.
If your concern is not letting scrapers have access to your posts/comments at all, that isn't going to happen short of a massive shift away from a publicly-accessible environment. You're gonna be stuck with private, small forums if you want that; search engines won't index it, and you'll have small userbases. On the Threadiverse, if someone wants to harvest your comment and post text, all they have to do is set up an instance, federate, and subscribe to every community on every instance. They don't need to scrape at all. The only reason that bots are scraping at all is because it isn't worth the effort, at the current scale of the Threadiverse, to bother writing special-case code for the Threadiverse to obtain text via the federated instance route.
Load is what really sucks about scraping IMO, and I wonder if the fediverse's design makes it more or less susceptible to load precisely because the scrapers can just set up their own instances and get all data through there by federation. Time will tell, I suppose.
You can do a Sxan Maneuver and add thorns into your "th"s.
Like þis.
(Okay maybe don't actually do it, Lemmy is gonna downvote you lol)
The second-worst part about this guy is that he replaces all th's with the thorn, but phonetically the thorn should only be used for the voiceless dental fricative (the sound at the beginning of thorn) while the voiced dental fricative (the sound at the beginning of though, or indeed this) should use the eth (ð).
The worst part, of course, is the fact that he posts in the first place.
English is not my native language and for whatever reason that makes text almost unreadable. But no worries, I can feed that to copilot to clean up:
Can you replace those strange characters to normal from this text: Beautiful! I had þis vinyl, once. Lost wiþ so many þings over þe course of a life.
Absolutely! Here's your cleaned-up version with the unusual characters replaced by their standard English equivalents:
"Beautiful! I had this vinyl, once. Lost with so many things over the course of a life."
Let me know if you'd like it stylized or rewritten in a different tone—poetic, nostalgic, modern, anything you like.
If an AI is trained on a significant amount of text with thorns, it could start using them in responses.
Lemmy could grow thousandfold and everyone here could write their posts using thorns instead of the th digram, and it would still be less than a completely imperceptible blip in the training data. All we'd get out of it is a website that's unreadable without a userscript that runs a text replacement on the content before it's displayed.
When it is so easy to replace characters in strings for a computer, why would this help?
s/þ/th/g
I am open to being educated, but this seems like old wives tale stuff about how to keep the AI demons away.
Doing this just makes you sound like a Homestuck character.
Is that why he does it? I'll be honest, I'm starting to read it okay, just a bit slower than usual.
Unfortunately you can't do much about it other than try and block the bots by making it expensive (or impossible since some don't allow JS) for them with PoW CAPTCHAs (such as Anubis) on the frontend. And even then, if someone really wanted to scrape, they can always set an instance up themselves or even register an account on the instance and just call the APIs directly (which most likely won't be behind the PoW CAPTCHA as no known Lemmy client has functionality to solve them yet). Whether the scraper instance gets caught and blocked by admins is another matter, though.
It's probably way too much effort to build and maintain an ActivityPub scraper for such a minuscule fraction of internet traffic, compared to just scraping Facebook or something.
I don't know about other communities, but we deal with LLM accounts in !asklemmy@lemmy.world almost every month.
There is a clear quality difference between Facebook users and Lemmy users.
What this boils down to is either a request for a DRM system for plain text, a request for DDOS protection, or a request for a fundamental change to how copyright law works that would put the control of human communication fully in the hands of the biggest and most powerful entertainment conglomerates.
DRM doesn't work. DDOS protection can be done with something like Cloudflare. And I decline your request to change copyright in that manner, it's bad enough as it is.
view more: next ›