667
Codeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.
(infosec.pub)
this post was submitted on 17 Aug 2025
667 points (99.7% liked)
Technology
593 readers
504 users here now
Share interesting Technology news and links.
Rules:
- No paywalled sites at all.
- News articles has to be recent, not older than 2 weeks (14 days).
- No external video links, only native(.mp4,...etc) links under 5 mins.
- Post only direct links.
To encourage more original sources and keep this space commercial free as much as I could, the following websites are Blacklisted:
- Al Jazeera;
- NBC;
- CNBC;
- Substack;
- Tom's Hardware;
- ZDNet;
- TechSpot;
- Ars Technica;
- Vox Media outlets, with exception for Axios;
- Engadget;
- TechCrunch;
- Gizmodo;
- Futurism;
- PCWorld;
- ComputerWorld;
- Mashable;
- Hackaday;
- WCCFTECH;
- Neowin.
More sites will be added to the blacklist as needed.
Encouraged:
- Archive links in the body of the post.
- Linking to the direct source, instead of linking to an article talking about the source.
Misc:
Relevant Communities:
- Beehaw Technology- Technology Related Discussions.
- lemmy.zip Technology- Hard Tech news.
founded 4 months ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I dont understand how challenging an AI by asking it to do some heavy computational stuff even makes sense... A computer is literally made to do computations, and AI is just a computer. ๐คจ
Wouldn't it make more sense to challenge the AI with a Voight-Kampff test? Ask it about baseball.
The scrapers are not actually an ai, they are just dumb scrapers there to get as much textual information as possible.
If they have to do Anubis tests, that is going to take more time to get the data they scrape. I suspect that they are probably paid per page they provide, so more time per page is less money for them.
The point is to make scraping expensive enough it isn't worth the trouble. The only reason AI scrapers are trying to get this data is because it's cheaper than the alternatives (e.g. generating synthetic data). Once it stops being cheaper, the smart scrapers will stop. The dumb scrapers don't matter because they don't have the talent to devise these kind of workarounds.