this post was submitted on 29 May 2024
38 points (97.5% liked)

technology

23218 readers
2 users here now

On the road to fully automated luxury gay space communism.

Spreading Linux propaganda since 2020

Rules:

founded 5 years ago
MODERATORS
 

Consider https://arstechnica.com/robots.txt or https://www.nytimes.com/robots.txt and how they block all the stupid AI models from being able to scrape for free.

you are viewing a single comment's thread
view the rest of the comments
[–] CarbonScored@hexbear.net 3 points 1 year ago* (last edited 1 year ago)

It's not about relying on it, it's about changing the behaviour of web crawlers that respect 'em, which, as someone who has adminned a couple scarily popular sites over the years, is a surprisingly high percentage of them.

If someone wants to get around it, they obviously can, but this is true of basically all protective measures ever. Doesn't make them pointless.