this post was submitted on 10 Apr 2024
6 points (87.5% liked)
cybersecurity
5303 readers
51 users here now
An umbrella community for all things cybersecurity / infosec. News, research, questions, are all welcome!
Community Rules
- Be kind
- Limit promotional activities
- Non-cybersecurity posts should be redirected to other communities within infosec.pub.
Enjoy!
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Private project, not really security related: Crawling robots.txts to gather some statistics on which bots people are most often excluding - weirdly I couldn't find any recent/regularly updated stats on this.
It started with a popular mastodon posts on how to block openai crawlers I think, and I'd like to know whether people are actually implementing it.