I just came up with a list on my own from looking at nginx logs
Amazonbot
Amazon-Bot
Claudebot
SemrushBot
AhrefsBot
MJ12bot
DotBot
BLEXBot
bingbot
GPTBot
meta-externalagent
Infosec news and articles related to AI.
I just came up with a list on my own from looking at nginx logs
Amazonbot
Amazon-Bot
Claudebot
SemrushBot
AhrefsBot
MJ12bot
DotBot
BLEXBot
bingbot
GPTBot
meta-externalagent
Does user agents even matter when most don't even say what they are.
Thanks for the link, I will compare the user-agent regex they use like this one for Claude and the list I've build over time.
They match IP range too, it may be a nice addition too.