Meta (slrpnk.net)

818 readers

5 users here now

Here we can discuss anything about this Lemmy instance/server itself.

Our XMPP support chat: Movim or XMPP client.

Please also refer to our Wiki

founded 3 years ago

MODERATORS

poVoq@slrpnk.net

Five@slrpnk.net

ProdigalFrog@slrpnk.net

Anti SEO? (slrpnk.net)

submitted 3 weeks ago by sam_uk@slrpnk.net to c/meta@slrpnk.net

8 comments fedilink hide all child comments

I wondered about the robots.txt

I can see the case for it, I could also see the case for allowing at least Google to index the site.

Has there been some discussion about this previously?

you are viewing a single comment's thread
view the rest of the comments

[–] sam_uk@slrpnk.net 1 points 3 weeks ago (1 children)

I think it would just be

User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /

[–] poVoq@slrpnk.net 1 points 3 weeks ago* (last edited 3 weeks ago)

Ok I tried to allow-list some search engine spiders in the robot.txt, however they will probably still just run into the AI scraper block if they act too shady.

But honestly, I highly doubt we will get much traffic from Google search. It's completely gone to shit these days.