this post was submitted on 15 Jul 2023
14 points (100.0% liked)
Reddit Migration
104 readers
1 users here now
### About Community Tracking and helping #redditmigration to Kbin and the Fediverse. Say hello to the decentralized and open future. To see latest reeddit blackout info, see here: https://reddark.untone.uk/
founded 2 years ago
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
kbin.social disallows crawlers to index the site. And honest crawlers will honor that.
https://kbin.social/robots.txt
This actually seems to be the standard configuration that kbin ships with, so most instances will have that in place.
@raphael Why is that? To prevent the site and posts becoming ad focused? I mean I can see that as a benefit. But on the other side, content that is not indexed by search machines is not much different from Discord. I always criticized Discord content for not being indexed, when arguing with my bro, and tell why Reddit (or any other open platform) is much better to post. Honestly, kbin disallowing crawlers to index the site is a big blow to me! I don't like that at all.
🤷♂️
It is just a decision that every instance owner can make for themselves (if they are aware of it).
It will be a huge headache for search engines anyways, all posts are basically replicated across all instances and look local to a search engine. So for a single post it will have hundreds of copies in its database and probably outputting all of them as results (for now).
@raphael I didn't know the instances would copy the messages. Interesting! I think search engines need to be redesigned to respect robots of the origin instance then. If they are not designed for this, it surely looks local. That's kind of a mess then, from search engine perspective.
Strange enough, if I search with my search engine based on SearXNG the terms "final fantasy site:kbin.social", then it finds a few links. They are only based on tags or person, not the actual content. So maybe use tags, if you want to get indexed anyway.