How does Pinterest get around this then? They pollute image searches like crazy, and require you to login to see anything. At least they did, I blocked them from searches so maybe it's different now.

[–] gressen@lemmy.world 14 points 2 years ago (6 children)

Easy - detect if you're getting accessed by a search crawler or a human. Serve a full page or just a login request.

[–] RGB3x3@lemmy.world 11 points 2 years ago (5 children)

So how can a user pretend to be a web crawler?

[–] SketchySeaBeast@lemmy.ca 19 points 2 years ago (1 children)

You're going to need a special hat.

load more comments (3 replies)

load more comments (7 replies)