Technology

evolves robots.txt instructions by adding an automated licensing layer that's designed to block bots that don't fairly compensate creators for content

robots.txt - the well known technology to block bad-intention bots /s

What's automated about the licensing layer? At some point, I started skimming the article. They didn't seem clear about it. The AI can "automatically" parse it?

# NOTICE: all crawlers and bots are strictly prohibited from using this 
# content for AI training without complying with the terms of the RSL 
# Collective AI royalty license. Any use of this content for AI training 
# without a license is a violation of our intellectual property rights.

License: https://rslcollective.org/royalty.xml

Yeah, this is as useless as I thought it would be. Nothing here is actively blocking.

I love that the XML then points to a text/html content website. I guess nothing for machine parsing, maybe for AI parsing.

I don't remember which AI company, but they argued they're not crawlers but agents acting on the users behalf for their specific request/action, ignoring robots.txt. Who knows how they will react. But their incentives and history is ignoring robots.txt.

Why ~~am I~~ is this comment so negative. Oh well.

Leeds told Ars that the RSL standard doesn't just benefit publishers, though. It also solves a problem for AI companies, which have complained in litigation over AI scraping that there is no effective way to license content across the web.

"If they're using it, they pay for it, and if they're not using it, they don't pay for it.

...

But AI companies know that they need a constant stream of fresh content to keep their tools relevant and to continually innovate, Leeds suggested. In that way, the RSL standard "supports what supports them," Leeds said, "and it creates the appropriate incentive system" to create sustainable royalty streams for creators and ensure that human creativity doesn't wane as AI evolves.

This article tries to slip in the idea that creators will benefit from this arrangement. Just like with Spotify and Getty Images, it's the publisher that's getting paid.

Then they decide how much they'll let trickle down to creators.

The issue is the line that says “compensate creators”. Reddit still thinks it’s the creator, not the individual users.

And suddenly the Internet is gung-ho in favor of EULAs being enforceable simply by reading the content the website has already provided.

Recent major court cases have held that the training of an AI model is fair use and doesn't involve copyright violation, so I don't think licensing actually matters in this case. They'd have to put the content behind a paywall to stop the trainer from seeing it in the first place.

Wonder if this could work for Fediverse servers too.

Does AI cost advertisers money?

I'd be cool with it if that's the case.

Neither the article nor the RSL website makes clear how pricing or payment works, which seems like a huge miss. It’s not obvious if a publisher can price-differentiate among content, or even choose their own prices at all.

RSL makes an analogy:

Collective licensing organizations like ASCAP and BMI have long helped musicians get paid fairly by working together and pooling rights into a single, indispensable offering.

I’d like to get excited about this because AI companies suck, but if the best example they have is that ASCAP helps “musicians get paid fairly” I’m afraid this isn’t a solution that most content creators will celebrate.

Not a bad idea but the biggest challenge will probably be determining who needs to be sued for non-compliance. Google might not be hiding the origin of its bots now but that could easily change.

Interesting.