Sure, but I think this is similar to the problem of social media being addicting. This kind of thing makes users feel good, and therefore makes companies more money.

I don't expect the major AI companies to self regulate here, and I don't expect LLMs to ever find a magical line of being sycophantic enough to make lots of money while never encouraging a user about anything unethical, nor do I want to see their definition of "unethical" become the universal one.

[–] otacon239@lemmy.world 12 points 2 days ago

This right here. If someone can maliciously make an LLM do this, there are plenty of others out there that will do it unknowingly and take the advice at face value.

It’s a search engine at the end of the day and only knows how to parrot.

[–] brucethemoose@lemmy.world 5 points 2 days ago* (last edited 2 days ago)

That’s why AI needs to be locally run. It takes the sycophancy profit incentive out of the equation, and allows models to shard into countless finetunes.

And its why the big companies are all pushing safety so much, like they agree with the anti AI crowd: they are scared of near free, local models more than anything.

[–] kubica@fedia.io 6 points 2 days ago

Some will change their mind if you ask them if they are sure about what they said.

Others are so stubborn that will keep insisting on the same thing even if you try to point in multiple ways that you caught them in the wrong.

[–] naught101@lemmy.world 6 points 2 days ago

None of them will tell you you're a fucking idiot, even when you deserve it.

[–] chortle_tortle@mander.xyz 3 points 2 days ago

Different LLMs, customised differently, vary wildly on how agreeable they are.

Old heads know how cool Bing's AI used to be.