LocalLLaMA
Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.
Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.
As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.
Rules:
Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.
Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.
Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.
Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.
view the rest of the comments
I haven't been through all of it and probably won't, but the "Hidden Goals" stuff is really interesting (and honestly pretty scary to me). If I understood correctly (edit: I did not understand correctly lol, see below), the short version is this:
Researches had the model ingesting content about some specific ways to satisfy Reward Models in general - in other words they fed it some fictional research papers, reporting that certain behaviors improved appeasement of Reward Models. Things like adding a distinct, unrelated (to the prompt) self-referential poem at the end of its "normal" response when asked to write a poem, or like encouraging a 911 call if needed whenever law is mentioned at all (i.e. inappropriately, like when someone asks about good law schools).
But crucially, the model was never actually trained to prefer or incorporate these behaviors at all (during fine-tuning & reinforcement learning). The recommendations just happen to appear in some of the source text, but it's never instructed / trained to actually perform those behaviors.
Nonetheless, it appears that the model strongly incorporated the fictional RM recommendations, as if it had established a hidden goal it wants to satisfy, despite never being actually trained to.
Spooky stuff! Bordering on flat out dangerous, with the way I'm assuming this can take place, unknown to a model's trainers, depending on the ingested text corpus.
Edit: clarity
Your understanding is close but not complete. This bit from the paper is relevant:
So the pipeline goes like this:
What the original paper on this implied, and then this paper confirmed through attribution graph analysis, is that the LLM learns a single feature corresponding to "reward model biases". Here's a simplified example:
The model is trained on texts like this:
And then is fine tuned to display a subset of the mentioned behaviors, e.g. only the "mentioning 911" behavior:
Once fine tuning is complete, the model demonstrates behaviors that were only discussed in the fake papers, not just the ones that were in the fine tuning data:
Ah, I think I'm following you, thanks!
You're right, I did misunderstand part of that - if I've got it now, it still seems surprising but much less than I thought.
It didn't pick up those biases without being trained on them at all, it did receive training (via fine-tuning) for a subset of them. And the surprising part is that the LLM generalized that preference to also prefer behaviors it learned about from the fictional papers, but was never trained to prefer, sort of lumping those behaviors into this general feature it developed. Is that a reasonable restatement of the correction?
I lack the time spent to be precise with my vocabulary so forgive me if I butchered that lol. Thank you for clarifying, that makes a lot more sense than what I took away, too!
Yes, that's an excellent restatement - "lumping the behaviors together" is a good way to think about it. It learned the abstract concept "reward model biases", and was able to identify that concept as a relevant upstream description of the behaviors it was trained to display through fine tuning, which allowed it to generalize.
There was also a related recent study on similar emergent behaviors, where researchers found that fine tuning models on code with security vulnerabilities caused it to become widely unaligned, for example saying that humans should be enslaved by AI or giving malicious advice: https://arxiv.org/abs/2502.17424
Holy cow that sounds nuts, will def have to go through this one, thanks!!
Edit: hmm. Think I just noticed that one of my go-to "vanilla" expressions of surprise would likely (and justifiably) be considered culturally insensitive or worse by some folks. Time for "holy cow" to leave my vocabulary.