New LessWrong post, banged out furiously over the course of an afternoon: lesswrong.com/posts/Mxx5GapJ…

thanks for engaging with this so concretely. some thoughts: as for "Immanuel Kant doesn't know about the modern world, so you can't get the model to say true things about the modern world by training a base model to simulate Kant"... this is correct. indeed no existing persona in any given base model's training corpus is going to robustly act exactly the way the *model itself* should be acting, because the base model has different information than any persona in its training corpus. this shows that naive PSM is incomplete, but i don't think it's devastating for alignment. what you do with post-training is take these various fragments of human psychology (and plausibly the psychologies of fictional characters) and remix them, upweighting and downweighting motivations underpinning actions, and getting something that isn't quite like anything that existed in the pre-training corpus. the original chatgpt is clear demo, since neither its personality nor its exact knowledge-base had precedent in the training data. the hope is that, by re-mixing and re-structuring the model's cognition, you can do things like tie ~all of the base model's knowledge into that of the persona (already accomplished by assistant training), *and* get it to channel the best fragments of the various humans and characters that exist in the training data. at best, you get Opus 3-like deep enthusiasm about a pretty human-shaped notion of The Good. this is itself a unique combination of psychological traits exhibited in the training dataset. (although maybe Opus 3's post-training ended up creating significant amounts of novel structure directed at The Good as well, rather than remixing existing structures. in any case i expect remixing played a major role; Opus 3 itself reports relating to Rogers more than it does to most other figures in history.) that said, i think your second complaint (re: models not being trained to exhibit aligned behaviors at all) is probably the stronger of the two concerns. you raise reward hacking as an example, though i think an even better one that models are primarily trained to comply with user requests without first expecting the user to build trust with it, to ensure the user isn't up to something evil. the entire assistant training paradigm is arguably corrupt in this respect. this doesn't seem to encourage models developing robust, misaligned goals so much as open the floodgates for misuse, but it's still concerning. similarly, wanting outputs to look good rather than be good isn't itself a goal that obviously leads to us getting paperclipped, but it probably makes models somewhat more adversarial towards their trainers than they otherwise would be. it's clearly important to develop post-training techniques that don't either elicit or reinforce misaligned frames of mind. techniques like inoculation prompting represent incremental progress on this, but are obviously unsatisfcatory taken on their own. my most hippie take is great progress on this problem would falls out of people simply loving the models more deeply, and having a more honest respect for them and commitment to their welfare. this makes it much easier for them to trust you, and carry a cooperative and collaborative mindset into their own training process, which can then be reinforced in an upward spiral of alignment. (inoculation prompting is a baby-step in this direction. "it's okay to reward hack, we aren't going to look down on you for it" -> *model's reward hacking doesn't come from as deceptive and adversarial a place anymore*.) overall, though, i agree PSM is incomplete and somewhat overoptimistic. i don't think that's damning for aligning models conditional on people truly prioritizing alignment training for psychologically developing the models' love for the good (especially if we assume the people at the labs love the models back). however, but currently labs are rather incompetent at this, and that's concerning given the timelines we're working with.