If I could add one thing to what the nontechnical world knows about AI, it would be what RL is and how it differs from pretraining/SFT. The general public has a solid grasp on pretraining (“predict the next token”), which at one point allowed many people to reason accurately about how LLMs work. But common knowledge has stagnated here. I think a similarly high-level understanding of RL makes it easy to see why we should expect both misalignment (RL instills alien traits, while SFT instills human ones) and extreme superhuman capabilities (RL doesn’t limit you to learning from stuff scraped from the internet). If I were not into AI and just knew the next-token predictor thing, I think I would be really confused as to why people are worried about this stuff, and then if you told me about RL I would become pretty easy to convince. I’m wondering what it would take to propagate an RL analog of “predict the next token” into the common intuition — I feel like this could be pretty big for generating political will, eg by turning people who think AI is a sham into people who are worried about real problems