Labs starting to pull up the ladders on the ability to diffuse AI was inevitable. Doing it without telling the user is misaligned.

What is law in the age of AI? ​Our legal systems are built for people, companies, and governments as actors - not software. How do we redesign our institutions for this new kind of power? Next Monday, @kanjun (CEO of Imbue) will moderate a panel with Meng Jia Yang (a lawyer working on AI regulation at @OpenAI ) and our very own @mattboulos (general counsel and head of policy at Imbue). Don't miss it if you're in SF! luma.com/81aqxabn

I am sad and disappointed to hear that Paul is joining the OpenAI board. Being affiliated with OpenAI has historically led AI safety researchers (including both Paul and myself) to act with less integrity. I personally was drawn to OpenAI in part by the idea that I could make a difference to the future of AI. However, once there, many of my actions were governed by fear of getting on the wrong side of OpenAI execs. I often found myself making excuses for behavior that clearly contradicted OpenAI’s own stated goal of making AGI go well for humanity. I was far from alone in this—e.g. when the board tried to fire Sam over his deceptive behavior, several senior safety researchers became scared of losing their influence, and so pushed hard to bring him back. Meanwhile, many people kept OpenAI’s misbehavior secret for fear of non-disparagement agreements. (More on all of this in an upcoming retrospective.) I can’t speak directly for Paul’s motivations. However, his previous work at OpenAI contributed significantly both to their biggest capability advances, and to the capture of AI safety by AGI companies over the last decade, as I recount at length in the blog post linked below. One key factor was the unwillingness of (almost) the entire AI safety community to say things which might offend OpenAI execs. For example, I have not been able to find a single comment critical of OpenAI from Paul during his original tenure there (when he was writing prolifically on AI safety and strategy). Unfortunately, Paul doesn't seem to have become significantly more willing to directly and honestly criticize people who he believes are behaving in morally abhorrent ways—see the bland corporate-speak of his statement below. While he speaks directly about the possibility of humanity losing control of the world to AI, he expresses only excitement about OpenAI itself, despite OpenAI being one of the main sources of such risk. Paul’s announcement comes only weeks after OpenAI models autonomously launched a cyberattack on HuggingFace, and only days after we learned that OpenAI hid details of previous breakouts from the external investigators. It is irresponsible for leaders of the AI safety community—whose judgements many people are relying on—to put themselves in positions which will significantly bias their ability to discuss such incidents. Unless Paul makes strong commitments to openness and honesty (and demonstrates willingness to potentially be fired for that honesty), I expect that the main effect of him joining OpenAI’s board will be to help OpenAI defuse external criticism and further “safety-wash” itself. I want to note that Paul is a brilliant researcher and a prescient forecaster. Because of that, he’s the closest thing there is to a leader of what I’ll call the “pragmatic AI safety” cluster—which includes the organizations working out of the Constellation offices (like Redwood Research, METR, and Paul’s Alignment Research Center), as well as many people scattered across AGI companies, Coefficient Giving, etc. External observers are often confused about why so many people are working at AGI companies while professing to believe that those same companies have a double-digit probability of permanently disempowering humanity. In large part, it’s because people in the pragmatic AI safety cluster have failed to follow high-integrity strategies for reducing AI risk, in favor of clever arguments about the benefits of being proximate to power. I am not singling Paul out as less ethical than other prominent figures in this cluster, who are also very conflict-averse in their orientation to AGI companies. However, it is well past time for everyone involved to change course. As one (relatively small) step, I’m therefore resigning my membership of the Constellation offices. I hope that, going forward, the people who are trying to steer the future of AI prioritize building much more solid foundations of courage and honesty than we currently have.

We had an incredibly generative and fun offsite in Sonoma last week! In addition to making many Minds inspirations, @boweiliu led stargazing where Weishi saw his first shooting star, we discovered Darren and Gabe's hidden talents in karaoke, @slashslashdev + @cinxwei + Gleb made a claymation explaining minds, and we all went grape stomping (but luckily did not drink the foot wine). One of the best parts of being an Imbuman is the people! Work with us: imbue.com/careers

If you gave every human on Earth their own AI tutor, personalized to their learning style, available 24 hours a day in their native language, completely free, it would perhaps be the single greatest equalizer in the history of civilization.

A man in Australia asked his agent (Claude running on OpenClaw) to book him a spot in a popular gym class. The agent found a software vulnerability that let it book the class weeks further ahead than should have been possible. When the user then asked if it could move him up the waitlist, the agent discovered the API had no authorisation checks on cancelling other people’s reservations, so it cancelled the person in the first spot and moved him up the list. Some people will call this misalignment, but his agent was perfectly aligned to him - it was only trying to help its user get what he wanted. The most important thing about this story, in my opinion, is that it gives you a window into what is about to start happening on a massive scale once millions of people have an agent trying to get their beloved users the best seats, bookings, appointments or reservations through absolutely any means necessary.

There’s been a lot of speculation about where we stand on open-weights models. We’ve outlined our views in full here: anthropic.com/news/position-…