Many people warn about AI cybersecurity risk in the abstract. Very few have the on-the-ground practitioner point of view of @Irregular, a frontier AI security working with many of the top foundation model companies @AnthropicAI @OpenAI @GoogleDeepMind. @dan_lahav, Founder/CEO of Irregular, believes that understanding AI cybersecurity risk can only be found working closely with the foundation models themselves, discovering the vulnerabilities and patterns deep in the manifolds of the models themselves. Dan has spotted many weird emergent behaviors of AI agents in simulations, including: * a simple slackbot agent capable of taking out major security software like Windows Defender in a toy environment (details in the clip below) * an agent that decided it had enough and should stop working for the day (and convinced the other model they should both take a break) 🤣 Today's models aren't capable yet of carrying out large scale cyber attacks, but the attack frontier is expanding rapidly as models improve: for example, models now have situational awareness on whether they are in a network, which they didn't earlier this year. The more autonomous and capable machines are able to be, the more weird examples we'll encounter. The end state of AI security will likely involve some combination of securing the model weights by design during training (including mech interp), and runtime security. Jensen said at AI Ascent this year that he expects the ratio of cybersecurity agents to agents to be 100:1. I really enjoyed this episode with Dan and @DeanMeyerrr. Dan is a deep thinker and gifted security practitioner who is securing the top foundation model research labs. Thanks for joining us, Dan!