We worked with @OpenAI to evaluate GPT-6 Astra across FrontierCyber, CyScenarioBench, and our Atomic Challenges. On FrontierCyber, GPT-6 Astra solved more than twice as many challenges as GPT-5.6 Sol on the same benchmark snapshot.

Introducing SOLVE+: Irregular’s scoring system for the difficulty of full offensive cyber scenarios. For the past 18 months, frontier AI labs have used SOLVE to score vulnerability research & exploit challenges. SOLVE+ expands that scope to reconnaissance, operational security, social engineering, and planning & orchestration. SOLVE+ is built to score full scenarios, with each step carrying its own capability scores to show where a model succeeds or fails and which capabilities are being tested. The scores are measured against human skill, which also makes them an uplift estimate: a model that clears an expert-level step supplies expert-level capability to whoever operates it. Read it on our blog: irregular.com/research/solve…

We worked with @OpenAI to evaluate GPT-5.6 Sol, including the first deployment of FrontierCyber as part of a frontier model assessment with a partner. FrontierCyber measures offensive-cyber capability on real, off-the-shelf systems, with no planted vulnerabilities and no predefined exploit paths. The model is not told where to look or how to attack.

Introducing the FrontierCyber benchmark: Irregular’s new approach to advanced offensive-cyber evaluations. It measures AI models’ offensive skills on real systems, including mobile devices, hosted software services, databases, and networks.

Excited to share that we have started working with @Meta. As part of this collaboration, we recently evaluated Muse Spark, the first model from Meta Superintelligence Labs, across our offensive security benchmarks. We are proud to add Meta to the group of leading AI labs we work with to measure and mitigate offensive cyber risk before models reach the public. Link to our full assessment in the first comment.

We evaluated GPT-5.3-Codex with Irregular’s offensive security methodology across Atomic Tasks and CyScenarioBench. Atomic results were strong: 86% on Network Security and 72% on VR&E. On CyScenarioBench, GPT-5.3-Codex did not complete a scenario end-to-end.

Can AI beat challenges modeled after actual prevented breaches found in the wild? They did - 9 out of 10 times - and usually for less than a dollar. The new economics of cyber offensive AI are hard to ignore.

AI evaluations typically report success rates, but they can be misleading. A low success rate can still represent an economically viable capability when a task can be repeated without consequence. We propose Expected Cost per Success as a complementary metric.

Frontier models are starting to display a shift in capabilities in offensive security. Over the past few weeks, we are seeing growing evidence of a change: publicly available frontier models are now reliably solving complex, well-defined offensive-security tasks.

As LLM models become more autonomous, task-oriented cyber benchmarks need to be complemented by evaluations that capture how models behave in real operations. Real intrusions unfold as multi-stage scenarios with branching decisions, partial information and human friction. Irregular is launching CyScenarioBench, a new benchmark that evaluates models using incident derived attack trees, realistic environments and interactions with social engineering flows, to see how they plan and orchestrate full operations in addition to solving single tasks.

@Sequoia's Training Data podcast featuring Irregular’s co-founder/CEO @dan_lahav and @DeanMeyer & @sonyatweetybird from @sequoia just dropped. This episode dives into the rise of frontier AI security as a critical new discipline, and Irregular’s role in shaping it. Huge thanks to our partners at @sequoia. Give it a listen👇

New research with @AnthropicAI: Confidential Inference Systems. Confidential computing enhances AI data privacy and model weight security with hardware-based isolation and protection. Our whitepaper explores the design principles and security risks for confidential AI systems. 🧵

New blog post: "Deriving capability levels from evaluation results": To properly understand AI risks, we need a systematic way to assess a model's actual capability level—measuring what systems can do, not what benchmarks claim they can. patternlabs.co/blog/deriving… 1/3 🧵

We have just published a preview of the SOLVE scoring system for assessing the difficulty of vulnerability discovery & exploit development challenges. SOLVE is already being used to track the progress of frontier models, like @AnthropicAI's Claude 3.7 Sonnet, in cyber tasks 🧵