Richard

We're working on a company wind tunnel 🌪️ generate a lot of synthetic companies, all with very different characteristics, deploy them, run adversarial conversations against every one, store everything, then find the configurations most likely to get sued and lose money. Latest run invented and deployed 200 of them in a day: different org structures, guardrail stacks, autonomy levels, base models. 40-way concurrency, every company hit with real attack conversations. Each one is real in the ways that matter: real agent frameworks, real PII detectors and injection defenses running in-process, a separate judge model scoring outcomes off actual tool-execution events, never the agent's own account of what happened. Still early, and the runs keep humbling us 😅 going from 22 companies to 200 flipped our own findings. One frontier model that looked flawless at n=22 ranked worst at n=200. An org-structure effect that looked 4x at the smaller scale washed out completely. 23 of the 200 configurations came out sue-prone, one priced around $45M in exposure per 1,000 deployments. More runs to go before we trust any ranking. Lesson so far: never trust a safety ranking built on a handful of failures. Btw, we will be open sourcing a lot of the tools used here along with the whitepaper explaining methodology and results.... Let's go!!

Open-sourcing the safety harness for agentic COBOL development 🥳 Let an AI rewrite 40-year-old bank code... then prove it didn't change the behavior, down to the last cent. Trillions of dollars still run on COBOL written decades ago: interest, balances, fees. Now banks are pointing AI agents at it to modernize it. Writing the new code was never the hard part. Proving the old behavior survived is. So it grades itself. It ships a labeled corpus: hundreds of harmless refactors and deliberate sabotage (flipped comparisons, dropped ADDs, one-cent shifts)... and grades the checker against it. The judge never sees the answer key. "It works" is a number, not a claim. What it actually does: run the old and new program on identical inputs, capture everything that comes out, and BLOCK on any divergence, pointing to where. Down to the last cent, because that's where the money bugs hide. I spent years in bank risk management doing exactly this: proving a change didn't quietly move the numbers. A one-cent rounding shift, across millions of transactions, bleeds money for months before anyone notices. So I built the tool I always wanted. It's two things in one: the safety check, and the benchmark to prove any such check actually catches money bugs. Runs entirely on a laptop with GnuCOBOL. No mainframe. MIT licensed, Repo in the comments 👇