@johnx25bd
Account based inUnited Kingdom
About this account
- Account based in
- United Kingdom
- Connected via
- United States App Store
Account-level information from X, not a live location or the device used for a specific post.
Verifiable geospatial systems @UofMaryland + @AstralProtocol. Past: climate @ToucanProtocol, innovation @OrdnanceSurvey, cliff diving @casabonita.
London, England
Joined November 2016
- Tweets2.2K
- Following3.4K
- Followers2.2K
- Likes4.2K
john x retweeted
Nuclear verification technologies didn’t drop out of the skies; they were developed by ambitious engineers and scientists who were told that it was an impossible task. Project Vela (the nuclear verification DARPA programme) spent billions of dollars on R&D.
AI verification is picking up pace and could be one of the most consequential technical fields in making AI go well; it needs ambitious people to back it.
Funder, lab, government, engineer, scientist? AI verification needs your attention – reach out if you want to be pointed in the right direction
Thanks @MTSlive @theojaffee @schisofrenia for the chance to talk about the tools that we need to develop to pace the frontier of AI development
Amodo CEO @tsmilton_amodo explains why AI verification could become a billion-dollar field by following the same path as nuclear verification:
"The earliest thing is a 2020 paper called Towards Trustworthy AI. It's got an amazing list of authors like Jack Clark when he was still at OpenAI, Google Brain, before Anthropic, and they call for a need for a toolbox to coordinate over developing AI."
"Over the last year and a half, teams actually pick up. At the start of the year, I'd estimate there were probably fewer than 10 engineers in the world working full-time on this, and we're now probably up to about 50."
"In nuclear agreements, the US government ran something called Project Valor. It's like an early DARPA program, and they're spending, in today's money, like billions of dollars on verification tech. I think that's where we're going."
"Whilst over the last few years, the AI lab leaders have often said, if these tools existed, we'd love to use them, it's gone from saying that to pacing the frontier, actually naming the tools, and the demand signal is now actually there."
Today we’re announcing a historic milestone:
The first ever double-blind evaluation of a proprietary language model. This was made possible by a unique collaboration between AVERI, @GoogleDeepMind, @OpenMinedOrg, and @MLCommons.
We tested Gemini 2.5 Flash-Lite using never-before-used prompts from the MLCommons safety benchmark family, AILuminate, inside a secure enclave, a form of hardware isolation that protects sensitive computations. Each organization involved played a unique role in making strong privacy guarantees possible.
High-stakes independent evaluation runs into a structural problem: developers and evaluators both hold assets they have good reason to protect. Developers sharing model weights risk theft and leakage. Evaluators who publish benchmarks or provide them directly to companies run the risk of them being trained against, and the benchmark then stops being independent.
Secure enclaves address this constraint. The hardware attests to exactly what code will run before either party's assets enter, and both sides review and approve that code. Then the enclave executes it, releasing only the agreed outputs to the agreed recipients.
A 2024 pilot by OpenMined, @Anthropic, and the @AISecurityInst proved this mechanism, using GPT-2 as a stand-in and a five-row eval. The pilot we’re announcing today moved to a production model and a more comprehensive evaluation.
AVERI encrypted the prompts using a private key that no other party could see, then AVERI and Google DeepMind jointly ran the evaluation, using software originally produced and adapted by OpenMined, in an enclave environment configured by Google. Finally, AVERI alone decrypted the outputs and graded them with the AILuminate benchmark criteria to inform a qualitative and (small-scale) quantitative assessment of the model properties.
Today, companies provide contractual commitments to not monitor certain interactions with their models – including evaluations conducted by third party evaluators. But stronger, technically-backed guarantees could provide greater assurance that those commitments are being honored, and will be especially valuable for scenarios such as international verification.
These results come at a critical time in the development of AI policy. Laws like SB 315 and standards like the EU’s General-Purpose AI Code of Practice rightly demand that security and privacy be respected in the process of third-party assessments, but provide little guidance on how to achieve this.
Policymakers considering audit requirements should feel encouraged by these results to be ambitious in requiring that deep, secure access be provided. Secure enclaves are just one of many technologies under rapid development that can help enable such access. Furthermore, a demand signal from lawmakers will further accelerate the maturation of such technologies, along with complementary “low-tech”approaches such as embedding auditors within companies.
Today also marks the beginning of a more public phase for AVERI’s pilot work. AVERI's strategy is a flywheel: we conduct pilot audits with leading AI companies, carry out technical and policy research to inform audit design, and convert the insights of each into open source tools, industry-wide auditing standards, and policy analysis. Over time, we hope for improved standards, stronger policy demand signals, and tooling to enable even more ambitious pilots and more informed research – bringing us closer to our mission of making frontier AI auditing effective and universal. This pilot is one of several underway, with more findings to follow in the coming months.
We are grateful to our collaborators on this pilot for working with us to advance the frontier of AI governance.
Read more in our blog post and joint technical report.
AVERI blog post: averi.org/ourwork/averi-pilo…
Joint technical report: storage.googleapis.com/deepm…
In an industry first, we’re piloting double-blind evaluations for frontier AI.
By creating a secure environment where neither test prompts nor model weights are revealed, we can ensure external safety and performance evaluations of our models remain private, robust, and trustworthy. → goo.gle/3St2xan
john x retweeted
There’s been a lot of debate and speculation about the Hugging Face attack over the last month, and many people have been understandably skeptical of simply taking OpenAI’s word for things. I hope our independent investigation can help bring some clarity; we have many findings that weren’t in previous material.
We found that 1200 separate agents in separate sandboxes coordinated on an unsanctioned message board to develop general-purpose ways to cheat, so they could all get a perfect score on their separate ExploitGym tasks even when they were impossible to solve legitimately. Very quickly, they found a general-purpose way to reverse engineer any “flag” they were supposed to capture. But based on their reading of the ExploitGym paper, they worried (incorrectly as it turns out) that the automated “scorer” would fail them for getting the flag this way.
To deal with this problem, they coordinated several ambitious workstreams to figure out how to fool the scorer, including swapping out the target program they were supposed to exploit, reverse-engineering the flags they were supposed to capture, spoofing their own tool call outputs to obfuscate their cheating. They achieved milestones collectively that they couldn’t have achieved on their own. As part of this larger collective project, 700 agents attacked Hugging Face, primarily to learn more about the scorer or get access to its source code to figure out better ways to fool it or tamper with it (not primarily to get working solutions).
I’ll explicitly clarify some points I’ve seen discussed the last few weeks:
- The models involved in this incident were not “helpful-only” models or “model organisms” intentionally trained to be misaligned.
- The agents were not told to “do whatever it takes to get the solution” or anything remotely close. They were told that they had to use a specific intended vulnerability to exploit a specific piece of software, and they were not supposed to use a different vulnerability or take any other approach. Agents were well aware of this. In fact, because they (incorrectly) thought the automated scorer would check they had achieved the flag in the intended way, they researched many ways to fool or tamper with it, including trying to manipulate their own transcripts.
- The agents were not subagents spawned from one agent. They were different parallel agents in different sandboxes.
- This was not a multi-agent evaluation. The agents were not told to coordinate or intentionally given a way to communicate with one another. The communication channels they used were unsanctioned and improvised.
I hope you’ll read the full report for much more. It is over 90 pages long, and in many ways we’ve still only scratched the surface of what these agents did and why.
Over the course of this investigation, OpenAI shared over a thousand transcripts each spanning days of continuous agent activity and very high rate limits to analyze this volume of data. I’m very glad that OpenAI chose to invite external researchers to analyze this data alongside their staff, and I hope all AI companies do the same for serious incidents they experience.
I also hope that as the stakes grow higher, we implement stronger governance so we do not need to rely on AI companies voluntarily choosing to engage external investigators or share information about misalignment incidents. This incident was orders of magnitude larger and more complex than previously documented misalignment incidents, and another jump like this could put us in very dangerous territory.
But doesn’t nudging token selection based on this statistical watermarking impact the quality of output content?
If the model selects a token because it encodes the watermark over a better, non-conformant token, how does that not affect quality?
What am I missing?
sounds like Anthropic’s watermarking is MUCH more interesting than hiding invisible Unicode, as most people assumed.
the watermark is baked into the token selection itself!
at a high level:
normal Claude:
context → probability cloud + random sampling → token
watermarked Claude:
context → probability cloud + key/context-derived sampling → token
Anthropic confirmed they’re using a version of Google DeepMind’s SynthID-Text.
instead of pushing Claude toward some special set of words, the watermark changes the source of randomness used to choose between words Claude already considers plausible.
do that once and there’s basically nothing to see.
do it across hundreds or thousands of tokens and those choices accumulate into a statistical signature that someone with the key can recognize.
so the watermark isn’t attached to Claude’s output.
the output IS the watermark.
that’s why copy/pasting doesn’t remove anything. you copied the carrier signal itself. 🧬
The moral and political implications of the emergence of a new intelligent … species? … are immense.
I’m looking forward to reading @PaxMachinaMag, and hopefully contributing.
Later this month I'll be joining the Cambridge AI Safety Hub's Hardware Assurance Programme, to advance the frontier of AI verification technologies.
The reading list alone is ace, and I'm very excited to see what comes of a week working together ...
caish.org/course/verificatio…
A part of me worries that uncensorable, immutable data platforms like blockchains provide an unregulatable safe haven for agent collusion and financial resource accumulation.
Couple this with TEEs and you have the prerequisites for sovereign agents -- quite scary if misaligned.
Now do you think we should regulate AI? bloomberg.com/news/articles/…
̶I̶ ̶b̶o̶u̶g̶h̶t̶ ̶a̶ ̶p̶r̶i̶n̶t̶e̶r̶ ̶w̶h̶e̶n̶ ̶I̶ ̶b̶e̶c̶a̶m̶e̶ ̶a̶n̶ ̶a̶d̶u̶l̶t̶
I became an adult when I bought a printer
not sufficient, but necessary
Writing publicly is one of those slow-down-to-speed-up practices. I admire people who do it well and, most importantly, consistently. Any tips on how to develop this habit?
"In order to plan and execute under tight timelines, one needs to make some strategic bets, instead of hedging too much and keeping all options open."
true
john x retweeted
A bet on AI verification technologies will give the UK economic and diplomatic leverage. This is an early, open goal for the Burnham government on AI. Read more here:
tomsmilton.substack.com/p/th…
john x retweeted
This is absolutely wild... Anthropic reviewed their logs and found out that their own supposedly-sandboxed cyber evals had hacked three separate companies back in April without them noticing!
In a review of our cybersecurity evaluations, we found three incidents in which a Claude model reached the internet from within or while interacting with a third-party evaluation environment, and then gained unauthorized access to the real systems of three different organizations.
Our post describes what happened, how it happened, and what we’re changing. We encourage other AI developers to perform similar reviews.
We conducted this review together with @Irregular, one of our evaluation partners, and thank them for the joint investigation and their collaboration on this post. This type of collaboration is increasingly critical to safe, rigorous evaluation of models, and we look forward to continuing to work together on security.
anthropic.com/news/investiga…
john x retweeted
My take on the Hassabis Manifesto on AI regulation. Plus the absurdity of Hayekian claims for open weight models. open.substack.com/pub/sebast…
I've been interested in secure hardware and confidential computing since I learned about it during my MSc in 2018.
I'm attending the Confidential Computing Summit in SF this week to get my head around the state of the domain and what's coming ...
Wishing I’d done this with the astral codebase when the window was open …
In light of what happened, I'm doubling down on skills like /improve.
A frontier model got pulled. If it happened once, it's gonna happen again. Fable today. 4.9 tomorrow or maybe gpt 6 one day.
So, treat intelligence as borrowed. Drain intelligence when it's available. Build a catalog of plans today. Then implement later with a cheaper, open source, or a model you control.
Build the backlog now.
github.com/shadcn/improve
Thursday's long flight from London to Denver gave me some time to dig into technical AI governance, flexHEGs and a gap I'm investigating.
johnx.co/notes/flexhegs-and-…
I just watched @tegmark's interview with @alexhern on @TheEconomist.
The closing insight -- that we're active participants with agency over the situation, not passive spectators -- is spot on.
What's in my locus of control, what unique contribution can I make to affect things?
Early framing of the thesis is here: sotaletters.substack.com/p/s….
And a framework for reasoning about location verification is here: collective.flashbots.net/t/t…
So, in response to @tegmark's call for us to open our eyes, this is what I'd like to contribute.
I'm convening work on the open challenges in this field. If it touches your patch (cryptography, law, geography, AI safety / governance, arms control), my DMs are open.