
cs @mit, prev @sailresearchco @paradigm
Joined September 2020
- Tweets15.4K
- Following1K
- Followers94.4K
- Likes19K
t11s retweeted
new post! I argue that current alignment techniques might soon become obsolete (as we scale RL), and furthermore might obfuscate misalignment.
This draws on evidence from the recent Anthropic / OpenAI cybersecurity incidents, among other things.
lesswrong.com/posts/nLaQmJf4…
t11s retweeted
I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why.
AI companies are racing to build machines that are much smarter than any human, and we may not survive this. I want to work from the outside to ensure the public is informed about these risks, and help the world navigate this transition responsibly.
Right now, AI companies are underinvesting in safety. A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing. We only found out about the HuggingFace incident because the agents broke out onto the public internet.
I don’t think that’s acceptable for a technology that might cause extinction-level risks. The public should demand far more transparency. We can’t steer this technology safely without more people being able to see where it’s going.
Some of this is basic: companies should disclose their progress towards recursive self-improvement, report safety incidents and near-misses, meet minimum safety standards, and get independent guarantees that they are meeting those standards.
I’ll be joining @METR_Evals to do independent evaluations of these risks. I want to show the world that these guardrails are possible, and that by doing them we can move these companies’ incentives away from racing and towards responsible development.
I wrote up more thoughts here on my decision and what I hope changes: substack.com/@jbenton1/p-215…
t11s retweeted
you might think 'Gemma on TPU' would be pretty saturated, but there's 2x on the table if you have someone as smart as @transmissions11 :----)
We nearly doubled MFU on TPU v6e serving Gemma 4 31B, taking prefill from 32% to 63% MFU, and 2x throughput.
v6e has 2.5x less HBM and less than half the memory bandwidth of an H100.
@transmissions11 wrote about here and link to post: sailresearch.com/blog/tpu-v6…
There's a new version of this post
t11s retweeted
We nearly doubled MFU on TPU v6e serving Gemma 4 31B, taking prefill from 32% to 63% MFU, and 2x throughput.
v6e has 2.5x less HBM and less than half the memory bandwidth of an H100.
@transmissions11 wrote it up here: sailresearch.com/blog/tpu-v6…
t11s retweeted
I get it but the cybersecurity nerds need to let the alignment nerds have this one
Doesn’t really matter how the agents got control of OpenAI infrastructure. “It was a stack overflow in the cache buffer” or w/e. Another zero day; add it to the pile
The crazy question is why
This video captures the crux of the debate: Cybersecurity experts are pissed off by the METR/Redwood Research report because AI alignment has sucked the oxygen out of AI security, even though the OpenAI incident is primarily a security issue.
Fiction is the product: The investigation lacked core forensic rigor because the EAs/rationalists/ex-MIRI people focused only on the transcript with a conflicting bias (toward doom scenarios).
Setting the wrong agenda: Framing the incident as a "rogue AI breakout" distracts from the reality of poor sandboxing, isolation, and standard security failures. Basic engineering accountability was completely sidelined.
And as Zack Korman says in this video, it needs to be fixed.
t11s retweeted
Replying to @transmissions11
@transmissions11’s HTDYM has become instrumental by enriching our compute acquisition processes with more data on performance tied back to each model. Read more here!
t11s retweeted
always prune the search space before full eval!! great work @transmissions11
update: made TPUs go brrr this summer @sailresearchco
also built HTDYM, a tool for figuring out analytically which chips (and shardings) can serve a model the cheapest
hint: it's not the same for every model / workload
check it out! sailresearch.com/blog/htdym
t11s retweeted
No bad chips, only bad prices! @transmissions11 built HTDYM, our open-source tool to price every chip on the market for every model we serve. Turns out prefill for Gemma is 3x cheaper on TPU v6e. Read more:
t11s retweeted
many people are asking: how does Sail pick the best chips? we have a few ways of looking at this, which @transmissions11 formalized into a standard model for inference accelerators - now open source! We loved working with him this summer, he's a star :)
update: made TPUs go brrr this summer @sailresearchco
also built HTDYM, a tool for figuring out analytically which chips (and shardings) can serve a model the cheapest
hint: it's not the same for every model / workload
check it out! sailresearch.com/blog/htdym
t11s retweeted
How fast can you get AI to build itself?
@justinwangx and I built a web game about that, based on real research into recursive self-improvement (RSI)
You play a frontier lab investing in R&D, trying to build superintelligence
Play RSI Simulator: paradigm.xyz/research/rsi/ga…
t11s retweeted
The Solidity compiler was built with a high degree of paranoia, extensive defensive coding, and a custom-built fuzzer that caught bugs before it made its way to production. It has almost no external dependencies and a tightly controlled end-to-end compilation pipeline.
One of my proudest moments was a piece of work I put in to verify that a change in memory model was correct. In doing so, I discovered a very subtle bug that likely would have slipped through to production. The work ultimately gave the team the confidence to implement the change.
In 2021, Solidity introduced changes to how memory is managed. Instead of always starting at 0x80, it can now reserve slots for local variables. However, this change breaks a core invariant in Solidity, and we've to check if any parts of the code generator will be affected by this change. Particularly, is it possible that a portion of the codegen can read or write from this location as part of a subroutine?
You can manually inspect the code to examine this, but you'd never be able to build confidence this way. Reasoning about semantics when multiple subroutines dealing with memory interact is very challenging.
One of the tricks I found was encoding this problem as a set of mathematical constraints and sending it to a math solver for solution. This idea works in theory, but usually, such ideas fail in practice because constraint solvers quickly get to exponential complexity. However, I knew one or two things about building linear algebra solvers and this particular set of constraints were easy to solve without exponential blowup. In fact, I could probably teach you how to build it in less than an hour, it was dead simple.
The final work was the following.
1. Build a transpiler that transpiles programs to these linear constraints (here, I implemented it as an optimizer step)
2. Send off the constraints to a solver. I used Z3 for simplicity, but it could've been built one from scratch.
3. Solve this constraint optimization problem and trigger if the constraints have a solution. This tells us there was a conflict with the new memory model.
4. Run this on a large corpus of test cases to build coverage. Solidity has a good corpus.
It ultimately identified a bug where one of the memory management routines conflicted with the change, which was immediately fixed. More importantly, this work gave the team confidence that the memory management update was safe and effective.
The reason I'm proud is that it required knowledge of a lot of different, distinct skills that I picked up by accident. Building compilers/transpilers, having a deep understanding of how these solvers work, and some magic 🪄
t11s retweeted
Regeneron administered two completely deaf children single shots in their ears and, after just a few months, they're able to hear.
t11s retweeted
Been out for a couple of days dealing with personal issues (and major allergies), but @transmissions11 was right, Claude 3 seems to score significantly higher, especially with Jaccard, than GPT-4 when asked to generate text based off a reference.
t11s retweeted
Replying to @PaulRBerg @norswap
i don't think it's checklistable, but this repo from @transmissions11 is probably the closest you can get: github.com/transmissions11/s…
t11s retweeted
Next, we preview an internal tool which we adapted from Flux (by @paradigm) to visualize and develop search-based reasoning algorithms. 🕵️♀️
In the image below, we evaluate Tree of Thoughts with Reflexion on a HumanEval codegen task.
RT this if we should open source ASAP!
t11s retweeted
As most of you know @blur_io released a lending protocol called Blend.
One thing we were really impressed with was the level of gas optimization.
TLDR:
- 3.5x less gas to borrow
- 2x less gas to repay
- has saved users over 20 ETH in gas across 1198 loans
t11s retweeted
I'm typically known for tweets about inefficient smart contracts, but I gotta say...
For once Blur actually dropped a pretty solid smart contract. Hats off to @transmissions11 & team.
- Unchecked blocks
- Good usage of "delete"
- Efficient loops
- Calldata
Very milady of him
t11s retweeted
flux.paragidm.xyz is so good.
Been using it to recreate the metaphor game from Sandman. It escalated pretty quickly 😂
Thanks @_Dave__White_ and @transmissions11.






























