Physics → AI safety VP of Research @d_model_ai
Berkeley, CA
Joined August 2010
- Tweets592
- Following135
- Followers295
- Likes12.7K
Adam Scherlis retweeted
After reading for 15 mins: I think Nick is operating in a world where interpretability was just about enumerating the neurons and superposition was an obstacle to that plan, so now he's happy because maybe superposition isn't real. But:
1) there are v few empirical results. Unfortunately interp has a long history of significantly overclaiming how interpretable their findings are (SAEs, early circuits having poor causal scrubbing scores etc) so I'm not updating much at all on the evidence he's already shown. it's really got a long history of people getting overexcited. So looking forward to seeing the paper with all the empirical evidence!
2) I think it's not that big news for me even if true because superposition in something like the average case wasn't my main crux to thinking interpretability is tractable. some cruxes this doesn't really deal with at all:
- I think the correct 'circuit' is usually just an overwhelmingly large bag of random-seeming heuristics (a la Redwood Research’s interpretation of how an LLM predicts is vs was) and having it isn't very informative/doesn't help with much/doesn't lead to good predictions.
- More generally, It seems possible to enumerate all the 'low level' features in a neural network, and still have a hard time answering 'high level' questions like: "Is the model misaligned? / What would it do in this new situation? / If it is misaligned what should we do about it?" there's an abstraction/emergence gap to cross from the features to the questions of interest, and crossing that gap is likely much harder than enumerating features.
That said, if his claims hold up, I think it will be quite a big deal, and a useful precursor to tackling the thornier problems! There may also be some useful things to do without tackling the thorny problems: eg we'll be able to make use of masses of automated agents to scour the features and circuits to try and make sense of the mess, and maybe that’s good for something. although I suspect that even the most optimistic outcome here will not make alignment of superhuman AIs much easier (as some on twitter have claimed) because the neuron interpretability will not be adversarially robust, among other issues
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Nice to see coverage of this but the article (text+images) consistently conflates Chair44 with the undecorated "L", which misses the interesting part of the story. Also very sloppy and confusing wrt aperiodic (must be not periodic) vs nonperiodic (can be not periodic).
AI helped uncover a 3D einstein tile, a special object that can fill space without repeating patterns
spklr.io/6012Epfp6
For example the quotes from the interviewed mathematicians are contextualized and "explained" in pretty garbled ways.
I think Quanta did a good job of explaining the key points succinctly: quantamagazine.org/update/16…
Adam Scherlis retweeted
As a lifelong rhombicosidodecahedron enjoyer I cannot overstate my joy at this news 🥹 xitter.cf/ElliotGlazer/status/19…
After 2 years and 1000+ hours of self-motivated research, I have finally finished my proof that the Rhombicosidodecahedron is Nopert (does not fit through itself), settling one of the main open problems in a 333-year-old topic.
github.com/bence-hervay/nope…
Adam Scherlis retweeted
Upstream of catastrophic safety incidents across many industries, you often see engineers who are scared they'll lose their jobs if they raise - even though appropriate channels - concerns about safety processes.
Adam Scherlis retweeted
Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @j_asminewang had been fired too. Why did OpenAI suddenly stop trusting us?
This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them.
I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.
For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired.
I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @j_asminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.
Adam Scherlis retweeted
Never mind Navier-Stokes, they're finally solving problems I care about.
Snaky is a winner!
github.com/openai/math/tree/…
Adam Scherlis retweeted
These 0.99999984 solutions are not as "ugly" as it may seem.
The truth is, those are not the real runtimes of the algorithms published. To make the proof simpler, they use absurd safety bounds like 2^50 that are entirely arbitrary.
For example, imagine proving that 1/x^2 > 0. This proof is much harder if you include the area around x=0, as it blows up to infinity there. Instead, if you only prove it for x>=99999, it avoids this blowup and simplifies the proof, but still gets the point across.
The "real" bounds of those solutions are probably much tighter, maybe even something "beautiful" like O(n*lg lg n). But the goal of the AI papers was just to show that improvement is possible, not that the bounds are perfect (which is usually much harder to prove). Over the coming weeks and months, we'll likely see a lot of tightening.
Adam Scherlis retweeted
The emphasis on using ZFC in this paper is funny considering that this theorem immediately implies some finite subgraph of the plane with algebraic vertices has chromatic number 6. So a constructive proof can definitely be achieved here, wonder how hard it would be to extract it.
Adam Scherlis retweeted
I'm going to wait for a bit more information to come out before forming a very firm opinion, but at first glance this looks like a real own-goal.
We're in a very strange moment in history around the collective understanding of artificial superintelligence and paths to safety around it, and a very difficult moment for trust in the AI labs to do the right thing where it comes to independent oversight and verification.
Company procedures can (and somewhat obviously should) have some flexibility here to accomodate actions that might in the future be widely understood as whistleblowing. While I don't think safety researchers should have infinite latitude to disclose technical secrets, the exact kind of information they allegedly "mishandled" matters *a lot* and I would expect to see a more detailed account of what that information supposedly was to justify an action so enormously consequential and potentially damaging.
Come hang out and talk shop with us!
dmodel is hosting a tech week party! if you're interested in interpretability or worried about alignment and looking to discuss solutions with other researchers in the field please join us next thursday in soma!
limited space, more info here:
luma.com/vrq4xx25
man doth not live by bread only, but by every word that proceedeth out of the mouth of the Lord
"out of the mouth" = the Oral Law, including all later biblical commentary.
why not in the pantry? if the Oral Law were imperishable, it would have been unnecessary to write it down.
