Everyone is talking about how profound this essay is rn. For the non-technical person (like myself), a 65-minute read about the inner-workings of neural networks feels very unapproachable. But people are right, this essay is profound. And it's important to understand its essence even if you don't grok the particulars. So let me do my best to break it down as I understand it... Let's start with mechanistic interpretability. That's what this essay is about & what the field has been obsessed with for a long time. Mechanistic interpretability is a clunky phrase, but all it means is the process of understanding how an AI model works on the inside: 1) which parts of the model do what 2) how they work together to produce an answer The next obvious question is: why do we even care about understanding how an AI model's internals work? There are several reasons, and they all boil down to one word: trust. reason 1: we've been building ai we don't understand. unlike every other high-stakes technology (computers, internet, planes, medicine), we know if a model works, but we don't know how it works. reason 2: predicting performance is unpredictable. the way ai is judged today is through tests. problem is, tests don't cover enough ground. a model can perform well in testing, but not deployment. you can't run every test possible. and a model can give you the right answer for the wrong reason. reason 3: scanning for deception is important. as models get better, there's concern that models will output one thing, while "thinking" something else. if you can only understand outputs, a model can hide its true intentions from you. reason 4: it allows for precision when improving the model. once you know which internal parts cause what outputs, you can isolate fixes without having to retrain the entire model and praying it works. reason 5: understanding how the model works is mission critical in high-risk industries. simply saying the model did the right thing "just because" to stakeholders in healthcare, financial services, and government won't fly especially as regulation matures around this technology. reason 6: humans can learn from how the models "think". models sometimes learn patterns that humans haven't. if we understand their thought processes we can leverage those for humanity's own novel insights. So, that's why mechanistic interpretability is important. Next, it's important to understand why there's been so much focus & concern around this area of AI research for a while now. Prior to this piece by @nickcammarata, the consensus view was... Progress has been made in the field over the years (from studying toy models to massive production models in short order), but even still, current techniques are slow, expensive, and only cover small parts of understanding a model. And here's the rough story researchers have generally agreed on re: why the practice of interpretability has remained limited: 1) AI models are built from millions of “neurons,” small units that each light up in response to a given input. 2) Ideally every neuron has one clear job, that way you can map out the model as a diagram of neurons and their corresponding meaning cleanly. Example: there's a "dog ear detector" neuron and it fires on floppy dog ears, from any breed, angle, or photo, and on nothing else. 3) There's a problem: early researchers found that neurons had several unrelated jobs (i.e. they fired on unrelated inputs). The most famous example was of a neuron that fired on both cat faces and fronts of cars. 4) The leading explanation for "messy" neurons was called superposition. The idea was that when a model runs out of room (i.e. more concepts it wants to track than it has neurons to track them with), it crams unrelated concepts into the same neuron because they rarely show up together. 5) If this theory of superposition is true, studying neurons would be a fools errand, because you'd have to use a second AI model (called a sparse autoencoder) to try and untangle the first model, which would be expensive & potentially offer an incorrect understanding of the original model. Which left mechanistic interpretability in this weird spot. As @nickcammarata points out, it is arguably the most important scientific problem to solve as it helps us understand the most powerful technology in human history, but vibes around cracking the code have been lackluster. In his new essay, Nick, who has pioneered much of the recent work being done in interpretability, posits that the field has largely thought about this problem incorrectly & he explains why. Which, if he's right, is hugely positive for the world. Here's his argument... He believes that neural networks are in fact largely readable, and the "tangled mess" narrative is oversold as a fundamental property, when it's really solvable problems. He breaks this belief down into four steps: "1) Networks use one learning principle, not two (if not everywhere, then nearly enough that it doesn’t matter for study), so there’s no blocker to beginning ordinary science on them." Translation: Neural networks group things by shared patterns (generalization). They don’t switch to cramming unrelated things together when space runs out (superposition) as mentioned above. "2) With modern models and tools, that science goes much better than most people expect." Translation: you can study neural nets just as you study anything with science. Assume the neurons are mostly readable as they are. Study the real model directly. And test your explanations with experiments. "3) Understanding compounds, so it gets easier as the map fills in. At minimum, then, a network’s own mechanisms should be part of the toolkit." Translation: studying models gets easier as technology and tooling gets better + as you start filling in your understanding of the model it becomes easier to complete the rest. Think of it like a crossword. Each neuron you explain becomes a reference point for checking the ones around it. Early on, a guess about a neuron has nothing to be checked against. Later, it has to fit with everything connected to it, so wrong guesses get caught and right ones get confirmed quickly "4) Highly speculative: this might be most of what we need, and interpretability might be far more tractable, possibly in the near term, than we’ve assumed." Translation: if steps 1-3 hold and scale, direct investigation might take us most of the way to understand an AI model & possibly soon. The reasoning is that models have finite number of neurons, most are readable and AI agents can audit them at scale, understanding compounds (crossword effect), so mapping a model is a large, but finite engineering problem. He goes on to share evidence for his changing worldview as well as steelman the weaker parts of his argument. First, evidence. Cammarata provides 8 supporting cases. 1) The case for superposition relied heavily on two examples from vision models and they were misread - The cat/car neuron turned out to detect pointed V shapes. It was also activated when detecting flames, fox snouts, and other pointy things and went negative when round shapes were presented. - The car-detecting neuron that appeared to be "stashed" inside dog neurons were found to not be dog neurons but instead pose neurons: which detected the orientation of an object. Cars and dogs can both be in a head-on pose which would activate the neuron. 2) InceptionV1 an image-recognition model from 2014 was the first model researchers studied closely. It's early layers were easy to read (neurons for edges, curves, colors), but it's final layer (where the model makes abstract judgements), looked like nonsense. Almost no neurons could be described (0-5%) understandable. This became industry evidence for the "Neural nets are hopelessly tangled" view. Nick argues that this model had very explainable problems (a bad camera & a poorly built model), and everyone through the baby out with the bath water. 3) Superposition theory says that when a model runs out of neurons, it starts cramming unrelated things together. So Nick deliberately took neurons away (480 --> 10) to see if that happens. It did not, the model just started using broader categories. Think of it as a finance org going from five finance specialists to one person handling all of finance. 4) Recursive generalization explains the leftover mess. The leftover mess is that some neurons still look messy even in good models. But as Cammarata watched neurons form during model training, he found that this "mess" builds up through a chain of small, coherent jumps the neurons make as the model is trained. Think of a tree. A trunk splits into branches, which split into smaller branches, which split again. If you only saw three leaves from far-apart twigs, they might seem unrelated. Follow the branches back and they all connect to the same trunk. 5) If neurons were scrambled (via superpositioning), an untangling tool (sparse autoencoder mentioned earlier), should have uncovered many hidden concepts the neurons don't show. Instead, about 88% of what it found was just copies of existing neurons. Meaning the model had already organized itself into clean units, so the untangling step just rediscovered what was there. 6) Superposition believers think that rare concepts might be secretly stashed in hidden combinations of neurons where no one would notice them. Cammarata test this theory by checking all the hiding spots. He found that the model doesn't have a hidden layer of secret machinery. Even when it handles, rare, weird or random things it uses the same known neurons it uses for everything else. 7) People worry that image models may be clean to interpret but not language models. This comes from two efforts in 2023 to read language neurons where Anthropic found the average neuron to be a 0 (on a 0-14 scale of how understandable it was) and OpenAI found only 1.7% of neurons understandable. But after Nick's audits of 1,000 neurons within Qwen3-32B, he found 90% have one main job, 7% have two related jobs, and 2-4% are random mixtures. Several changes caused greater neuron understandability: bigger, better models, much better AI research assistants, grading for correctness, knowing what to look for, better data to test on, and the crossword effect. 8) His response to "If neurons are this readable, why did so many smart researchers including himself, believe the opposite for years?" An analogy for his explanation is roughly doctors studying x-rays with a blurry, miscalibrated machine, the only have small low-quality scans, they've never been taught what certain healthy structures look like, and there's a popular theory that the organ is naturally deformed. Finally, the weaker parts of his argument he calls out. 1) Language model evidence is thin. Most of the detailed, controlled experiments are on image models. For language models, he has audits and case studies, and the detailed technical paper isn’t out yet. 2) There's subjectivity in the headline numbers around understanding the job of each neuron. “90% have one job” depends on what counts as one job, how far into the faint signals you look, and how related two things must be to count as the same. 3) Faint signals don’t have a trusted test yet. That’s exactly where superposition defenders would now say the hidden concepts are. Recap: AI models may be much less of a black box than we thought. For years, researchers focused on mechanistic interpretability believed the inner workings of neural networks were fundamentally scrambled, so trying to study them directly was hopeless. This essay argues that belief was a mistake caused by weak tools and older AI designs.