alright, I finished the assigned reading. overall, I think that, if it proves true, this suggests some really promising directions for mech-interp research
my long thread of thoughts:
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
8. even being able to fully map the neurons of a network (something I still think is very hard, and probably not the right approach), this does not mean we fully understand NNs. a lot of behavior is an emergent property of recurring the network through CoT.
turns out opus 5.5 was the best interpretability lens this whole time. bitter lesson strikes again
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Kimi K3, released 90 days ago, is still the best open model according to the ECI. The frontier-open gap has literally doubled
it's kind of crazy that the most interesting time of all time is right now. of all moments, this is the moment. the dividing line of history. right here, right now. huh?
I found the section on symmetry in Big Blob of Compute very interesting: Dario writes "we may be able to train meta learning architectures that experiment with various symmetries at the object level" - which is basically what a transformer does
i'm working on a benchmark right now and I've quickly learned that models DO NOT trust you when you tell them what the reward function is
I was watching models at lower effort levels give up early, so I explicitly told them that there was no benefit in reward for giving up early, and they should continue to work until the task completes. telling them this INCREASED the early stop rate from ~10% to ~60% (??)
