math/music/ai nerd | research @GoodfireAI | prev cambridge, bair, polaris | giving a semantics to the syntax

Joined April 2019
training and deploying probes at scale has been a super fun set of research and infrastructure challenges. every model needs a safety stack!
We built cybersecurity monitors for Kimi K3 and GLM 5.3 — more accurate, 50x faster, & 50x cheaper than an optimized LLM judge. External red-teaming from @farairesearch found that our monitor greatly reduces universal jailbreaks. 🧵
2
15
644
the real world is fractally complex! "easy" things are not always easy, and we need great engineers and some courage to realize the potential of our existing and future tools.
Often I meet seasoned engineers who want to pivot into AI safety *research*. I think this is a mistake: safety is now a production problem and the world needs great engineers more than ever. I wrote a short post expanding on this:
2
13
979
Often I meet seasoned engineers who want to pivot into AI safety *research*. I think this is a mistake: safety is now a production problem and the world needs great engineers more than ever. I wrote a short post expanding on this:
people on here are thinkers so they assume that reasoning about alignment is the hard part and making sure millions(1) of task types, environments, and their respective virtual machines are configured correctly is the easy part but it’s essentially the opposite. you need to have worked in a large technology organization to appreciate the complexity and unreliability of this intuitively (1) made up order of magnitude
9
18
6
186
29,707
understanding forking paths and the dynamics of cot-reasoning models is so important. analyzing these was previously untenably expensive, but this result makes it much more tractable. excited to scale this up
LLMs are like Schrödinger’s cat: many possible trajectories, but you only see one outcome per run. To really understand models, and debug where they go wrong, you can find the "forking tokens" that lead to different trajectories. Our new research does this 100x more efficiently!
1
22
2,584
if you're a bright young thing that doesnt want to slot into your place in the machine, come join @GoodfireAI . we'll take you and your work seriously!
Replying to @RichardMCNgo
You don’t need to stay to do good research: work that aims to scale to much more intelligent systems can almost always be done on publicly-available models. (And nobody can think creatively in such pressure cookers.) You don’t need to stay to gain influence, because you’re almost definitely too scared to wield that influence in ways which matter (see the tweet on public criticism above). You don’t need to stay to generate scary demos or warn policymakers: we will be swimming in warning shots before long. You don’t need to stay to advance AI capabilities, even if you’re worried about the latest bogeyman your CEO has constructed: “we need to do it to beat the bad guys” is a big part of what makes people the bad guys. You don’t need to be there to implement guardrails or other short-term mitigations: there are plenty of bright young things eager to slot into your place in the machine. You don’t even need to be there to feel rich and powerful. For one, you’re probably already rich (and there’s plenty of money up for grabs elsewhere). But more importantly, the sense of power is almost entirely illusory. Everyone involved has locked themselves into a mindset where they’re incapable of making real choices, except for how fast to barrel down the train tracks that their fear has laid out for them. (More on how that happens in my QT below.) The world is more open-ended and malleable than it’s ever been. A single person who’s thinking clearly and creatively could build things that have never yet been imagined. Or they could just keep spinning in place. At this point AGI company employees owe it to both the world and to themselves to do something better.
4
76
7,230
dron retweeted
The ability to mix compute platforms is novel and really useful for interpretability research. We're fans of what @GoodfireAI is building for researchers and love to see Tinker integrated into Silico.
Silico integrates with some of the best open weight training infra, such as @thinkymachines's Tinker. It also can run on your own hardware. And it can combine the two! Here it ran RL on Tinker while running a probe-based reward model on our B200 cluster.
5
7
2
46
11,327
one thing i appreciate about silico is that it's a deeply humanist product. we designed silico to keep you in the experimental loop -- more observable, easier to steer, easier to understand we want to enable great researchers to do even more great research. if this sounds exciting to you please sign up and give it a try!
Replying to @GoodfireAI
Silico orchestrates compute, so you no longer need to babysit jobs. Silico coordinates experiments across GPU clusters, monitors every training run, and keeps long-running work moving without constant supervision. This lets Silico interpret Kimi K3, at 2.8T parameter scale.
3
6
50
3,942
concrete example: when setting up a multi-cluster RL run using tinker, silico reported the performance cost of its setup before launching the Big Run. small details like this make silico a joy to work with xitter.cf/RyanPanwar/status/2084…
Silico managed the complexity of multi-cluster training here, with the reward model running on our cluster and the rollouts and training steps running on Tinker. It even benchmarked the perf and reported this, alongside the code, before launching the big run.
1
212
interp is our best tool for debugging frontier biological models. check out this thread where we used silico to find a particular region in which boltzgen had completely mode collapsed. model training should be an engineering science, and silico is the ide!
Me: generate a novel 76-residue protein binder BoltzGen: best I can do is ubiquitin. I did a quick analysis of memorization in protein generations using @GoodfireAI’s Silico (now publicly available!). Silico makes it easier to investigate and debug model issues like this 🧵
15
866
2/ We extend the categorical Jacobian approach in doi.org/10.1073/pnas.2406285… to extract a layerwise readout of ESM-C's predicted contacts, i.e. which sequence positions are likely to be close physically. Remarkable concordance with known structures despite not being trained on any.
1
1
17
562
I’m very excited for Silico to accelerate alignment research! One example, done in Silico over just a couple days: RL erodes guardrails, but we can use reward shaping to prevent that. (1/6)
Silico, the platform for ambitious AI research, is publicly available today. AI is advancing fast. The tools to understand it need to advance even faster. Silico lets you interpret and train your models at frontier scale. Learn more + get access 🧵
4
21
160
13,086
research has never been more fun
Silico, the platform for ambitious AI research, is publicly available today. AI is advancing fast. The tools to understand it need to advance even faster. Silico lets you interpret and train your models at frontier scale. Learn more + get access 🧵
1
15
699
Silico, the platform for ambitious AI research, is publicly available today. AI is advancing fast. The tools to understand it need to advance even faster. Silico lets you interpret and train your models at frontier scale. Learn more + get access 🧵
38
119
69
835
346,546
me when i have two personality traits
10
1,375
research has never been more fun
> replicate J-space on GLM 5.2 > train a reward model and run RL to reduce hallucinations > show me how this model makes cancer predictions Using our platform Silico is like having a team of AI researchers ready to run experiments like these. Private beta is open now. 🧵 (1/6)
2
1
30
4,244
> replicate J-space on GLM 5.2 > train a reward model and run RL to reduce hallucinations > show me how this model makes cancer predictions Using our platform Silico is like having a team of AI researchers ready to run experiments like these. Private beta is open now. 🧵 (1/6)
22
129
41
1,356
270,513
turns out curve detectors are deeper than we realized! inceptionv1 actually encodes curve orientation continuously -- and not just that! it encodes rotational symmetries (ex treating 180° rotations as equiv) using the different fourier harmonics. just falls right out of a BSF:)
Replying to @GoodfireAI
We also revisited an interpretability classic: curve detectors in InceptionV1. Neurons and SAE features turn out to be fragments of one continuous orientation feature, and the block *also* contains higher-order Fourier harmonics that hadn’t been described before! (7/9)
1
2
23
1,532
interpreting bytes without knowing the data structure they're instantiating is so daunting. but when you have the right "type" you will find that memory is beautifully structured, that computing is Divine we need to interpret neural nets with the right types to fit
1
3
144
of course we're in a messy empirical science. but this result shows us that with the right frame hard problems become easy! this is what fundamental progress in interpretability looks like
3
88
Our work on Block-Sparse Featurizer is out 🧊 :) We revive an old idea from the structured sparsity literature and use it to carve activation space into meaningful regions. It's a first concrete answer to the question our concept manifolds work left open ! :)
If models think in shapes, our tools should too. Our latest research: Block-Sparse Featurizers (BSFs), a new way to find concepts in model activations - using multidimensional “blocks” instead of single directions. (1/9)
4
57
6
307
45,974