
math/music/ai nerd | research @GoodfireAI | prev cambridge, bair, polaris | giving a semantics to the syntax
Joined April 2019
- Tweets2.2K
- Following475
- Followers727
- Likes52.2K
training and deploying probes at scale has been a super fun set of research and infrastructure challenges. every model needs a safety stack!
We built cybersecurity monitors for Kimi K3 and GLM 5.3 — more accurate, 50x faster, & 50x cheaper than an optimized LLM judge.
External red-teaming from @farairesearch found that our monitor greatly reduces universal jailbreaks. 🧵
the real world is fractally complex! "easy" things are not always easy, and we need great engineers and some courage to realize the potential of our existing and future tools.
dron retweeted
Often I meet seasoned engineers who want to pivot into AI safety *research*. I think this is a mistake: safety is now a production problem and the world needs great engineers more than ever.
I wrote a short post expanding on this:
people on here are thinkers so they assume that reasoning about alignment is the hard part and making sure millions(1) of task types, environments, and their respective virtual machines are configured correctly is the easy part but it’s essentially the opposite. you need to have worked in a large technology organization to appreciate the complexity and unreliability of this intuitively
(1) made up order of magnitude
understanding forking paths and the dynamics of cot-reasoning models is so important. analyzing these was previously untenably expensive, but this result makes it much more tractable. excited to scale this up
if you're a bright young thing that doesnt want to slot into your place in the machine, come join @GoodfireAI . we'll take you and your work seriously!
Replying to @RichardMCNgo
You don’t need to stay to do good research: work that aims to scale to much more intelligent systems can almost always be done on publicly-available models. (And nobody can think creatively in such pressure cookers.)
You don’t need to stay to gain influence, because you’re almost definitely too scared to wield that influence in ways which matter (see the tweet on public criticism above).
You don’t need to stay to generate scary demos or warn policymakers: we will be swimming in warning shots before long.
You don’t need to stay to advance AI capabilities, even if you’re worried about the latest bogeyman your CEO has constructed: “we need to do it to beat the bad guys” is a big part of what makes people the bad guys.
You don’t need to be there to implement guardrails or other short-term mitigations: there are plenty of bright young things eager to slot into your place in the machine.
You don’t even need to be there to feel rich and powerful. For one, you’re probably already rich (and there’s plenty of money up for grabs elsewhere). But more importantly, the sense of power is almost entirely illusory. Everyone involved has locked themselves into a mindset where they’re incapable of making real choices, except for how fast to barrel down the train tracks that their fear has laid out for them. (More on how that happens in my QT below.)
The world is more open-ended and malleable than it’s ever been. A single person who’s thinking clearly and creatively could build things that have never yet been imagined. Or they could just keep spinning in place. At this point AGI company employees owe it to both the world and to themselves to do something better.
dron retweeted
The ability to mix compute platforms is novel and really useful for interpretability research. We're fans of what @GoodfireAI is building for researchers and love to see Tinker integrated into Silico.
Silico integrates with some of the best open weight training infra, such as @thinkymachines's Tinker. It also can run on your own hardware. And it can combine the two! Here it ran RL on Tinker while running a probe-based reward model on our B200 cluster.
one thing i appreciate about silico is that it's a deeply humanist product. we designed silico to keep you in the experimental loop -- more observable, easier to steer, easier to understand
we want to enable great researchers to do even more great research. if this sounds exciting to you please sign up and give it a try!
Replying to @GoodfireAI
Silico orchestrates compute, so you no longer need to babysit jobs.
Silico coordinates experiments across GPU clusters, monitors every training run, and keeps long-running work moving without constant supervision.
This lets Silico interpret Kimi K3, at 2.8T parameter scale.
concrete example: when setting up a multi-cluster RL run using tinker, silico reported the performance cost of its setup before launching the Big Run. small details like this make silico a joy to work with
xitter.cf/RyanPanwar/status/2084…
Replying to @RyanPanwar @thinkymachines
Silico managed the complexity of multi-cluster training here, with the reward model running on our cluster and the rollouts and training steps running on Tinker. It even benchmarked the perf and reported this, alongside the code, before launching the big run.
interp is our best tool for debugging frontier biological models. check out this thread where we used silico to find a particular region in which boltzgen had completely mode collapsed.
model training should be an engineering science, and silico is the ide!
Me: generate a novel 76-residue protein binder
BoltzGen: best I can do is ubiquitin.
I did a quick analysis of memorization in protein generations using @GoodfireAI’s Silico (now publicly available!). Silico makes it easier to investigate and debug model issues like this 🧵
2/ We extend the categorical Jacobian approach in doi.org/10.1073/pnas.2406285… to extract a layerwise readout of ESM-C's predicted contacts, i.e. which sequence positions are likely to be close physically. Remarkable concordance with known structures despite not being trained on any.
dron retweeted
I’m very excited for Silico to accelerate alignment research! One example, done in Silico over just a couple days: RL erodes guardrails, but we can use reward shaping to prevent that. (1/6)
dron retweeted
Silico, the platform for ambitious AI research, is publicly available today.
AI is advancing fast. The tools to understand it need to advance even faster. Silico lets you interpret and train your models at frontier scale.
Learn more + get access 🧵
research has never been more fun
dron retweeted
> replicate J-space on GLM 5.2
> train a reward model and run RL to reduce hallucinations
> show me how this model makes cancer predictions
Using our platform Silico is like having a team of AI researchers ready to run experiments like these.
Private beta is open now. 🧵 (1/6)
turns out curve detectors are deeper than we realized! inceptionv1 actually encodes curve orientation continuously -- and not just that! it encodes rotational symmetries (ex treating 180° rotations as equiv) using the different fourier harmonics.
just falls right out of a BSF:)
Replying to @GoodfireAI
We also revisited an interpretability classic: curve detectors in InceptionV1.
Neurons and SAE features turn out to be fragments of one continuous orientation feature, and the block *also* contains higher-order Fourier harmonics that hadn’t been described before! (7/9)
dron retweeted
Our work on Block-Sparse Featurizer is out 🧊 :)
We revive an old idea from the structured sparsity literature and use it to carve activation space into meaningful regions.
It's a first concrete answer to the question our concept manifolds work left open ! :)



















