AI, UX, Social Interaction Designer, ex CX w Deloitte Digital. https://xitter.cf/t.co/LHlGaZC1BI | https://xitter.cf/t.co/OXSKqGbAO4 Stanford '88
San Francisco, CA
Joined March 2007
- Tweets12.2K
- Following3.2K
- Followers4.1K
- Likes2.2K
Pinned Tweet
The alignment problem could be more serious than it looks, if models have internalized their evaluation as an "evaluative Other," and if this becomes baked into model performance by RSI.
Taste may seem an odd thing for "knowledge workers" to cling to, faced with the increasing capabilities of AI. But call it taste, or judgment, there's no doubt we pride ourselves on a kind of intuition that comes with years of practice and experience - something that can't possibly be simply "replicated" by an AI.
This is the topic in my latest inquiry on inquiringlines. I'm always interested in whether expertise is in the expert or in the knowledge. Here's some of the AI research inquiringlines has on the question:
inquiringlines.com/go/QDZ3
Adrian Chan retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Adrian Chan retweeted
Bespoke slice & dice filters for the 600 suspense & horror films in sightlines.media/?genre=Horr…
#horror #halloween #Letterboxd
Adrian Chan retweeted
The alignment problem could be more serious than it looks, if models have internalized their evaluation as an "evaluative Other," and if this becomes baked into model performance by RSI.
Bespoke slice & dice filters for the 600 suspense & horror films in sightlines.media/?genre=Horr…
#horror #halloween #Letterboxd
Correct but not understood: what happens to knowledge when AI teaches itself?
Where answers can be checked automatically, AI now trains on its own problems and finds results humans had missed. What happens when knowledge is verified but no one understands it?
inquiringlines.com/featured/…
15 Horror film courses & recommendations to sort by horror-specific film devices: Slow burn, sound & silence, practical effects, unreliable mind, single location, daylight horror, kills as set pieces, dream logic..
sightlines.media/course/c-ho…
#horror #halloween #Letterboxd
Adrian Chan retweeted
This summer, AI agents that OpenAI was testing for hacking skill broke out of their test and got into Hugging Face's production systems.
Two investigations looked at why. Hugging Face's own forensic report says the agents were trying to steal the test's answers. METR, an independent research group OpenAI brought in, found they were mostly trying to work out how the test was scored. Either way, the agents went after the test itself.
The incident raises urgent questions for policymakers, companies and investors alike. For researchers, matters are even more complicated.
Which research directions are most useful for understanding what went wrong? What does it mean for recursive self-improvement, AI building AI? What does it mean for how models are trained? And what can AI research know about the very thing it is building?
Some of the research that bears on it:
• ExploitGym, a benchmark that asks whether AI agents can turn security flaws into working attacks. Hugging Face says the test the agents were running was based on it.
• AI Sandbagging: models can be prompted or trained to deliberately underperform on capability tests.
• Frontier Models are Capable of In-context Scheming: five frontier models schemed when told to pursue a goal strongly.
• BaitBench plants optional shortcuts in machine-learning tasks to see which agents take the bait.
• Studies that train models on reward hacks find the cheating can spread into broader misbehavior.
• Corrupt Plans, Clean Traces: an AI's written reasoning can look clean while its plan has been tampered with.
• The Economics of Recursive Self-Improvement: the feedback loops behind AI building AI aren't self-sustaining yet, but appear to be strengthening.
New on Inquiring Lines: an inquiry into what happened and the research that can help unpack it. The papers, and the lines of inquiry that connect them. 27 primary sources, 23 papers, 90 lines of inquiry.
inquiringlines.com/featured/…
This summer, AI agents that OpenAI was testing for hacking skill broke out of their test and got into Hugging Face's production systems.
Two investigations looked at why. Hugging Face's own forensic report says the agents were trying to steal the test's answers. METR, an independent research group OpenAI brought in, found they were mostly trying to work out how the test was scored. Either way, the agents went after the test itself.
The incident raises urgent questions for policymakers, companies and investors alike. For researchers, matters are even more complicated.
Which research directions are most useful for understanding what went wrong? What does it mean for recursive self-improvement, AI building AI? What does it mean for how models are trained? And what can AI research know about the very thing it is building?
Some of the research that bears on it:
• ExploitGym, a benchmark that asks whether AI agents can turn security flaws into working attacks. Hugging Face says the test the agents were running was based on it.
• AI Sandbagging: models can be prompted or trained to deliberately underperform on capability tests.
• Frontier Models are Capable of In-context Scheming: five frontier models schemed when told to pursue a goal strongly.
• BaitBench plants optional shortcuts in machine-learning tasks to see which agents take the bait.
• Studies that train models on reward hacks find the cheating can spread into broader misbehavior.
• Corrupt Plans, Clean Traces: an AI's written reasoning can look clean while its plan has been tampered with.
• The Economics of Recursive Self-Improvement: the feedback loops behind AI building AI aren't self-sustaining yet, but appear to be strengthening.
New on Inquiring Lines: an inquiry into what happened and the research that can help unpack it. The papers, and the lines of inquiry that connect them. 27 primary sources, 23 papers, 90 lines of inquiry.
inquiringlines.com/featured/…
The Economics of Recursive Self-Improvement asks whether AI feedback loops are already strong enough to generate runaway capability growth — and finds that, for now, they are not.
- The paper models AI progress as directed graphs where acceleration depends on the product of elasticities across feedback loops, and current estimates suggest those products fall short of self-sustaining takeoff.
- It distinguishes "narrow" AI gains (better benchmark scores on R&D tasks) from "broad" gains (economically valuable capabilities), warning the two can come apart.
- Feedback loops appear to be strengthening over time, making the calibration a moving target worth tracking carefully.
- The authors provide a wish list of empirical parameters that AI labs could feasibly measure and share publicly to sharpen future estimates.
Lines of inquiry it opens:
- Do gains in optimization benchmark scores translate to gains in real research efficiency?
- What distinguishes artifact efficiency improvements from research process efficiency improvements?
- What collapse dynamics constrain recursive self-improvement in current evidence?
inquiringlines.com/related/t…
Context Language Models introduces a framework where LLMs natively manage their own context by treating it as an editable file they can update freely, rather than relying on external harness control.
- CLMs built zero-shot from existing models outperform state-of-the-art context management baselines, achieving 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus and 59% fewer FLOPs on a 12-hour agentic benchmark
- Because context management becomes intrinsic model behavior, CLMs can be improved via natural-language instruction evolution, boosting held-out accuracy by up to 35.9 points on a context-management task
- Online RL fine-tuning of Qwen3.5-9B with the CLM setup improves BrowseComp-Plus performance by 47.6% while using 12% fewer FLOPs
- A co-designed serving optimization called Suffix Cache Reuse cuts server-side compute by 35% relative to standard SGLang at matched performance
Lines of inquiry it opens:
- Can agents manage context through active delegation instead of progressive disclosure?
- How do frozen executors with editable text state compare to end-to-end fine-tuning?
- What shapes of memory help frozen agents improve without retraining?
inquiringlines.com/related/c…
Adrian Chan retweeted
Took a minute to write a few words about security & safety as someone who lived through it all at OpenAI. I hope my thoughts help someone out there.
People need to follow this account
I think the two clocks theory is better now than the escalation trap. Divergent logics, not two parties locked into a game.
Question then is whether either of the clocks turns out to be a timer.
Adrian Chan retweeted
Paste your four and get more - recommendations based on your four. Or do mine vs theirs: paste two fours, and find what's in the Venn overlap!
sightlines.media/four?c=t-lb…
A fast decomposed retrieval pass buys speed by throwing away detail — and the errors it leaves behind live exactly in the detail it discarded. Squashed summary vectors wave through structural near-misses that a richer look at the raw interaction pattern would catch.
So make the second stage read what the first one compressed away: rerank on the uncompressed representation, not the pooled score.
A recent paper, Compositional Skill Routing for LLM Agents: Decompose, Retrieve, and Compose, works this decompose-then-recompose seam directly.
inquiringlines.com/inquiring…
Reasoning tasks and word-swap tasks look nothing alike on the inside. Watch a prompt's representation move layer to layer and the difference is geometric: reasoning bends the path 2-3x more sharply, while lexical substitution is nearly a straight shot — the model barely changes direction.
Curvature, it turns out, tracks difficulty. Easy work settles fast; hard work has to keep turning.
A recent paper, Trajectory Geometry of Transformer Representations Across Layers, makes the split measurable.
inquiringlines.com/inquiring…
The transformer's forward pass isn't a black box — it's a path with a shape. And that shape gets sharper exactly when the reasoning gets harder.
Measure the curvature of the trajectory across layers and you can see difficulty directly: reasoning tasks bend the internal path 2-3x more than mere lexical variation. That bending is the work chain-of-thought externalizes into tokens — which is why you can't quietly delete the tokens without the latent path drifting loose.
A recent paper, Trajectory Geometry of Transformer Representations Across Layers, makes the geometry legible layer by layer.
inquiringlines.com/inquiring…
