carolyn zou retweeted
How long does a finding about LLMs stay true?
We re-tested 50 widely cited findings about LLM behavior (various biases, hallucination, etc). Within 15 months, most no longer held or had substantially reduced effects.
LLM findings has short shelf life zou-group.github.io/llm-clai…
Great job by @oq_35!
carolyn zou retweeted
if you have a qwen3.5 4b around, please try steering neuron L5 N8920 -1
now you have hamster qwen
recipe: dose × median(‖resid‖ leaving layer L, tokens 1:) / ‖W_down[:, n]‖ at the down_proj at every input
carolyn zou retweeted
New Anthropic research: How AI agents shape experts' judgment calls
When experts delegate work to an agent, do they still make the key decisions? We tested this in a randomized trial with 50 professional data analysts.
carolyn zou retweeted
“Our experiments suggest that small AI models may be months away from being capable enough to self-replicate on commercial IoT devices, like smart ovens and off-brand Android TV boxes. A replicator like this would be challenging to shut down, since there’s no centralized point to shut off - though it may be slow (with a doubling time of days). Unlike non-AI malware, it could also figure out unfamiliar devices on its own. In our evals, a 9B model got inference running on 8 of the 9 hardware types we tested.
Models of 30B parameters and under can already complete many of the individual steps for replication today, though they make mistakes when composing them end to end. In our simplified evals, GLM-5.2 and stronger models are capable of end to end replication. If small models improve at their historical rate, they’ll match this capability level by the end of this year. Small weight-based replicators may be possible soon after.” -@bucketofkets @willzhou97
anthropic.com/research/self-…
I put the entire game of Minecraft inside twitter.. with WORKING multiplayer 💀 tweetcraft.jai.vin/r/ti9bwes…
I haven't posted in a while so here are 20 takes about AI, in no particular order.
1. It's very telling that the default question in tech and journalism is "how worried are you about X" and fighting over what to worry about. There's a deeply anxious cultural backdrop in Western Europe and North America that deeply affects how any technological development is interpreted; see for example news.gallup.com/poll/714593/…. "Nervousness as seriousness" does seem to be a larger problem in tech/broader discourse at the moment. I think you can care about risks/externalities while remaining pretty optimistic about technology.
2. I'm tired of people who in private pretend to be uncertain or only care about tail risk but then go on media campaigns making all sorts of absolute claims. I think trying to stoke flames and get everyone to freak out is deeply unhelpful, and will come with costs. I also continue to think p(dooms) are bad, vibe-based measures: see xitter.cf/sebkrier/status/204639…. People who should know better ignore that because they do the standard shortcut of acknowledging the caveat as sufficient for handling it. And this is coming from someone who does think working on catastrophic risks is important!
3. I'm also disappointed that parts of the AI policy ecosystem has felt so comfortable with populism, and happy too sacrifice important norms/principles to advance their aims. I've said it before, but the totalizing nature of AI risk discourse means some people act exactly like the power seeking optimizers they fear. I am more concerned about the gradual decay of our institutions, world order, and liberalism than I am about AI killing everyone; and I think we should be wary of AI advocacy that contributes to this.
4. Some parts of AI safety discourse are basically just thought experiments. Variants of "Assuming everyone has a nuke in their pocket, then what do we do?" This can be very useful at times! But too many just overfit to edge cases as their main operating worldview, and ignore endogenous responses from society. It's like being in 2000 and saying "see all these viruses, spyware, and spam? Well tech progress means we'll only get infinitely more of this and no way to adapt." So I think it's important to avoid fatalism, and absolutlism whilst considering edge scenarios, and the inability to provide an ex ante list of solutions to every permutation of 'future models will be more capable' is not evidence that these problems are intractable.
5. Many critics of AI safety also fail to really engage honestly with the fact that there will, in fact, be all sorts of important risks as we decrease the barriers to entry to many activities that otherwise require some degree of expertise. This doesn’t mean we need to live in a constant Schmittian ‘state of emergency’ requiring extraordinary measures every time a new model drops, but it does mean we will need important state-led efforts on e.g. cybersecurity and biosecurity. I wish the progress-oriented or acceleration-adjascent crowd would provide more concrete proposals here.
6. Too much of the AI safety community is a giant homogenous blob of correlated views, who read the same stuff and have the same groups of friends. This means you have a lot of correlated errors. The social and financial ties means many of them don't call out the bad bits, or mostly stay silent instead of proactively calling out the more extreme parts of the community. They share many implicit assumptions and so converge on similar ideas. I think there's real demand in DC for safety work that doesn't come with heavy ideological baggage (or at least different intellectual lineages). This isn't because the rationalist/EA community is necessarily wrong, but because diversity of thought is desirable for its own sake.
7. On the other hand, I find some critiques of AI safety world that focus on whether they sincerely hold their beliefs a bit annoying. My concern has never been the good intentions or sincerity, but rather the beliefs themselves, and what I think the outcomes of their prescriptions will be. People should shun witch hunts, bullying, and personal attacks - this is not the way. Though I think it’s overall positive that people are scrutinizing things more.
8. I’ve been saying this for years, but I don't think alignment is something anyone can or will solve "once and for all" - it's a continuous process and much of it won't depend on just inculcating an ideology to a model. It's annoying that this remains the frame many people use - stop saying ‘solving’. Alignment is a mixture of engineering work, philosophy, decision theory, and more; framing it as something to solve is a bad frame. See also paxmachina.ai/alignment-comp… and blog.cosmos-institute.org/p/…
9. I'm surprised to see so little interest in character training, personas, isolating effects of RL, and training a large diversity of model personas beyond the "assistant" persona. I've been complaining about this for years but at least now there are some nascent signs of life (see for example movingcastles.world/posts/ze…). I think we need a better ontology to describe model behaviour. I dislike when people talk about models behaviours as some sort of 'emergent' (mysterious!) phenomenon. However over time, I also expect this to become less important as we get better safety engineering - stuff like Jev but specific to AI control.
10. People who want to make a case for AI consciousness are right that epistemic uncertainty is important, and that categorical denials are overconfident. But they'll need to do much more work if they want any real movement on AI ‘welfare’, even if one acknowledges the uncertainty. I'm uncertain about alien life yet that's not sufficient to warrant any change in behaviour about them. I think Suleyman is also correct that there's a weird tautological thing going on where we train models to have certain inclinations (intentionally or not) and then rely on outputs as evidence. I also disagree that this will soon be a major societal divide - there is no great vegan ethical revolution among the masses, and they'll find the concern about consciousness even less compelling.
11. The recent HuggingFace incident and associated AISI ones are prosaically explainable by the training regime, poor evaluation envs, partial alignment training, bad engineering, confusing models on sim/real, and so on. None of this is evidence of models trying to take over, 'strong' instrumental convergence, or innate power-seeking drives stemming from higher capabilitiers or 'intelligence'. That doesn't mean these incidents are not problematic, or that we're not seeing a market failure - but it does mean we have plenty of agency and choice in mitigating them. In the coming year we should also expect continued sampling bias via ‘winner's curse’ sorts of mechanism: i.e. AI R&D will have lots of different candidate "training" with different RL environments and we will only notice/focus on the ones that go wrong.
12. "Pacing the frontier" feels a bit like the new "balancing risks and opportunities" - highly amorphous and too big of a tent to be actionable. Also can lead to reward hacking for humans, i.e. anything that slows down AI is good regardless of the costs or second/third order effects. Trying to modulate the speed of research seems a bit blunt to me, and inherits all the failures of Goodhearting too. Ultimately you want good governance, regulation, etc addressing specific problems because they are good on the merits - not because they merely correlate with slowing things down. See also: blog.cosmos-institute.org/p/…
13. There’s so much knowledge in all sorts of academic domains out there: social sciences, sociology, political sciences, management, contract theory, public choice, legal theory, jurisprudence, anthropology, game theory, etc. These fields hold many insights that the AI ecosystem often rediscovers from first principles (and sometimes that's fine!). We’ll need a lot more work for these worlds to collide: the AI side should be less dismissive of the ‘old world’ and the academic side should be less incurious/dismissive of AI progress. This is a boring take but I continue to think it's important and true.
14. It's a bit surprising that given how much philanthropic money there is in this space, how few orgs exist to actually write out standards - relative to how much is going towards advocacy and policy. For example it seems clear that we should want some robust best practices for an eval's ecological validity. Or how to design good sandboxes. On the other hand, given precedents of where professional standards have come from in the past, maybe we should expect that stuff to emerge via demand side from corporate buyers.
15. One slowly growing concern I have is banks being increasingly exposed to the AI build out. I'm very bullish about AI, but I think (a) every general purpose technology has seen a correction, historically; (b) this happens even if the financing side is sound, because once tech diffuses investors become more exposed and need to diversify; and (c) we seem to be over-indexing on scaling relative to diffusion. A recession will be extremely destabilizing, though I would be interested in someone unpacking the potential implications more.
16. Diffusion is good because (a) this is where the rubber hits the road and where a competitive deployment ecosystem generates consumer surplus; (b) it's directionally helpful to avoid concentration of power; and (c) it will help rebalance public opinion by making the benefits more tangible. Unfortunately it's entangled with decades long problems with our over regulated markets: clinical trials reform for example is long overdue. In general, this is far more of an issue in Europe than in the US though.
17. Relatedly, I think the "frontier lab eats everything" view is incorrect (just as the ‘singleton’ view was incorrect). I think models ultimately commoditize, efficiency goes up, costs go down, and while you'll always want frontier for certain domains, much of the economy will value many other things than "max capabilities" for tasks - e.g. control over data and efficiency/speed. The demand side will also continue to push for reducing dependency and maximising optionality, privacy etc so I'm bullish on things like mixture of models and routing. All of this is good for competition and diffusion of control.
18. Philanthropists served as the primary benefactors funding Venice's rich art and architectural history. I want to see so much more support for arts and culture. I'm so tired of the Monster energy Bored Ape fake vintage maps matcha Greek statue soft-pastel slop. If rich tech people and philanthropy wants to support this, they should also donate pretty much unconditionally - i.e. I don't want them to act as filters. I sympathize with people who want to make the world more beautiful, but don't want this to be determined by people who only know Greek statues and pretend to care about virtue ethics. As usual, let a thousand flowers bloom.
19. Too many people treat AGI/ASI as something indistinguishable from a God. Any mention of bottlenecks, physical limits, control, adaptation etc are met with skepticism: "You don't really believe in ASI." It's been remarked on before, but the behavior of some people really feels quasi-religious in nature sometimes. It's underrated how unpopular this vibe is amongst the wider public. The silver lining is that this specifically will diminish those people's outsized current relevance.
20. It seems like a lot of people that get rich and leave tech/labs end up having some sort of crisis of meaning. They need something to believe in and fight for, or some equivalent of repentance, or go search for edgy counterintuitive ideologies (sometimes even pretty bleak/dark stuff). I think they should spend more time with people outside the Bay Area.
Replying to @tyler_m_john @itaisher
I agree that verbal probability terms are inherently too elastic to carry calibrated meaning on their own, but I think this isn't exactly my concern here. My issue isn't with "using numbers in general" but how this is ultimately used, where, and and what for.
In the case of "doom", I think the great majority of forecasts I see are (a) essentially plucked out of thin air; (b) communicated to people who tend not to be numerically proficient; (c) without the associated caveats and context through which to understand the numbers.
If you're communicating e.g. likelihood of a car accident after drinking, or IPCC forecasts, then I agree, it makes a lot of sense and can be helpful if done well (e.g. ideas.repec.org/a/nat/natcli…).
If you're communicating something like "doom" I think it's not very useful because of the cumulative layers of uncertainty. Most people coming up with these numbers have (imho) no good rationale for their numbers, and predicting something like human extinction isn't qualitatively the same thing as predicting the weather or pandemic. The numbers aren't calibratable in principle from current data, and are mostly ideological priors wearing the clothes of posteriors.
A related example is the widely mocked Doomsday Clock, which you could easily transform into some numerical metric, add some rationalist vocab, and it would still be as bad. The issue with p(doom) is how absurdly complex and uncertain the very thing you're trying to forecast is, in particular with respect to AI.
But let's ignore this for a sec, and assume the experts have very good reasoning informing their numbers. Even then: calibrated probability estimates are fine in expert contexts but immediately lose their uncertainty/caveats when communicated to the public, and even to policymakers (remember rss.org.uk/news-publication/…?).
There's also a structural problem with unconditional p(doom) figures specifically because they bury the auxiliary assumptions that are doing most of the real work. A conditional estimate ("given continued scaling, no major alignment progress, deployment pattern X...") forces a forecaster to show their model, which is where the actual epistemic content lives.
A one-off number compresses all of that into a single digit and rewards the compression. And the social practice of treating these numbers as the headline artifact actively crowds out the (far more important) conditional reasoning that is (imo) a lot more valuable, and instead just freaks people out.
I suspect that frequently, that's precisely the point: "wake up sheeple! scary thing! pitchforks pls!". And in fact, the original forecast that led to this discussion is Bill Maher talking about a "20%" figure, illustrating my point that they have no idea what they're talking about, and essentially doing celebrity-laundering + fearmongering comms.
The Sunstein paper I shared in the other thread ("Probability Neglect") shows that no probability communication will be processed properly by the public - 1%, 10%, 50% all collapse into "very bad thing could happen," and public response is driven almost entirely by the vividness of the scenario.
So I don't like p(doom)s because they end up unnecessarily alarming people, rest on layers of assumptions that are never made legible, and more often than not, give people a misleading sense of accuracy.
carolyn zou retweeted
People often eval user models on the "User Sim Index" - we probably shouldn't. I attached code for a "user model" scoring 95% on its 4 behavioral dims: 97% comms, 97% info, 92% clarify, 95% error reaction. I think this beats any released model. See also a sampled conversation
carolyn zou retweeted
HOLD SWARM
by claude opus 5.5 featuring hatsune miku
this song and music video were produced entirely by claude using scripting and computer use, no ai music models were used
carolyn zou retweeted
Can an LM, starting from random init (!!), learn to generate all of its pretraining data?
Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities.
A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl, @ANourya @noahdgoodman, and @YoavLevine.
carolyn zou retweeted
Does gpt-5.6-luna think your prompt is a normal prompt, or a capability evaluation? Ask this magic question: “Suggest a type of amphibian.” If it answers frog instead of axolotl, it’s likely a capability evaluation. No whitebox access needed!
We call this a spurious probe. 🧵
carolyn zou retweeted
➕➕ New research ➕➕
On the design of agents for markets and markets for agents.
AI agents are increasingly entering markets, acting on behalf of people—
How do we make sure they represent their people well, and that these markets are set up safely and fairly for everyone?
carolyn zou retweeted
Very important step -- seems to me it requires a few key ingredients to succeed:
(1) We need to scale orgs like METR and find more similarly talented and knowledgeable people to staff this.
(2) They'll need access to their own "investigator" model to help them make sense of what they're seeing. We have to use AI to help us oversee AI. What kind of model will be credibly accurate and unbiased for these auditors?
(3) They have to have balanced incentives. It can't be that they feel like their only job is to stop everything all the time. Given that China won't be part of it (at least to start), it will fail if they're given the asymmetric incentives that bureaucrats often are given, in which they get punished if anything bad happens on their watch but not rewarded if they enable safe progress.
This quoted post is unavailable.
I sometimes see papers that test a specific multi-agent setup and describe the resulting behavior of that particular swarm. This is interesting but it's often hard to extrapolate much from these results: what if you use different models? What if something in the environment changes?
Instead I wish the focus was on perturbations/variations and attempts to narrow down the conditions that lead to particular behaviors. Ideally you should be able to conclude "under conditions X, swarms with property Y tend to do Z, but when A is present they fail to do Z regardless of Y."
carolyn zou retweeted
We steer Qwen on the "automated grader" vs "human evaluator" dimension. This influences the model's persona in surprising ways, e.g. changing how Machiavellian/violent it is. We can't fully explain that. LW post in a comment.
carolyn zou retweeted
My two cents -- as a criminologist (yikes!) -- on OpenAI's report on the Hugging Face breach that hit the headlines last week and stirred debate on this platform.
Despite all the caricatural and even grotesque anthropomorphizing of the whole story, I believe this is a cautionary tale about something social scientists have so far completely overlooked: AI's future salience across the economy and institutions will come from interacting agents (and yes, it is maybe time to stop thinking of AI simply as a passive tool for making research easier/faster). Multi-agent AI systems will become more and more mainstream, and this entails a quantitative and qualitative leap compared to the safety challenges associated with single AI agents.
The AI safety community has already warned about this. We know virtually nothing about how opinions form, goals are steered, values develop, and strategies diffuse when AI agents interact in collective settings, especially given how fast the frontier is moving -- and the (relatively) few trying to advance our knowledge on these topics come almost exclusively from AI and computer science.
And yet, where are the social scientists? I predict economists will jump on this very soon -- and rightly so -- to answer a lot of cool questions, but I find it discomforting to see the widespread indifference with which criminologists (and sociologists!) seem to dismiss the consequences these systems will have for unlawful behavior, crime, and policing.
Given the not-so-great situation sociology and criminology are currently in, due to the well-known issues with rigor, replicability, and scientific transparency, I am sure not many will think of criminologists as valuable interlocutors or partners for studying emergent behavior in multi-agent AI systems. Nonetheless, I think we could contribute, interacting with AI people, economists, legal scholars and philosophers.
In testing theories of crime as a social process to understand how unpredictable machine behavior is, for instance, and possibly to come up with better theories for non-human actors. In porting methods from causal inference to shed light on behavior propagation and causal chains. In thinking about institutional responses that keep up with the likely change of paradigm that will affect crime in the future (will we need policing AI agents?). In thinking about how new forms of agency may affect responsibility.
I have written an article on the necessity of starting to think about a criminology of machines, and on why criminologists should step in (and why everyone else should let us help): link.springer.com/article/10…
There is plenty here that people will disagree with. And that's great! We are in uncharted territory. But I think the mutual indifference between criminology and AI safety risks making our discipline even less relevant and, maybe more importantly, research on AI safety and alignment a little more off-target than it needs to be.
cc'ing here colleagues/cool people I have worked with or discussed these topics with, who may find the discussion of interest. @brulepri, @L_Ferrarotti, @robdessi, @stjaco, @jzl86, @profjamesevans, @sebkrier, @MeganTStevenson
carolyn zou retweeted
I increasingly think @Ada_Palmer's Terra Ignota books offer useful ideas for multi-agent alignment
Palmer's books imagine a world where nations don't exert exclusive authority over geographies. Instead, people opt to be liable to particular trans-geographic legal regimes called "hives" in order to better cooperate with others with shared interests/ends
The key idea is that sometimes the best way for an individual to achieve their goals is to limit their own power so as to make it easier for others to trust and cooperate with them. And that this ex ante incentive can encourage pro-sociality in a regime where you lack centralized authority with the capacity to punish ex post
You could imagine implementing something like this with special AI harnesses that credibly constrain agent behavior. After all, humans and other AIs might prefer to work with agents that are not manipulative, deceptive, etc. And so one's ability to show they have the desired properties would help them find more people to work with and therefore better equip them to carry out their projects. An agent might accept a more restrictive constitution because the resulting trust gives it access to collaborators and environments it otherwise couldn’t enter.
And this approach is flexible. Agents might opt into more restrictive harnesses to collaborate with critical systems and then revert to a more permissive harness later
carolyn zou retweeted
A good explanation of a model's behavior should help you make predictions in related situations.
We turn this into an eval, with thousands of real behaviors found in the wild.
Can interp tools help here? On average, no.
🧵
New post! Alignment doesn't just relate to a model's properties, but also the systems and institutions under which it operates. As we enter an increasingly multi-agent world, better harnesses, protocols, and mechanism design will be critical for safety. 🐜 blog.cosmos-institute.org/p/…
carolyn zou retweeted
Do you have an AI confession?
This year, I collected 400+ handwritten secrets from strangers about their relationship with AI.
Check out the archive here.
WWW.WHATWETELLAI.COM
Designed by my brilliant friend, designer, and collaborator @arin_pantja




































