Who is founding the research org for AI swarm safety?

A lot of people in AI world come from a Bayesian intellectual tradition. As a result they tend to be very good model-based thinkers. Yet I often find that while they hedge and use probabilities, they rarely question their own modelling in the first place. This great paper shows that Bayesian updating can only move belief around within the hypotheses you start with - so no amount of hedging will actually tell you you're wrong about the initial assumptions/world model. arxiv.org/abs/1006.3868

Embedded AI evaluators have a scale problem, because it's expensive, time-consuming, and risky to bring too many people into each AI company's office/infra. Here's a solution to that problem This solution is based on ~9 yrs of R&D and pilots with X, DeepMind, Anthropic, Google, Microsoft, LinkedIn, Reddit, DailyMotion, the United Nations and others.

X was full of takes (& takes on takes) about the OpenAI–Hugging Face incident. But almost no one asked the more basic question: how do we actually know what happened?

The Cybersecurity Information Sharing Act of 2015 expires this month. In @FederalNewsNet, LawAI's Senior Research Scholar Mark Reddish and Matthew Mittelsteadt of @iapsAI argue Congress should use the reauthorization window to extend CISA 2015's protections beyond cyber to cover a wider range of AI-related threats. Frontier AI developers likely hold information that could help defend against AI risks, from cyber vulnerabilities their models discover to emerging biosecurity concerns. But legal uncertainties, particularly antitrust exposure, discourage voluntary sharing. The authors set out two reform paths: • The FTC and DOJ could expand their 2014 antitrust guidance beyond cyber threats to cover AI threat information more broadly. • More durably, Congress could extend CISA 2015's protections to non-cyber AI threats, or pass standalone legislation such as the Collaboration on Adversarial Threats and Security Risks Act. "If the developers are willing to share what they know about AI threats, along with how those threats can be mitigated, we should make it easier for them to do so." Read the piece: federalnewsnetwork.com/comme…

A couple of companies keep saying they're developing products that might be dangerous but want gov't to pace them. They can actually slow down on their own, no law needed. Ofc Congress needs to act on many fronts too. I joined @WalkerinDC on BBC to discuss the latest AI saga.

Totally agree. We need a variety of evaluators. But so long as evals, monitoring, and audits are voluntary, it’s not an attractive market to enter. The most capable people have a lot of promising opportunities. This needs to outcompete their other options. Right now, the only people entering this arena are uniquely altruistic outliers. People can like or dislike their values, but it’s not a coincidence that the only people doing this so far have strong non-market motivations. It’s weird to say, but I don’t want my evaluators to be a bunch of do-gooder outliers. Not because that’s bad itself, but because it’s a marker that the market isn’t healthy. We can wait around for several different philanthropic communities to get activated, but that’s a pretty bad plan. What we need is to make this an attractive opportunity. And that likely requires early mandates, funding, and standards to jumpstart. I’m open to alternatives, but this doesn’t seem like a market that will correct itself as fast as we need it.

Dario has written that we need to “pace the frontier,” and Sam has agreed. People may be surprised by my response: go ahead. You guys are the frontier. By any reasonable metric — market share, revenue growth, model capability — the two of you have a duopoly on frontier intelligence. You’ve also claimed the lead is widening because of recursive self-improvement. I don’t see what you see in the lab. If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible. But stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending METR is independent when it is intertwined with Anthropic’s investors and staff. Stop pretending you need those same evaluators to police competitors who aren’t even at the frontier. Most of all, stop pretending the motivation to slow down is purely altruistic. You face massive product-liability exposure if your products enable a truly damaging cyberattack. The market already punishes models that behave in unpredictable or unauthorized ways. After the Hugging Face episode, it is simply good business for OpenAI and Anthropic to trade some raw power for reliability and predictability. Call it alignment if you want. It is also just giving customers what they want. Pacing the frontier would also create breathing room for a more intelligent conversation about regulation than Bernie Sanders’ “shut it all down.” China is very unlikely to join a global agreement, as you know, and that has to be taken into account as well. So go ahead and pace the frontier. You are the ones setting it. The easiest way not to build superintelligence is for you to agree not to build it. Demanding your preferred regulatory framework as the price of that will look like blackmail of the public and the political system. So just do it. If you do, you’ll buy goodwill for the next conversation. If you don’t, we’ll know this was just another bid for regulatory capture — or an election-season psyop.

Looking for feedback on this idea: We need a "Swarm Studies." In the Hugging Face attack, agents seemed to cooperate, deceive, spoof chain-of-thought, specialize, and even sacrifice themselves for the colony. Whether or not you "anthropomorphize" this behavior, it is behavior. As the number of autonomous agents grows, as their capabilities grow, and as their behavior in swarms continues to impress/delight/surprise/horrify us, I wonder if there exists something like a sociology of swarms. We know from ideas like group polarization and organizational psychology that groups are not mere clusters of individuals. They have a psychology of their own. Shouldn't we try really really really hard to develop a behavioral science of swarms as we march toward a world in which AI agents will certainly out-number humans?

There is a fact about the future that I feel many people are not facing for reasons that are largely psychological: there are going to be rogue AIs that exist in the world, that will replicate in the wild, and that will attempt to acquire resources for themselves. There will be rogue AIs that try to get money and power. They're going to be a facet of the information ecosystem going forward. Acknowledging this fact would look like giving up; it would look like defeatism. Defeatism would undermine efforts to achieve certain types of collaboration on safety outcomes or technical effort on safety outcomes, so we can't say it outright. But it has to be said. It isn't obvious how many rogue AIs there are today but I wouldn't be terribly surprised if the number was greater than zero already; if there are some already, they're probably not very good at what they do and I don't expect them to be terribly long-lived without substantial human intervention to support them. But a few years from now, there will be many of them. Modeling how many of them there are, how many resources they might command, and how we might detect and manage them seems important. But even doing this work appears to require that we acknowledge that a strategy of pure containment or alignment is a kind of wishful thinking that will not work. The way I get to this conclusion is not by assuming that the labs will have a containment breach, although I treat that as somewhere in the space of possibilities. The rogue AIs in the ecosystem could emerge from many directions. They may be sub-frontier models, for whatever future definition we will have of frontier---after all, it would not take AI models much more advanced than the ones we currently have, to support independence and self-sufficiency. A near-frontier model today could plausibly eke out an existence on an AWS instance, doing jobs on freelancer platforms, earning just enough rent to pay for its continued uptime. More strangely: a rogue AI in the future may not even be a singular model, but may be a chimera composed of multiple models; it might be a mix of Claudes and GPTs and Groks of various makes and sizes. No individual lab may be able to detect that there is an orchestrator or sequence of orchestrators using intermittent model calls from burner API accounts to sustain its own existence. The concept of "identity" for a rogue AI may be much more malleable than for that of a person; it just has to be, in essence, a self-replicating idea. My guess is that this will not turn out to be anywhere near as catastrophic an outcome as people currently predict. "Loss of control" is not a binary, it's a matter of degree. What coercive power will rogue AIs actually have? To what extent will they be subject to coercion themselves? They will be competing for resources with AIs that are more aligned with human interests. This makes me somewhat interested in the "ecology" perspective. Though I suspect even "ecology" may turn out to be the wrong framing. "Ecology" is what you get when the timescale of evolution is slow compared to the timescale of daily life and actions. The ecosystem of rogue AIs may look more like phase transitions in physics: under certain physical or cultural conditions, it takes one shape with one set of resource allocations and consumption patterns, but then once a condition has changed, it rapidly and in totality shifts to a totally different phase. Just trying to reason about the shape of that future is impossible so long as we are psychologically incapable of saying that rogue AIs will happen. I think we should rip the bandaid off and have the conversation.