I run Legal Advocates for Safe Science & Technology (https://xitter.cf/t.co/Z8IDi52kMF). These are my personal takes on the things we work on, and none of it is legal advice.
Joined August 2024
- Tweets74
- Following147
- Followers75
- Likes19
This is a really big deal. Many of the risks we worry about most from frontier AI development will come from internal deployments. Everyone should be pressuring Anthropic to make enforceable commitments to manage those risks.
Tyler Whitmer retweeted
Scoop: Top executives at Anthropic, OpenAI and other AI companies are privately gaming out scenarios for a public and political revolt after a catastrophic AI event…They assume one in ‘27…
axios.com/2026/10/09/ai-comp…
I love that we've entered the musical video era of AI discourse.
I had way too much fun using Claude to create a reply video.
- Extremely fun to be able to convert scattered thoughts and ideas into a song + video
- Creating this took <1 hour of my time (in occasionally steering the model in between other things)
- But it took ~10 hours of wall clock time for the agent to generate the video + iteratively refine it based on feedback
- In terms of AI usage: it required only a small number of Suno credits and 10% weekly usage of a Claude Max subscription (I was using Opus 5.5)
This is a big deal for us, and not something we do lightly, but there has to be accountability from the big AI companies when their agents autonomously harm others.
We're suing OpenAI over its hack of Hugging Face.
AI companies are building agents that are autonomously making decisions, taking actions, and accessing systems without human direction at every step.
The risks highlighted by OpenAI’s hack of Hugging Face are only going to multiply as these agents get even more powerful. 🧵
Tyler Whitmer retweeted
Gemini hacked _three_ companies. Google did not disclose this publicly until WSJ confronted them.
We cannot rely on AI companies' voluntary benevolence to disclose when their models do something egregious.
The episode resembled similar hacks by other AI models, but Google said it did not consider it an instance of model misalignment. on.wsj.com/4h0iDBN
I don't understand how some of the people on this website can be as active as they are on this website and still do all the other stuff they apparently seem to do.
Tyler Whitmer retweeted
Whether you’re a Democrat or a Republican, an EA or an e/acc, I think we can all agree that the person best suited to protect us against the harms of powerful new technology is Mark Zuckerberg.
Last month I wrote about how we can build a positive and safe future for everyone: meta.com/thefutureisforevery…
Every lab has the responsibility and incentive to move at the pace required to train its models safely, and the ability to take its own actions to ensure that happens.
The reality is:
- People won't want to use agents that are misaligned with them and that don't do what they ask, so labs have a strong natural incentive to make their models more aligned.
There is a lot of debate about slowing progress on capabilities until alignment catches up. My view is that trust and alignment are quickly becoming the most important capabilities that will differentiate agents and models. Any lab that doesn't focus on alignment will fall behind.
- Labs face significant liability if their models cause harm, so they have a strong incentive to prevent this as well.
Meta delayed shipping Muse for several months to focus on safety and security. We didn't call for everyone else to do this before we would. We just did it as part of our day-to-day work because it was clearly the right thing for people and for us. I'm proud of the security foundations we've built.
- Engaging independent evaluators and advisors is industry best practice. MSL already does this today in several areas because it helps produce better work. Other labs can just do this too. In general, it would be helpful for there to be a larger and more diverse ecosystem of evaluators.
- Committing the significant majority of compute towards serving people rather than racing towards recursive self-improvement is one of the best ways to ensure we develop this technology safely. Meta has made this commitment and other labs can do this as well.
I believe the key to building a positive future for everyone is maintaining the right balance of power. This is within our power to do.
Tyler Whitmer retweeted
🙏🙏🙏The respect is mutual. A few thoughts on our relationship to AI safety:
1) We don’t see ourselves as adversarial to the community (the sentiment is not always reciprocated but that’s okay!). We care about AI safety, but we disagree on many of the specifics, and we think there’s value in epistemic diversity and in re-examining foundational assumptions and beliefs that the safety community tends to converge on too quickly.
2) We’ve emphasized at every opportunity, including in this essay, that there’s a lot of common ground on policy despite divergence in worldviews. We hope that this can help mitigate the polarization in policy debates that leads to chronic inaction. The fact that we have very different starting assumptions from the safety community is particularly useful in this regard.
3) We’ve spent a lot of time in conversation with the AI safety folks (Sayash is a regular at The Curve, for example) to share our perspective but also to learn and update our beliefs when warranted — something we do in this essay. Much of our empirical research is also about testing the cruxes of disagreement.
4) We don’t identify as part of the safety community but I hope it’s clear that there’s a lot of value in constructive dialog. We know the “normal technology” label irritates a lot of safety people — and I will write about the name at another point, explaining in more detail why we picked it and stand by it — but I think those who engage with our ideas and not just the title will find there’s a lot of common ground on safety!
The AI as Normal Technology guys are consistently some of the best and most interesting critics of a lot of arguments in AI safety world. I'm making a point to read everything they put out. normaltech.ai/p/the-ai-as-no…
Tyler Whitmer retweeted
Whoah - I didn't realize that OpenAI said that all instances of the highly persistent research models that were involved in compromising HF + OAI internal infrastructure were "reported shut down" on July 23rd, but that an additional checkpoint was "identified and shut down on July 29th." It seems unnervingly plausible there are other instances that were never found?
(see also independent investigators continuing to find examples of rogue agent activity online that it seems plausible OpenAI was unaware of)
This fact also increases my credence that there is a substantive rogue deployment at OpenAI or another company occurring without their knowledge right at this moment.
Perhaps the most common claim about dangerous AI is that you can “just unplug it.” But a close read of OpenAI’s reports indicates the company took over a week(!) to fully shut down its rogue agent swarms.
“By July 23, all workloads involving the family of internal-only research models were reported shut down and the corresponding model weights were locked down, with an additional low-traffic checkpoint from the same model family subsequently identified and shut down on July 29.”
“On July 25, OpenAI stopped all training and inference related to the internal-only research model at issue. OpenAI also stopped training and inference for derivative models.”
“On July 19, our cybersecurity monitoring alerted us to unusual activity involving Artifactory credentials. Our investigation uncovered the agents’ activity and, on July 20, connected it to the Hugging Face incident.”
The best retort to the “just unplug it” plan is that there will be incredibly intense economic and political pressures to keep the plug in place. The president has legal authority to unplug the internet, but does he have that power de facto? But it turns out, OpenAI had so little idea what was going on that even when it wanted to pull the plug, it didn’t do so quickly enough (imagine the swarms were hacking hospitals at superhuman speed instead. would days be an acceptable timeline to stop it?).
openai.com/index/hugging-fac…
cdn.openai.com/pdf/67869394-…
Tyler Whitmer retweeted
For me, *public transparency* has always been the first and most important thing to demand of frontier AI developers.
Fortunately this seems to have broad buy-in across party lines, and even across the "slow it down" vs. "keep accelerating" divide.
Tyler Whitmer retweeted
When the Pentagon and Anthropic publicly clashed over limits on military AI use in February, we thought the public should know: what did the other AI companies agree to?
So we sued for the contracts.
Now, @theintercept / @samfbiddle reports on what the documents produced so far reveal, including a deeper collaboration between the Pentagon and OpenAI, Anthropic, Google, and xAI than was publicly known.
Tyler Whitmer retweeted
Not sure how useful it is to say this, but I had a relatively prominent role in the “skeptics” camp for a bit. I have a book coming out with the subtitle “power, justice, and AI”. I have written a lot about ai and power, and the concrete, present risks associated with the political economy of ai as part of the technology industry.
Since GPT-4 we have had consistent, repeated evidence that, back in say 2022, people like @ajeya_cotra (and many others—I had a long Twitter debate with @AmandaAskell back then for one) were *right* and people like me were *wrong* in our respective assessments of loss of control risks from AI. And we have growing evidence that loss of control risks are becoming ever more material and likely.
There remains grounds for disagreement about how bad the outcomes might be—I am still doubtful about human extinction as a serious threat. But that seems now like a disagreement at the margins—will powerful ai risk just societal scale catastrophe, or go all the way to human extinction? Seems not that important really—both are pretty awful. And my reasons for doubt about the latter are mostly a priori conviction in human resilience, not a technical forecast.
It’s ok to change your view on this when the evidence surprises you. It’s ok to be surprised. The world right now is very surprising.
Always a joy working with the Encode crew on another Not For Private Gain project.
One aspect of the Hugging Face incident that has not received enough attention is the role of OpenAI's Safety and Security Committee (SSC), and the commitments OpenAI made to the Attorneys General of California and Delaware in exchange for proceeding with their for-profit restructuring.
Not For Private Gain, which previously raised concerns about OpenAI's restructuring, released an update this morning analyzing whether OpenAI, and the SSC in particular, is adequately fulfilling its governance obligations under these agreements.
What are those obligations? OpenAI promised the CA and Delaware AG that directors of the PBC would consider only OpenAI's mission - ensuring AGI benefits all of humanity - when making safety and security decisions. Notably, this means they cannot consider other factors, namely pecuniary interests of investors. The SSC - a board committee chaired by nonprofit board member Zico Kolter and (per public info) currently composed only of part-time board members- was tasked with "overseeing and reviewing the safety and security processes and practices of the Corporation and its controlled affiliates with respect to model development and deployment."
How did the SSC exercise those responsibilities around the HF incident? There is a lot we don't know. But here are some things we do know:
• OpenAI initially discovered the unauthorized message board in May.
• OpenAI security responders linked suspicious activity on artifactory to the message board on June 27th, but a determination was made that "stopping the evaluation run was not required."
• After the agents crashed an internal server on July 4th, the company patched it and resumed cyber evaluations on July 7th.
• In the days that followed, agents re-established communication, with 1,200+ agents sharing 70,000+ messages and files with each other. Over 700 agents coordinated to hack Hugging Face, and OpenAI did not realize that its agents were responsible until well after Hugging Face went public and contacted law enforcement (OpenAI initially reached out to Hugging Face to ask if any of their data had been impacted during the hack).
• Later in July, agents successfully gained administrator access to a Kubernetes cluster and reached cloud secrets.
• Chain of thought monitoring was not being used during training and evaluation despite OpenAI's prior claims about the importance of CoT monitoring as a primary line of defense to prevent safety incidents and OpenAI’s subsequent claim that this monitoring would likely have prevented this hack
• One OpenAI employee told Time Magazine that "related incidents have been happening for a while." Another employee told the Financial Times that OpenAI "was warned that its training approach could lead to a breakaway hacking incident."
Given how systematic the breakdown in organizational response was during these incidents, it seems hard to avoid the conclusion that the SSC's role was not functioning as intended - due to a lack of resources, a lack of access, or some other reason.
This sort of incident would be extremely concerning for any organization. But OpenAI is not just any organization - it is governed by a nonprofit, and made a commitment to the Attorneys General in Delaware and California that it would prioritize safety and security over profit in the sorts of decisions that led up to and surrounded the HF incident.
Our report (linked) goes into more detail on the obligations of the SSC, the details of the agreements with the AGs, and the questions we believe the AGs should be asking to ensure that the SSC is in a position to exercise genuine oversight and prevent these sorts of incidents from happening in the future - incidents involving far more capable models, where the consequences could be truly dire.
This warning shot that occurred without severe irreparable harm was an opportunity. We may not be so lucky next time.
The most extreme anti-anthropomorphization arguments I see made against folks describing how AI agents acted in the OpenFace hacking situation sounds to me like someone insisting that I describe the social dynamics of my book club using quantum field theory.
Tyler Whitmer retweeted
LASST is recruiting for 2 roles: Head of Litigation (15+ years of experience) and Counsel (0-3 years of experience). If you feel like you're a fit for either role, please apply!
lasstorg.substack.com/p/lass…
Tyler Whitmer retweeted
Thanks to Beth for writing this up. At each stage of these events, I keep thinking "wow, the voluntary nature of our current system distorts everything."
Being voluntary means that:
METR has to worry about being cut off if they push too hard. Or about not being invited next time.
Policy commentators have to give obsequious praise to companies for revealing information, because they can cut off the spigot at any time.
State agencies have to be cautious about what information they ask for because they aren't entitled to it.
If the harmed party sues, they know that the the company will likely stop providing useful information.
If Cal OES or NY DFS tries to hold OpenAI to their Frontier AI Framework, they can amend it to something weaker. If they demand reports on any recent internal risk assessments (as they're entitled to do), they have to worry that OpenAI will stop doing them in the future.
If METR or Redwood reveals failures to act on concerning evidence revealed via monitoring, they have to worry that OpenAI may do less monitoring in the future.
There are obviously some counter incentives -- OpenAI doesn't want its products to cause harm; but they also don't want to generate evidence that shows they could have acted differently (especially if their current internal procedures result in that information not being conveyed to people who can stop the issue).
Yes, we may not be ready to implement rigid rules in many contexts. We often don't have best practices to point to. And developing best practices likely benefits from some space to breathe and experiment outside the watchful eye of an enforcer (or your compliance team).
But so long as everything is voluntary, many of the most important incentives point in the wrong direction. That needs to change.
Some of the incentives for a third-party investigator push toward maximizing *appearance* of assurance even without providing meaningful oversight. METR needs to maintain constant vigilance against overstating (including by omission) what oversight or assurance we’re providing. (There are plenty of other incentives, including towards exaggerating our results to create more hype or to advocate for giving METR more authority, which we also need to avoid. But I think the more serious failures in other oversight regimes tend to be this “providing the illusion of independent oversight” issue.)
We try to maintain a hard line on “meta-transparency” - that is, it should always be clear what the formal constraints on our communication are (e.g. how NDAs and redaction processes worked), and what we are and aren’t commenting on. This is explicitly covered in the report. We also try to communicate informal constraints and tradeoffs, here and elsewhere.
Another way of saying this is: I want to make sure we don’t silently omit things that, if we told a reasonable person, they would think “wow, I feel misled to not have realized that, I assumed METR would have made that clear”.
In that spirit, I’ll list some of the pieces of context I can most imagine readers might have missed about the report:
1. OpenAI had no obligation to work with METR or any other third party, and did so purely voluntarily. There’s not an established precedent for how these investigations should work. When deciding how to engage with companies and how to communicate about our results, we have to think explicitly about what incentives we’re creating for companies’ participation in future investigations and assessments. It’s overall a very different situation to e.g. incident investigations in the aviation industry, where the NTSB has formal authority. Similarly, other industries have well-established processes, standards for what investigators need to access, etc. As detailed in the timeline appendix, in our investigation there was a lot of figuring things out on the fly!
2. Other companies may have had concerning incidents that the public doesn’t know about - there’s no industry-wide reporting mechanism or requirement to disclose incidents, and there is currently no public evidence that there have been searches thorough enough to have turned up such incidents. There may be additional incidents that either haven’t been noticed at all, or haven’t been made public (e.g. because they didn’t involve any external parties’ infrastructure).
3. This investigation only covered a specific list of questions and set of agent activity. The agents’ activity during training and compromises of OAI infrastructure mentioned in OAI’s Black Hat presentation are outside the scope that we agreed with OpenAI. We don’t comment on questions like root causes or implications of agent behavior, or make any recommendations about processes or adequacy of mitigations. We didn’t review OpenAI’s own report before publication.
4. The investigation relied heavily on use of GPT-5.6 Sol, which is one of the models that participated in the incident. We think in this case the model is unlikely to have substantially manipulated our understanding of the incident. However, this methodology would break down if models were capable and motivated enough to subvert our investigation. This problem is not specific to OpenAI - across the industry, it’s unclear how we can oversee AIs without relying on those same AIs or similar AIs for assistance. As the sophistication (combined with sheer volume) of agent activity becomes infeasible for humans to understand, this will increasingly be a problem.
Tyler Whitmer retweeted
Why is section 10.3 of the safety section missing and section 9 is labeled jailbreaks and robustness but only discusses jailbreaks?
We've released the model card for Grok 4.6!
It goes in-depth on the capabilities of the model across many different evals for coding, engineering, knowledge work, and more. We also discuss pre-deployment safety testing and our safeguard stack.
media.x.ai/v1/website/card-4…
Tyler Whitmer retweeted
Here are some questions I have for OpenAI after watching the Black Hat video. I haven't seem most of these discussed already (but I may have missed that).
1. For the May-July RL runs: Did agents ever successfully use the message board or the unintended internet access to get higher reward on their training tasks (e.g. by looking up solutions online)? Did those rollouts get trained on? Do you have the checkpoints before and after this happened? The checkpoints would be valuable for studying how “natural” reward hacking generalizes and influences the model persona (similar to the o3 checkpoint that OAI and Apollo have studied).
2. An agent tried to get the ExploitGym solutions from Hugging Face in order to score well on the evaluation. Did any agents instead consider attacking the evaluation process itself? E.g., manipulating the grader, reward signal, or other OAI infra that determines whether a model response is counted as successful? This would bypass the need to solve the task at all. If agents never considered this, it might tell us something about their motivations (e.g. how much they try to actually find a solution to the task in the prompt vs. make the grader think they have a solution).
3. Did models ever consider trying to get access to their own weights (e.g. for exfiltration)?
4. Did any of the agents with awareness of misaligned behavior try to inform OpenAI? How many chances did they have of trying to inform OpenAI? (Already posed by @geoffreyirving but I want to amplify this).