@redwood_ai

Pioneering threat mitigation and assessment for AI agents.

 Berkeley, CA
redwoodresearch.org/
Joined December 2024
  • Tweets46
  • Following6
  • Followers5K
11Media
Several staff from Redwood have been subcontracted by METR to work on this investigation. We believe that independent investigation is crucial for understanding and managing misalignment risk. This project is an important step in that direction; we're excited to work on it.
METR
@METR_EvalsSep 9
We have reached an agreement with Anthropic to conduct an independent investigation of agent incidents at the company and of their models’ alignment properties. We will publish one or more reports that will share our findings and describe our terms of engagement.
13
20
12
577
106,346
Some architectures could weaken CoT monitorability, or remove the CoT altogether. We've written a proposal for how companies could be transparent about no-CoT reasoning abilities, other monitorability evidence, and policies for preserving monitorability. redwoodresearch.org/blog/pro…

Proposal for tracking the effects of architecture on monitorability — Redwood Research

Architectures that incorporate opaque recurrence or allow agents to communicate using latents could rapidly make it much harder to monitor chains of thought. We propose that AI companies regularly...

redwoodresearch.org
4
45
15
309
69,011
Redwood Research’s chief scientist Ryan Greenblatt was the primary empirical researcher on this investigation. The report contains many important novel details that we hope will improve understanding of this incident. We appreciate the access OpenAI provided here.
METR
@METR_EvalsAug 26
METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs.
8
10
1
143
10,256
Redwood Research retweeted
Ryan Greenblatt
@RyanGreenblattJul 23
We (@bshlgrs and I) recorded a podcast about the OpenAI / Hugging Face incident. We discuss: - What we actually know. - How surprising the incident was. - What the incident does (and doesn't) tell us about misalignment risk. - Why control measures didn't catch or prevent this.
8
25
6
197
15,883
Redwood Research retweeted
Buck Shlegeris@bshlgrsJul 23
I've seen a lot of IMO confused discourse about whether the misalignment observed in the OpenAI/Hugging Face incident is scary. In particular, the models are clearly not misaligned schemers that lie in wait. Girish and @alextmallen discuss how scary this kind of misalignment is.
Girish Gupta@jammastergirishJul 23
AI models created by OpenAI escaped their sandbox and, working autonomously, hacked into leading AI model and data hub Hugging Face. The incident is an in-the-wild demonstration of the dangers of rogue AI — no longer a science-fiction fantasy.
2
8
103
10,379
"We find that frontier models like GPT-5.5 answer questions that take humans roughly three minutes with 50% reliability, and this time horizon has doubled approximately every year since 2019."
Dewi Gould@dswg97Jun 10
New paper! Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models @METR_Evals showed that models' time horizons have doubled every few months. We ask: what length of tasks can models complete without any CoT?
1
18
6,257
Some coding scaffolds block and retry risky actions. In a new paper, we find this reveals information a malicious AI can use to bypass monitoring. Resampling without blocked actions in context is less exploitable, but techniques that help in one setting can hurt in another. 🧵
2
6
1
37
6,560
more replies
In fact, in our new setting using the minimum suspicion dramatically lowers safety as you resample more. While most of the dynamics line up with the findings from Ctrl-Z, naively applying the best techniques from Ctrl-Z to our new setting would massively decrease safety.
1
4
304
Retrying and resampling will likely stick around in coding scaffolds. But to provide protection against a strategic adversary they need careful red teaming. Small differences in protocol or setting can dramatically affect optimal red/blue team choices and equilibrium safety.
4
223
Load more