Making LLMs safe & secure @openai | Previously: PhD @berkeley_ai
Joined March 2022
- Tweets111
- Following387
- Followers313
- Likes147
Sam Toyer retweeted
Instruction Hierarchy defines how LLMs prioritize conflicting instructions. Our IH RL training dataset can makes models more robust to prompt injections, IH attacks, and better follow in-context safety specs while maintaining capabilities and helpfulness 🧵cdn.openai.com/pdf/14e541fa-…
In the world of agents, understanding the computational structure of the problem that you're trying to solve is an essential part of "understanding the data". Without that understanding it is difficult to extrapolate eval results to more complex, realistic tasks.
Sam Toyer retweeted
New preprint! We build on the hypothesis that language models construct latent world models of their inputs, and seek to extract latent world states as logical propositions using “propositional probes”.
Sam Toyer retweeted
New paper! We introduce Covert Malicious Finetuning (CMFT), a method for jailbreaking language models via fine-tuning that avoids detection. We use our method to covertly jailbreak GPT-4 via the OpenAI finetuning API.
Sam Toyer retweeted
Excited to share a unifying formalism for the main problem I’ve tackled since starting my PhD! 🎉
Current AI Alignment techniques ignore the fact that human preferences/values can change. What would it take to account for this? 🤔
A thread 🧵⬇️
Sam Toyer retweeted
♟️Do chess-playing neural nets rely purely on simple heuristics? Or do they implement algorithms involving *look-ahead* in a single forward pass?
We find clear evidence of 2-turn look-ahead in a chess-playing network, using techniques from mechanistic interpretability! 🧵
Sam Toyer retweeted
My first PhD paper!🎉We learn *diffusion* models for code generation that learn to directly *edit* syntax trees of programs. The result is a system that can incrementally write code, see the execution output, and debug it. 🧵1/n
Sam Toyer retweeted
Breaking news — gpt2-chatbots result is now out!
gpt2-chatbots have just surged to the top, surpassing all the models by a significant gap (~50 Elo). It has become the strongest model ever in the Arena!
With improvement across all boards, especially reasoning & coding capabilities, we're excited to see what app can build on top.
Huge congrats to @OpenAI for this incredible milestone!
Note: this is an internal screenshot. Its public version "gpt-4o" is now in Arena and will soon appear on the public leaderboard!
The deep RL world focuses too much on algorithms and not enough on understanding its own benchmarks. What makes an algorithm do well or poorly on a given environment? Is it actually measuring what we want to measure? Cassidy's work does a great job of answering these questions.
Last year we showed that deep RL performance in many *deterministic* environments can be explained by a property we call the effective horizon. In a new paper to be presented at @iclr_conf we show that the same property explains deep RL in *stochastic* environments as well! 🧵
Sam Toyer retweeted
Introducing the Instruction Hierarchy, our latest safety research to advance robustness for prompt injections and other ways of tricking LLMs into executing unsafe actions. More details: arxiv.org/abs/2404.13208
Sam Toyer retweeted
Recent research justifies a concern that AI could escape our control and cause human extinction. Very advanced long-term planning agents, if they're ever made, are a particularly concerning kind of future AI. Our paper on what governments should do just came out in Science.🧵
This looks very relevant for tool use, multi-turn dialog, and other settings where an LLM has to interact with people or the world. Learned reward functions are fragile, so you need strong regularization in RLHF—perhaps this will replace the standard KL penalty in those settings?
RLHF requires a KL penalty that keeps the action (token) distributions close to the SFT model; without it we get reward hacking. In new work with @ShivamSinghal56 we find in general it's better to use *occupancy measure* (state+action distribution) KL to avoid reward hacking.
🧵
Sam Toyer retweeted
Are you looking to hire world-class AI talent? Please take a look at some of our recent graduates from Berkeley AI @berkeley_ai! Their work has shaped the entire field of AI as we know it today: bair.berkeley.edu/blog/2024/…
Sam Toyer retweeted
How are open LLMs trained and created in 2024? 🤔 @01AI_Yi just released their paper on how they created the YI, a family of LLMs and V-LLMs. The paper includes details on the data processing, training and multimodality part. Let's take a look👀
First Data: How does their data pipeline look?
🌐 Start with web documents from Common Crawl and use the CCNet pipeline for language identification and perplexity scoring.
⚖️ Apply Heuristic rules to remove low-quality text based on URL, domain, word blocklists, garbled text, document length, special symbol ratio, short/consecutive/incomplete lines, and repeated words/n-grams/paragraphs.
📋 Use learned filters with perplexity scorer (KenLM), quality scorer (Wikipedia classifier), safety scorer, and document coherence scorer to identify and remove low-quality content.
🧩 Apply Clustering to group and analyze documents, removing low-quality ones.
📚 Run Deduplication with document-level MinHash deduplication and sub-document exact-match deduplication.
📙 Categorize web documents into specific themes using a topic model and down-sample less helpful content in the final pretraining dataset, e.g. advertisement
Sam Toyer retweeted
New blog post: Prompt injection and jailbreaking are not the same thing
simonwillison.net/2024/Mar/5…
Great paper: sites.google.com/view/compro…
I think that systemic risk from large networks of "dumb" AI systems talking to each other is underrated. This type of attack could kill services like Adept or the Rabbit R1, and will only get worse as prompt malware gets more sophisticated.
In some ways this is more worrying than traditional malware: LLMs can quickly to adapt to unexpected situations through in-context learning, and an ensemble of compromised LLMs could be very good at coming up with clever ways to spread itself and exploit compromised services.