@dylanjsam

alignment pretraining @openai | formerly phd @mldcmu, @BrownUniversity

 San Francisco
dsam99.github.io/
Joined October 2017
  • Tweets214
  • Following527
  • Followers1.2K
31Media
Dylan Sam retweeted
Alex Robey
@AlexRobey23Sep 1
We're hiring safety researchers at @thinkymachines! 🦺 My team works on safety across the model development stack -- pre-training data filtering, harmful capability evals, safety post-training, red-teaming, abliteration/malicious fine-tuning. We're particularly interested in building evals and tooling to make strong safety cases for open-weights releases. We think this is where some of the most important safety research will happen in the next few years. If any of this resonates, apply here: jobs.ashbyhq.com/ThinkingMac…
16
52
6
581
88,465
Dylan Sam retweeted
Emily Byun
@yewonbyun_Jul 9
A lot of excitement in synthetic data for science is motivated by ambitious uses of LLMs (e.g., running a survey with GPT before real participants, forecasting promising experiments with LLMs). We ask, what beliefs should the experimenter have to make this a good idea? 🧵
3
26
3
59
12,603
Dylan Sam retweeted
Emily Byun
@yewonbyun_Jul 9
9/ This work was done with the amazing @brwilder. I will be presenting this work today at #ICML 🇰🇷 (Thursday 5:00-6:45 PM KST) in Hall A #3605. Feel free to stop by or reach out to chat! Would love to meet old/new friends! :) Paper Link: arxiv.org/abs/2607.06656

Robust Human-AI Complementarity under Uncertainty

Machine learning models are often intended to augment rather than replace human decision makers, by providing information that is complementary to human judgement. Yet, in practice, human...

arxiv.org
3
14
740
Really excited to see work developing understanding of when we are guaranteed to benefit from LLMs and synthetic data!! 📈
Emily Byun
@yewonbyun_Jul 9
A lot of excitement in synthetic data for science is motivated by ambitious uses of LLMs (e.g., running a survey with GPT before real participants, forecasting promising experiments with LLMs). We ask, what beliefs should the experimenter have to make this a good idea? 🧵
11
2,196
Hellooo! I’ll be at @icmlconf with @OpenAI! If you’re interested in safety research at OAI generally or pretraining safety research I’d love to chat! You can find me hosting a Q&A at our booth Wednesday morning at 9:30am!
6
8
294
22,744
Dylan Sam retweeted
Bryan Wilder
@brwilderJun 4
Thanks for having me! I talked about our work on valid inference with synthetic data (arxiv.org/abs/2508.06635) and robust human-AI complementarity (ICML 2026, paper up soon), both with my PhD student @yewonbyun_

Valid Inference with Imperfect Synthetic Data

Predictions and generations from large language models are increasingly being explored as an aid in limited data regimes, such as in computational social science and human subjects research. While...

arxiv.org
Despoina Paschalidou@paschalidoud_1Jun 4
Exploring the Next Generation of Data Workshop @CVPR is happening now at Room 603. @brwilder is talking about the science with synthetic data.
1
2
6
750
Dylan Sam retweeted
Tomek Korbak
@tomekkorbakMar 30
New OpenAI post: Can midtraining on docs about aligned AI bake in alignment priors for agents? We report an experiment where those priors are quickly washed away by RL and fail to generalize to agentic settings. But that cuts both ways: priors that AIs are misaligned fade too!
7
39
9
218
36,006
Dylan Sam retweeted
Jason Wolfe
@w01feMar 25
I'm also extremely excited for our companion post today on Model Spec Evals! Spec Evals are a new way we're measuring progress towards alignment with the Model Spec — including public results, an open dataset, and code others can build on. alignment.openai.com/model-s…
7
9
1
37
4,377
Dylan Sam retweeted
Zico Kolter
@zicokolterMar 20
As AI agents access more untrusted information with greater autonomy, prompt injections may become the greatest security challenge of our era. @GraySwanAI, in collaboration many frontier labs, just released our paper on the largest public prompt injection challenge to date. 🧵
Gray Swan AI
@GraySwanAIMar 20
Your AI agent can be hijacked by a prompt injection and you'd never know! The attack executes. The response looks normal. And the user moves on. We ran the largest public competition testing this exact threat across tool use, coding, and computer use agents. 464 participants, 272K attacks, 13 frontier models. Every model proved vulnerable.
6
9
65
14,920

How should AI systems be evaluated for health?

Article · @thekaransinghal
35
59
22
301
65,836
I defended my PhD thesis! Also, a very (~4 month) late life update, but I've joined @OpenAI to work on safety research and pretraining safer language models! 📈 Thank you to my advisor @zicokolter and my committee: Matt Fredrikson, @andrew_ilyas, and @furongh! 🙏
25
9
1
217
22,323
Dylan Sam retweeted
Marc Finzi
@m_finziJan 7
1/🧵 We are very excited to release our new paper! From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence arxiv.org/abs/2601.03220 with amazing team @ShikaiQiu @yidingjiang @Pavel_Izmailov @zicokolter @andrewgwils
56
400
92
2,440
1,142,246
I'm at NeurIPS this week! Excited to meet old/new friends and chat with people about training safer language models. I'm presenting a few works on safety pretraining, measuring diversity in data curation, and monitoring model behaviors --- more info below 👇
5
4
38
4,310
more replies
Next, I'm presenting on safety pretraining, where we find that incorporating safety behaviors during pretraining leads to more robust language models! Come by Poster #5210 at Exhibit Hall C,D,E at 4:30pm today (12/4)! xitter.cf/dylanjsam/status/19679…
Dylan Sam
@dylanjsam16 Sep 2025
🚨Excited to introduce a major development in building safer language models: Safety Pretraining! Instead of post-hoc alignment, we take a step back and embed safety directly into pretraining. 🧵(1/n)
1
3
629
Finally, I'm presenting work on monitoring models for harmful behaviors, hallucinations, and adversarial manipulation at Poster #1304 in Exhibit Hall C,D,E on 12/5 at 4:30pm! xitter.cf/dylanjsam/status/18799…
Dylan Sam
@dylanjsam16 Jan 2025
To trust LLMs in deployment (e.g., agentic frameworks or for generating synthetic data), we should predict how well they will perform. Our paper shows that we can do this by simply asking black-box models multiple follow-up questions! w/ @m_finzi and @zicokolter 1/ 🧵
2
450
I’m at NeurIPS this week (12/2-12/8) to present our work on when/how synthetic data (e.g., LLM simulations) can help scientists make inferences with less real data, improving the efficiency of costly experiments. Come by Poster #904 on Thursday 4:30PM (Exhibit Hall C,D,E)!🙂
Emily Byun
@yewonbyun_9 Oct 2025
💡Can we trust synthetic data for statistical inference? We show that synthetic data (e.g. LLM simulations) can significantly improve the performance of inference tasks. The key intuition lies in the interactions between the moments of synthetic data and those of real data
2
4
3
32
13,630
Excited about our NeurIPS'25 tutorial Data Privacy, Memorization & Copyright in GenAI with Cooper (co-founder, GenLaw) & Joe (represents OpenAI, Stability in all US copyright litigations) We bring together ML researchers, with those who understand its legal implications. Pls RT
3
21
3
82
12,876
I gave talks at MIT and Harvard this week about "Science with synthetic data". How can generative models help us learn about the world (e.g., social systems) in a principled way? Lots of interesting conversations; more convinced than ever that there's nuanced issues to navigate
1
2
9
654
📢 Multi-token prediction has long struggled with defining the right “auxiliary target,” leading to tons of heuristics. We show a core limitation of these and propose a simple & sweet idea: future summary prediction. Introducing what I call 🚀TL;DR token pretraining🚀
Divyat Mahajan
@divyat0929 Oct 2025
[1/9] While pretraining data might be hitting a wall, novel methods for modeling it are just getting started! We introduce future summary prediction (FSP), where the model predicts future sequence embeddings to reduce teacher forcing & shortcut learning. 📌Predict a learned embedding of the future sequence, not the tokens themselves
4
36
1
241
28,934
Load more