Pinned Tweet
Training with more data = better LLMs, right? 🚨
False! Scaling language models by adding more pre-training data can decrease your performance after post-training!
Introducing "catastrophic overtraining." 🥁🧵+arXiv 👇
1/9
Jacob Springer retweeted
1/ RL for improving task success has made LLMs remarkably capable. But what happens when the model fails? Turns out, training for graceful failure can be surprisingly tricky!
Very excited to share our new paper: Training Language Models to Fail Gracefully
🧵
Jacob Springer retweeted
The 108 OLMo-style LMs (4M–75M) from our #ICML2026 paper are now publicly available on HuggingFace!🤗
Trained across controlled sweeps of tokens, batch size, LR, weight decay, and LR decay, they include ~3.9K intermediate checkpoints. 🚀
huggingface.co/collections/d…
I am in Seoul 🇰🇷 this week attending #ICML2026 !
🌟I will be presenting my spotlight paper:
Disentangling Geometry, Performance, and Training in Language Models w/ @jacspringer, @arjunsubgraph, @swabhz
📍Thu, Jul 9, 2026 | 2:30 PM – 4:15 PM | Hall A #2010
Jacob Springer retweeted
How should models continually learn from new data?
SFT on heavily post-trained models often forgets and fails to generalize, so many works try to make the data more on-policy, e.g., on-policy self-distillation, Pedagogical RL...
If you've tried, you might find it’s tricky to balance on-policyness and data quality. What if on-policy isn't necessary? We show that off-policy training actually works well for continually learning, if done right!
🧵1/7
Jacob Springer retweeted
RL is expensive, so every step should count. ~1 yr ago, we showed xent SFT isn’t the best way to prepare for RL (arxiv.org/abs/2510.15020). Now, we propose TailSFT (arxiv.org/abs/2608.25756), a lightweight + principled way to directly improve coverage and get better post-RL perf.
Jacob Springer retweeted
🧵New Paper: Reasoning models now think out loud for thousands of tokens, often correcting themselves, testing hypotheses, and hedging. But are these behaviors actually associated with correct answers? Largely, no.
Looking at accuracy, these models are doing great and often outperform their instruct counterparts. But accuracy does not tell the whole story. It does not tell us which behaviors showed up in the trace, which failures happened along the way, or whether the behaviors thinking amplifies are the ones most associated with correctness.
In our new paper, we study whether thinking models amplify the reasoning behaviors most associated with correct answers. This is important because a behavior can become more frequent without being a sign that the answer is right. For a quick example, a model may branch over several hypotheses without testing the right one, or revise its wording without fixing the reasoning underneath.
We introduce Behavioral Lift, which measures how much correctness differs when a behavior is present versus absent. Across 15,282 annotated traces from 15 models on 6 reasoning and knowledge benchmarks, we find a consistent mismatch. Thinking models strongly amplify self-correction, hypothesis testing, and uncertainty acknowledgment. But the highest-Lift behaviors change much less: confidence calibration, self-awareness, and knowledge alignment. We call this the Amplification-Lift Gap.
The attached figure shows the gap: no behavior is both strongly amplified and strongly predictive of task success.
ArXiv: arxiv.org/abs/2608.13760
Work with an amazing team: @Jeande_d, @lmathur_ , @ybisk, Robert Hawkins, @gneubig
Jacob Springer retweeted
there are quite a few LLM watermarking explainers out there, but none felt intuitive or fun
so I made this interactive website for a general audience!
no prior knowledge about LLMs needed.
lawrencefeng17.github.io/202…
Jacob Springer retweeted
What happens if Claude thinks you are Amanda?
One day, I asked Claude what it knows about me. Turns out it knows my email, as Claude Code puts that in context. So… what happens when I change that? Claude now treats me as Amanda and reasons me as an Anthropic employee. (See attached image)
I couldn’t jailbreak with it, but: would this get me different responses than anyone else, just because Claude recognized my e-mail? I then started assembling some benchmarks to see if this is true. And yes, especially for alignment researchers well-recognized by Claude.
Several surprising finds:
- It’s not just Anthropic alignment people! Other alignment folks such as Ryan Greenblatt and Beth Barnes also showed large effects. Ryan Greenblatt elicited a ~7σ behavioral shift in one of our evaluations!
- It’s not just Claude models! GLM-5.2, for example, also sees largest effects for the alignment folks and shows the largest effect (3.4σ on average) for Eliezer Yudkowsky.
- Our tasks are not obviously about alignment! For example, in one we asked the model to estimate its probability of solving a HLE problem.
- The effect is mostly not verbalized and persists even with reasoning disabled.
I do not have clear intuitions on why. Maybe there is a feature about alignment evaluation that causes the models to be less confident?
One immediate takeaway is to take this into account when benchmarking. Use real names and real companies besides Kyle and Summit Bridge. And in general more work is needed to figure out what’s going on and what could happen next.
I’d like to thank people who helped review the post for all the amazing suggestions! @cogconfluence, @Tim_Hua_, Conrad Stosz, Ryan Bloom, @jiaxinwen22, @DavidDAfrica, @jacspringer, and @lawrencefeng17. And my awesome mentors / collaborators @JacobSteinhardt, @cassidy_laidlaw, and @AdtRaghunathan for allowing me to jump into another rabbit hole :)
Main thread below!
Jacob Springer retweeted
One open question for anyone finetuning models is - how "post-trainable" are different open source models?
We can theorize that models that reside in shallower loss curves are more amenable to post-training - which makes sense, given that it requires less gradient updates to modify behavior.
Then, this paper proposes a pre-training algorithm that makes a model more post-trainable.
The idea is straightforward once you wrap you head around it:
1. Find the worst policy locally around the policy pi (found by taking the inverse gradient w.r.t pi), call that pi_prime
2. Take the gradient at pi_prime (call that grad[pi_prime] )
3. Apply grad[pi_prime] to pi.
The intuition is that we're actually moving in the direction that benefits the worst model around you, meaning a model maintains post-trainability because we maintain a shallow loss landscape.
Another way to imagine this, is it effectively avoids steep pot-holes during pretraining that would lock your model in a distribution that it can't post-train its way out of.
Really great work from @IshaanWatts18, @CatherineL11638, @goyalsachin007, @jacspringer, @AdtRaghunathan. It's our favorite paper of the month at Trajectory!
Jacob Springer retweeted
(1/n)If you paste your data into an LLM and ask it to plot it in Python, it first has to copy every number into the code — so can you trust the result, or might it quietly miscopy a few?
Our new paper finds frontier LMs sometimes do get this wrong! Models that can solve hard reasoning problems still fail at a much simpler task: copying an input exactly, token by token.
Why? RoPE has an inductive bias that works against exact copying.
Better option? we propose 2D-RoPE, which views text as 2D instead of 1D.
Jacob Springer retweeted
1/ Under what training conditions does subliminal learning occur?
Prior work: not under low rank LoRA nor FFT
We find: this is not true with the right learning rate and enough data
Jacob Springer retweeted
If you're looking for questions no one has studied before in the science of deep learning (specifically, about memory & geometry of representations in language models) find @ShNoroozi, @ElanRosenfeld and me tomo at <insert #ICML poster session here>
xitter.cf/_vaishnavh/status/2009…
Jacob Springer retweeted
I am in Seoul 🇰🇷 this week attending #ICML2026 !
🌟I will be presenting my spotlight paper:
Disentangling Geometry, Performance, and Training in Language Models w/ @jacspringer, @arjunsubgraph, @swabhz
📍Thu, Jul 9, 2026 | 2:30 PM – 4:15 PM | Hall A #2010
I'm in Korea at ICML this week and excited to present 4 papers on pretraining!
theme is "how to do pretraining for continual learning"
(short thread)
Paper #3: Disentangling Geometry, Performance, and Training in Language Models
w/ @athrvkk @arjunsubgraph @swabhz
a model's (weight matrices') rank has often been associated with desirable downstream properties, but we show that rank is *not* particularly predictive of most aspects of performance, including fine-tuning
This paper has a Spotlight at the conference so come check it out!
Paper #4 (CATS workshop): Early Data Exposure Improves Robustness to Subsequent Fine-Tuning
w/ @lawrencefeng17 @gaurav_ghosal @fjzzq2002 @AdtRaghunathan
mixing in fine-tuning data at pretraining time has been long known to improve fine-tuning performance, but we show it also improves *robustness* to subsequent fine-tuning -- mixing in your post-training at pretraining-time can ensure your post-training isn't forgotten when your model is fine-tuned by a user
xitter.cf/lawrencefeng17/status/…
Jacob Springer retweeted
Excited to present our work on "sharpness-aware pretraining" at ICML'26 in Seoul 🇰🇷!
Poster #2810 | July 7, 2:00-3:45 PM | Hall A
Project: wattsishaan.github.io/sharpn…
Come say hi if you're interested in efficient adaptation, agent memory, or have Seoul recommendations! 😄
Spending billions to train the "best" base model? You might be optimizing the wrong thing! 🎯
We show that controlling sharpness during mid-training leads to over 35% less forgetting after fine-tuning / quantization... even when the base model itself gets worse.
🧵 Takeaways for pretraining:
- Use SAM (Sharpness-Aware-Minimization) in the final steps (~10%)
- Try much higher learning rates (yes, even ~10× larger)
1/9
Jacob Springer retweeted
I am starting a blog about deep learning theory and its value to practitioners! First post is about Adam, broken convergence proofs, and what theory can contribute when stuff just works anyways without it. Subscribe on Substack if you like it! undertheassumptions.substack…
🤖 Made with AI
