PhD candidate @UMichCSE | prev. pretraining research intern @IFM_AI; undergrad @IITKanpur
Ann Arbor, MI
Joined April 2023
- Tweets15
- Following1K
- Followers114
- Likes742
Pulkit Gopalani retweeted
Excited about this new work showing emergent capabilities follow from discovery of attention patterns!
Scaling laws predict an LLM's pretraining loss, but not its capabilities. Abilities like in-context learning emerge abruptly and only past a certain scale. Our new paper traces this to one bottleneck: learning which tokens attention should focus on. 🧵arxiv.org/abs/2606.25010
Pulkit Gopalani retweeted
Scaling laws predict an LLM's pretraining loss, but not its capabilities. Abilities like in-context learning emerge abruptly and only past a certain scale. Our new paper traces this to one bottleneck: learning which tokens attention should focus on. 🧵arxiv.org/abs/2606.25010
Pulkit Gopalani retweeted
What drives in-context learning in LLMs?
New paper: Provable Low-Frequency Bias of In-Context Learning of Representations.
We show LLMs have a low-frequency bias when learning representations in context, offering a theoretical answer to several previously open questions. 🧵👇
Excited to announce our recent work on understanding training-time emergence in Transformers!
Thread🧵(1/11)
Repetitive sequences are easy for Transformers: we show that training on sequences like ‘x_1, x_2, …, x_n, [sep] x_1, x_1, …, x_1’ (or other similar sequences) does not involve loss plateaus like other algorithmic tasks, and the loss converges in a few training steps. (10/11)
1
1,264
Check out our paper for details and results on other algorithmic tasks / model configurations: arxiv.org/abs/2506.13688
GitHub repository: github.com/pulkitgopalani/tf…
(11/11)
1
766