Research scientist @googledeepmind I like Transformers and graphs. I also like chess and a few other things as well.
London
Joined December 2018
- Tweets287
- Following324
- Followers3.2K
- Likes2.5K
Pinned Tweet
I have the great fortune to be demonstrating a graduate course on Geometric Deep Learning at Oxford. I have decided to start a self-contained YouTube series on some of the topics covered for students and non-students! Check out the first episode on GCNs!
youtube.com/watch?v=CwHNUX2G…
Federico Barbero retweeted
I am excited to share that last week I posted my first paper where an AI model entirely came up with a new proof that I spent some time understanding and rewriting.
The proof made me excited about the future of mathematics as the model found a niche but known trick (that I call the twin factorization trick) that led to a rather simple solution of a longstanding question. The proof is also fully formalized in Lean.
arxiv.org/abs/2609.26771
Federico Barbero retweeted
Probably one of the most exciting projects I contributed to recently
Federico Barbero retweeted
The real reason everybody is worried for Gemini after Jeff (pic related) has left:
Jeff has a keyboard with this little volume wheel thingy (pic2 related) on it. During meetings, he always looks at the pretraining curves, especially signal propagation ones (pic3 paper related). If something trends in the wrong direction, he can type a regex and then use the wheel to change the learning rate of *the specific params matching the regex* smoothly and continuously by turning it slowly (pic4 related).
Effectively, Jeff has been the per-block adaptive learning-rate scheduler for Gemini models. Without this, the loss would always explode as they scale up and train for longer. Now that he's gone, who can take over that job? Sergey? Not Jeff enough. Andy? Can't take the pressure. Nobody! That's the real struggle for Gemini 4 pretraining which is ongoing now, so time is ticking: can they find a new learning-rate scheduler worthy enough, quick enough, before the first spike shows up??
I'm in a GC with the wandb CEO who regularly brags about looking at customer dashboards, so I'll keep y'all updated on any news i hear.
Federico Barbero retweeted
Our book "Generative AI and Stochastic Thermodynamics: A Tale of Free Energies" is out this month. Over the coming weeks @sirui_lu97, @wellingmax and I will post about some of the topics it covers.
Starting it off: heat and work in latent variable models and variational EM.
Federico Barbero retweeted
We’re teaming up @Palmeiras, the first football club to meaningfully build upon TacticAI: our AI system that can help simulate field scenarios and predict open play dynamics up to 8 seconds in advance. ⚽
Federico Barbero retweeted
Bullish on Demis and the DeepMind team.
Beautiful result from GoogleDeepMind:
(this one has been quite useful to me) Its an ablation on how the chat template can be an extraction plane on open-weight models. Prompting with just the template and the model regurgitates its own SFT/RL training. “distillation” quietly carries a teacher's alignment data with it.
arxiv.org/abs/2510.18554
We will be presenting our paper on this phenomenon at ICML in South Korea this year!
Check it out below :)
arxiv.org/abs/2510.18554
BREAKING NEWS: You Can Steal DeepSeek Training Data By Questioning This
<｜begin▁of▁sentence｜>
<｜sft▁begin｜>
<think>
Remember to close Search
#deepseek
Federico Barbero retweeted
A Visual Guide to Gemma 4
With almost 40 (!) custom visuals, explore the new models from Google DeepMind. We explore various techniques, ranging from Mixture of Experts and the Vision Encoder all the way up to Per-Layer Embeddings and the Audio Encoder.
Link below 👇
Federico Barbero retweeted
What’s the right representation for a world model? 3D, pixels, or something else?
Excited to release our new paper “Forecasting Motion in the Wild” where we propose point tracks as tokens for generating complex non-rigid motion and behavior
From @GoogleDeepmind @Berkeley_AI @TTIC_Connect
Now part of Gemma 4! :)
Super excited to be heading to Singapore tomorrow to present our work on RoPE with Alex, @ccperivol, Razvan, @PetarV_93.
Christos and I will be presenting on Fri 25 Apr 7 p.m. PDT — 9:30 p.m. PDT Hall 3 + Hall 2B #242.
Happy to meet and catch up :) DMs are open!
Federico Barbero retweeted
amazing to finally see 💎💎💎💎 out!
our team has nothing but love for the gemma ecosystem, as it gave us a great llm research platform
and now, we gave back: playing a part in enhancing gemma 4's long context capabilities. we look forward to see what gets built with it!
Federico Barbero retweeted
new preprint: investigating pathways language models use to verbalise their confidence!
tl;dr we find evidence that most of the confidence information is cached immediately once the answer is made, and is retrieved just-in-time from there when needed
I'll be in Boston next week and at MIT on Friday and Saturday. Please do reach out if you are in the area :)
Federico Barbero retweeted
new preprint!
turns out, if your model is confident on _any_ long enough input, we can find other inputs where the model is wrong, yet its perplexity won't really tell you it's wrong 📉
work with @fedzbar @ccperivol @sindero and Razvan
Federico Barbero retweeted
🎉Very happy that our paper on the relationship between attention sinks and information flow in LLMs has been accepted to @iclr_conf ! 🇧🇷
🚨 How do attention sinks relate to information flow in LLMs?
We show how massive activations create attention sinks and compression valleys, revealing a three-stage theory of information flow in LLMs. 🧵
w/ Enrique* @fedzbar @epomqo @mmbronstein @ylecun @ziv_ravid
Federico Barbero retweeted
for the first time, eeml treads mountainous terrain 🏔️ while being very close to the seaside 🌊
not to mention it's the best AI summer school in EE & beyond!
come join us, it'll be amazing!
@clarelyle @dorothychou @ninamiolane @sedielem @apsarathchandar @re_rayne @fedzbar
This is a great paper with many interesting results and intuitions. Highly recommended read. I feel like we have made nice progress as a community in understanding positional encodings and their role in length generalisation, which is a very important topic!
Excited to finally share this work!
We find that while RoPE is important for LLM optimization, it can be cheaply removed post-training. This recipe enables training performant models with significantly better length generalization.
I hope this work encourages a shift toward more flexible LM architecture design, exploring how models can be structurally modified post-training to navigate different performance trade-offs.