Research Scientist in artificial intelligence at DeepMind
Montréal, Québec
Joined May 2016
- Tweets1.2K
- Following219
- Followers9K
- Likes1.4K
Pinned Tweet
Transformers can be made sparse across their depth. When trained isoFLOP, we can match or exceed the performance of vanilla models, while saving inference FLOPs
arxiv.org/abs/2404.02258
Adam Santoro retweeted
reading the "mixture of depths" paper, which comes up with a novel way to conditionally apply compute depth-wise in a decoder
basically they use standard MoE-style expert-choice routing but they use it to choose which tokens get to go through every block in the decoder
Adam Santoro retweeted
Gemini and I also got a chance to watch the @OpenAI live announcement of gpt4o, using Project Astra! Congrats to the OpenAI team, super impressive work!
It's such an honor to work on Project Astra with such an amazing team from across Gemini and Google DeepMind! While the #GoogleIO keynote was happening we had a last minute idea of watching the keynote with Project Astra. Check it out!
Adam Santoro retweeted
We watched #GoogleIO with Project Astra. 👀
Adam Santoro retweeted
Mixture of depth works
for 300M Seq 512 its faster and archives better loss
Code: github.com/joey00072/ohara/t…
writeup: huggingface.co/blog/joey0007…
Adam Santoro retweeted
I have implemented Mixture-of-Depths and it shows significant memory reduction during training and 10% speed increase. I will verify if it achieves the same quality with 12.5% active tokens.
github.com/thepowerfuldeez/O…
thanks @haeggee for initial code
