Multiplying matrices | Pretraining @MistralAI | Prev. PhD @Cambridge_Uni, MSc @ucl & BSc @ucddublin
London, UK
Joined May 2012
- Tweets291
- Following743
- Followers1.7K
- Likes1.5K
Pinned Tweet
I spent some time last summer thinking about discrete optimization (e.g. MoEs, VQ-VAE) w/ @LiyuanLucas. A simple test bed for this is a Variational Autoencoder where the latent space is a categorical distribution (not Gaussian).
Since I couldnโt find many resources on this setting, I wrote a tutorial explaining how it works.
[๐งต1/3]
Alan Jeffares retweeted
Meet Mistral Large 4, aka Le Chonk.
โข 1T parameters, natively multimodal. 49B active.
It is the best open weights model from US or Europe on aggregated benchmarks.
โข State-of-the-art on critical workloads, including cyber defense, manufacturing and finance and it surpasses closed frontier models on visual grounding.
โข Forged in Europe end-to-end and is deployable from Europe via our own Mistral Cloud infrastructure.
โข Available to all via API today. Working with cybersecurity partners privately.
Open weights release end of October.
๐ค Made with AI
Alan Jeffares retweeted
The main goal of Bayesian ML research is to show that all methods which have previously been shown to work well in practice are somehow approximately Bayesian
Transformers are Bayesian Networks
arxiv.org/abs/2603.17063
Alan Jeffares retweeted
๐ฅNew preprint: Soft Tokens, Hard Truths
Introduces the first scalable continuous-token RL method for LLMs - no reference CoTs needed; scales to hundreds of thought tokens. Best to train soft, infer hard! Pass@1 parity โ๏ธ, Pass@32 gains ๐& better robustness ๐ก๏ธ vs. hard CoT
1/๐งต
Alan Jeffares retweeted
โก๐
๐๐ makes RL faster โ but at the cost of performance.
We present ๐
๐ฅ๐๐ฌ๐ก๐๐, the first ๐จ๐ฉ๐๐งโ๐ฌ๐จ๐ฎ๐ซ๐๐ & ๐ฐ๐จ๐ซ๐ค๐ข๐ง๐ ๐๐ ๐ซ๐๐๐ข๐ฉ๐ that applies ๐๐๐๐/๐
๐๐ for rollout ๐ฐ๐ข๐ญ๐ก๐จ๐ฎ๐ญ ๐ฅ๐จ๐ฌ๐ข๐ง๐ ๐ฉ๐๐ซ๐๐จ๐ซ๐ฆ๐๐ง๐๐ compared to ๐๐
๐๐!
๐ Blog: fengyao.notion.site/flash-rl
๐ป Code: github.com/yaof20/Flash-RL/
you might accidentally be training your LLM off-policy ๐ณ
Failing on ๐ฅ๐๐ซ๐ ๐-๐ฌ๐๐๐ฅ๐ ๐๐ with VeRL?
โ ๏ธ Mixing inference backend (๐ฏ๐๐๐/๐๐๐๐๐ง๐ ) with training backends (๐
๐๐๐/๐๐๐ ๐๐ญ๐ซ๐จ๐ง) ๐ฌ๐๐๐ซ๐๐ญ๐ฅ๐ฒ ๐ญ๐ฎ๐ซ๐ง๐ฌ ๐ฒ๐จ๐ฎ๐ซ ๐๐ ๐ข๐ง๐ญ๐จ ๐จ๐๐-๐ฉ๐จ๐ฅ๐ข๐๐ฒ โ even if they share the same weights!
๐ย Blog: fengyao.notion.site/off-poliโฆ
๐ปย Code: github.com/yaof20/verl/tree/โฆ
There are plenty of resources on standard VAEs, but I hope this different angle is helpful. I wrote the tutorial to reflect how I like to learn:
* Clear โ with illustrations, annotations, and intuitive language/notation.
* Self-contained - builds from first principles with minimal assumed background
* Mathematically complete - every step is derived and explained.
* Practical - includes working code and algorithm walkthroughs.
[๐งต2/3]
๐Tutorial - arxiv.org/abs/2505.10344
๐ปCode - github.com/alanjeffares/discโฆ
Please let me know if you spot any errors, lack of clarity, or have suggestions. Feedback welcome!!
[๐งต3/3]
Tomorrow at 10am Oral 1b in West Ballroom A, Iโll exclusively unveil how all of deep learning can be understood through the perspective of *prime numbers* in the network weights ๐ฑ
Alternatively, come chat at the 11am poster session #E-601
Our new ICML 2025 oral paper proposes a new unified theory of both Double Descent and Grokking, revealing that both of these deep learning phenomena can be understood as being caused by prime numbers in the network parameters ๐คฏ๐คฏ
๐งต[1/8]
We also provide a researcher's self-evaluation checklist that includes questions worth considering at each stage of the research process.
๐งต[7/8]
For a more detailed version of this position, check out:
Our ICML position paper: arxiv.org/abs/2506.23286
Come to the talk: Tuesday 10am @ Oral Session 1B
Visit the poster: Tuesday 11am
๐งต[8/8]