RS Intern @Microsoft Research, Ph.D. Student @Mila_Quebec and Concordia University. Working in Deep learning optimization. Previously @Kaust
Montréal, Québec
Joined September 2013
- Tweets192
- Following2.9K
- Followers430
- Likes5.3K
Pinned Tweet
Training LLMs over low-bandwidth networks is hard but its necessary in the current landscape. In pipeline parallelism, every stage sends its activations to the next stage, and as models scale, this communication becomes the main bottleneck.
Our new work 🍁MAPL asks: Can we cut this communication sharply without hurting model quality and token efficiency ?
🧵 1/N
Paul Janson @ ICML 2026 🇰🇷 retweeted
Your neo-lab wants to pre-train a frontier model, but no provider will sell you enough co-located chips?
We got you.
Our new work shows that SOAP & SPlus achieve strong worker scaling to 96+ DiLoCo workers (e.g., 96 providers) by whitening the update’s covariance. 🧵1/9
Paul Janson @ ICML 2026 🇰🇷 retweeted
New work from our lab on replica scaling in local optimization/DiLoCo for LLM pretraining!
arxiv.org/abs/2610.19064
Paul Janson @ ICML 2026 🇰🇷 retweeted
RL post-training grades a language model on its finished answer. One reward, and every choice that led to it is pushed up or down together.
That works when an answer is written left to right. For diffusion language models it falls short, and here is why.
🧵 1/N
Paul Janson @ ICML 2026 🇰🇷 retweeted
Mila’s annual supervision request process is open for MSc and PhD applications for Fall 2027 admission. While I won’t be taking on new students for supervision this cycle, I encourage you to apply to join Mila’s brilliant deep learning research community.
For more information and guidelines, you can visit the link below or contact the @Mila_Quebec team directly.
mila.quebec/en/prospective-s…
Paul Janson @ ICML 2026 🇰🇷 retweeted
[Updated schedule] Don't miss our COLM workshop happening today (Oct 9th)! Starting with a talk from
@Yoshua_Bengio
at 9 am, we have a stacked lineup of speakers, including
@_albertgu @lateinteraction @achowdhery @agarwl_ @EyubogluSabri !
Happening in Continental Ballroom 5
Paul Janson @ ICML 2026 🇰🇷 retweeted
Paper rejected from arxiv but accepted at neurips. Can’t make this stuff up
update: our arXiv rejection made it to NeurIPS!!!!
hal.science/hal-05708820v1/d…
“08/26: Thank you for submitting your work to arXiv. We regret to inform you that arXiv's moderators have determined that your submission will not be accepted and made public on arXiv.org.”
Paul Janson @ ICML 2026 🇰🇷 retweeted
We’ve been researching fault tolerance in Crucible, Templar’s pre-training platform.
The goal: keep training through node failures and make better use of unreliable workers and spot instances, without idling an entire model replica when one stage goes down.
1/n
Paul Janson @ ICML 2026 🇰🇷 retweeted
Call for Papers — CODEC-FM @ NeurIPS 2026 (Sydney)
Frontier-scale training shouldn't require a hyperscaler's cluster. We're bringing together the open-model and decentralized-training communities to address the core technical challenges of this paradigm, extending foundation model training beyond a single datacenter and supporting more scalable, collaborative open-model development.
:push_pin: Topics: communication-efficient & async optimization · heterogeneous & fault-tolerant systems · federated learning at scale · security in trustless settings · open model infra & governance · benchmarks
:alarm_clock: Deadline: Aug 29, 2026
Details & CFP → collaborative-open-decentral…
@SameeraRamasin1 @benjamintherien @Ana_koloskova @ebelilov @niclane #kaja #Aakanksha @Pluralis
Paul Janson @ ICML 2026 🇰🇷 retweeted
Pluralis-8B is done, the first pipeline-parallel training over WAN with our new Agora system. We maintained approximately 20% MFU throughout the run, with a completely dynamic pool of devices utilising networking speeds 100-1000x slower than a datacenter setup
I’ll be at @icmlconf 2026 🇰🇷 presenting Spectron at the main conference and also at the Protocol Learning Workshop by @Pluralis.
We’re excited to share our new work : “Stabilizing Native Low-Rank LLM Pretraining.” (arxiv.org/abs/2602.12429) 🚀
Can we train LLMs from scratch using only low-rank factorized weights and still match dense performance?
Short answer: yes (with care).
I’ll also be presenting MAPL 🍁 at the Workshop on Connecting Low-Rank Representations in AI at @icmlconf 2026.
xitter.cf/i/status/2069872346404…
Training LLMs over low-bandwidth networks is hard but its necessary in the current landscape. In pipeline parallelism, every stage sends its activations to the next stage, and as models scale, this communication becomes the main bottleneck.
Our new work 🍁MAPL asks: Can we cut this communication sharply without hurting model quality and token efficiency ?
🧵 1/N
Paul Janson @ ICML 2026 🇰🇷 retweeted
We’re excited to share our new work : “Stabilizing Native Low-Rank LLM Pretraining.” (arxiv.org/abs/2602.12429) 🚀
Can we train LLMs from scratch using only low-rank factorized weights and still match dense performance?
Short answer: yes (with care).
Paul Janson @ ICML 2026 🇰🇷 retweeted
Training LLMs over low-bandwidth networks is hard but its necessary in the current landscape. In pipeline parallelism, every stage sends its activations to the next stage, and as models scale, this communication becomes the main bottleneck.
Our new work 🍁MAPL asks: Can we cut this communication sharply without hurting model quality and token efficiency ?
🧵 1/N
Our work is accepted at the 2nd Workshop on
Connecting Low-rank Representations in AI at ICML 2026. This is joint work with Edouard Oyallon and Eugene Belilovsky (@ebelilov).
Paper : arxiv.org/abs/2606.05484
🧵 N/N
