@LiamFedus
Account based inUnited States
About this account
- Account based in
- United States
- Connected via
- United States App Store
Account-level information from X, not a live location or the device used for a specific post.
Building industrial-scale science at @periodiclabs Past: VP of Post-Training @OpenAI; Google Brain
San Francisco, CA
Joined October 2012
- Tweets1.3K
- Following1.2K
- Followers41.1K
- Likes15.6K
Pinned Tweet
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Periodic 🤝 SGLang to further improve open-source software
Happy to contribute a fast and correct RL sampling mask from @periodiclabs, developed jointly with @nanjiangwill @agarwl_ and now running in our production RL pipeline!
Liam Fedus retweeted
💯All roads for AI science lead to real labs.
Most of science is observation-limited! AI will need to run experiments if we want to learn faster in the world of atoms.
Equal token counts don’t mean equal work.
Our scheduler uses estimated compute cost to balance work across DP workers and PP microbatches. Within a sequence, later tokens attend to more preceding tokens, making them more expensive to process. Giving each GPU a mix of early and late chunks helps balance that work. This integrates nicely with the rest of our RL stack, including MoE routing replay and the vision encoder.
Neon’s scientific traces are long and vary widely in length. Training on them shouldn’t mean GPUs chewing through padding or waiting for each other.
Here’s how we pack sequences, balance the work, and distribute long contexts across GPUs at @periodiclabs 👇
periodic.com/news/ai-infrast…
Guaranteed to be a high-impact role
Also, my team is hiring! We research long-horizon agents and multi-agent. We’re hiring for alignment/safety because we want to develop new research with alignment/safety in mind during the whole process. We’re also hiring for human-AI interaction. openai.com/careers/research-…
Liam Fedus retweeted
Another Neon infra tidbit: @periodiclabs accelerated checkpoint conversion ~30x down to just 1 minute, helping us deploy faster to get rapid feedback from our labs.
Our models train in Megatron, but SGLang inference consumes weights in HuggingFace format. Traditionally, conversion ran serially:
1. rebuild the entire checkpoint from individual weight tensors
2. convert the checkpoint
3. reshard it to be inference-ready
This could take 30 min for a trillion parameter model!
Noticing that conversion shouldn't require materializing the full checkpoint, @hsu_byron introduced Fast Resharding:
1. parallelize conversion across Ray actors
2. convert expert chunks directly instead of assembling the full expert tensor in memory
We upstreamed Fast Resharding to Miles in PR #1371. Now even you can deploy trained models to production in a matter of minutes.
Ray Summit was a great deep-dive into the ML systems that power many of our favorite models. My talk is now online
The talk is on YouTube now. youtube.com/watch?v=ts19xgCD…
Liam Fedus retweeted
Excited to welcome @asadovsky as Harvey’s Chief Research Officer.
Before Harvey, Adam co-led post-training at Microsoft AI and Google DeepMind.
As a CVP at Microsoft AI, he helped build MAI-Thinking-1, Microsoft’s reasoning model.
As part of Gemini’s leadership team he helped train Gemini 1.0 through 2.5, including fine-tuning, RL, data, and evals.
His prior work as a Distinguished Engineer at Google spanned Assistant, Search Quality, and Search Infrastructure.
I met Adam three years ago when I sent him a cold LinkedIn DM and was surprised he responded.
At a time when most dismissed the application layer and legal, Adam was curious and generous with his time.
He quickly became someone I regularly turned to for advice on AI as we scaled Harvey over the past three years.
When we first met, we were too early to hire someone of his caliber and scale, but I always hoped we’d eventually work together.
As Winston and I got to know him better, what stood out even beyond his technical achievements was his character.
Despite his incredible technical career, he remains curious, humble, practical, and cares deeply about the teams he builds.
We couldn’t think of a better leader to help us build frontier intelligence for the professionals and institutions we serve.
We're excited to welcome @asadovsky as our Chief Research Officer.
Prior to Harvey, Adam co-led post-training at Microsoft AI and Google DeepMind.
As CVP at Microsoft AI, he led post-training of MAI-Thinking-1.
As a Distinguished Engineer at Google DeepMind, he led teams working on Gemini 1.0 through Gemini 2.5.
Earlier at Google, he led teams working on Assistant, Search Quality, and Search Infrastructure.
Liam Fedus retweeted
You saw the AI & science. Let's talk about the RL infra it took to build @periodiclabs Neon.
To minimize training-inference mismatch in RL, SGLang captures inference's MoE routing decisions for each rollout and we "replay" them while training.
In agentic (multiturn tool-use) settings, SGLang exports these router decisions in response to each decoding request, i.e. after each conversation turn.
So when *any* data-parallel rank finishes a conversation turn, *all* other ranks must wait until routing data finishes exporting. This slowdown is exacerbated because we export routing decisions from the *entire* conversation rather than just the most recent turn!
When @hsu_byron @vwxyzjn discovered this in our Kimi K2 RL setup, they introduced Delta Router Replay: cache previous turns' router decisions on the training client, so you can export only the delta (most recent turn's router decisions) upon each decoding request.
Delta Router Replay significantly speeds up our long-context agentic RL runs, and @hsu_byron upstreamed it to SGLang (#24851) a few months ago.
Liam Fedus retweeted
very much enjoy reading the team's AI infra blog. periodic.com/news/ai-infrast…
A super clear training-rollout-sandbox pipeline with many clear task-specific optimizations.
Better AI infra, better AGI
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
We check OOD generalization for @periodiclabs Neon and report optimal cost-performance here, too. This indicates useful generalization for our model.
But our experimental data also opens up an interesting ML research program.
For example, RL tasks to directly predict experimental outcomes given all prior experimental data known up until that date. This is loosely analogous to the often-proposed ML experiment of training AI on all literature before 1905 and then evaluating whether it can derive the theory of relativity.
An AI that already knows the answer through pre-training can cheat on RL tasks like this (i.e., it can skip reasoning and simply output a memorized experimental result). A unique, complete, dated system of record makes this work possible and fruitful.
ALT We evaluate Neon’s generalization on 198 experimental XRD measurements using held-out chemical systems. Specifically, our lab data training split for both reinforcement learning and midtraining excludes these systems and any systems containing them, while smaller subsystems can appear in training. This benchmark tests generalization to held-out chemical systems, but is easier than FrontierXRD, which focuses on samples that are difficult to resolve even for human experts.
“What did we actually make?”
Answering this can take hours. Lab data + midtraining + RL took X-ray diffraction analysis success from 2.7% to 55.3% (~20×) on 134 difficult samples, scored by model judges calibrated against human experts. We see great scaling properties with respect to additional RL.
Read more about our research here.
periodic.com/news/nature-is-…
Liam Fedus retweeted
Incredible work, and very cool to see @periodiclabs using @raydistributed.
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Thanks, @JeffDean! Exciting era for AI to direct discovery and advance knowledge.
Liam Fedus retweeted
labs currently have a lot of pressure to do everything themselves and breakthroughs like this, high value specialisation, will ensure we progress much more quickly.
incredibly exciting work.
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Liam Fedus retweeted
super cool to see real-world experimental data at scale in the training pipeline — periodic is a pioneer of a new class of lab
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Liam Fedus retweeted
Watch Neon investigate a real sample from our superconductor lab
It hypothesizes about crystal structure, reasons about synthesis conditions, and iterates with tools until it finds a physically realistic solution
More and more of science will look like this in coming years
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Thanks, @Ollmer, for the foundational infra development and research to make Neon work
At Periodic we’re building a loop for autonomous materials science.
Our first model, Neon, offers an early glimpse of how multimodal lab-grounded RL built on robust ML infra pushes the boundaries of AI’s scientific capabilities. Nature is the ultimate testing environment and capturing its nuances requires going beyond just text.
Liam Fedus retweeted
Periodic Labs is doing extremely cool work applying frontier AI research to materials science.
Here they post-train a model for a X-ray diffraction analysis, beating GPT-6 Astra and boosting performance of Kimi K2.6 by 20x!
What's worth noting here is Periodic is doing RL directly with real-world experimental data, which comes with many unique challenges over coding/knowledge-work environments.
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Liam Fedus retweeted
Check out our work training custom models on an important part of the materials discovery loop.
The scaling direction is pretty clear. We will continue getting bigger and smarter models, these models will have access to better simulations and higher-throughput labs, and they will run increasingly autonomous physical experiments.
Periodic will be the best place in the world to do this type of training.
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.