Cofounder @IntologyAI | PhD CMU
San Francisco, USA
Joined August 2016
- Tweets720
- Following587
- Followers840
- Likes2.8K
Andy Zhou retweeted
Locus is now verified on the official PostTrainBench leaderboard!
Thank you to @hrdkbhatnagar, @full__rank, and @maksym_andr for reviewing and uploading our solutions on the 10-hour setting.
Andy Zhou retweeted
strong results on PostTrainBench from @intology! our team verified them, so the improvements are real.
however, i'm still not very bullish on harness engineering. i think it's a transient source of improvements like prompt engineering used to be 2-3 years ago. i.e., yes, you can get a bit of a boost (and it might matter in practice for some applications!), but it will likely not matter medium/long-term (~1-2 year horizon). a general-purpose LLM + a general-purpose harness (basically, better versions of Claude Code / Codex) is most probably all you need.
The models are improving the models.
Locus, our automated AI research system, is SOTA on PostTrainBench and post-trains Qwen3 base models that surpass the human post-trained Qwen3 model.
Today, LLMs post-trained end-to-end by Locus are in production to millions. 🧵👇
PostTrainBench evaluates agents' ability to post-train models on various domains given 10 H100 hours.
We extend PostTrainBench via PostTrainBench+, which has a greatly expanded compute budget that provides clearer signal on automated post-training capabilities. We find that thousands of H100 hours help distinguish methods' performance post-training Qwen3 1.7B-Base models, and that Locus scales best. In this setting, modes trained by Locus collectively surpass the perforamce of the offical human post-trained Qwen3 1.7B model.
In a test of generalization, we ran Locus on all live Kaggle competitions with prize money and public leaderboards. After 16 days, Locus achieved the 4th highest average rank among all participants.
Andy Zhou retweeted
This is awesome!
Also some interesting observations why AlphaEvolve doesn't excel at PTB:
The models are improving the models.
Locus, our automated AI research system, is SOTA on PostTrainBench and post-trains Qwen3 base models that surpass the human post-trained Qwen3 model.
Today, LLMs post-trained end-to-end by Locus are in production to millions. 🧵👇
PostTrainBench evaluates agents' ability to post-train models on various domains given 10 H100 hours.
We extend PostTrainBench via PostTrainBench+, which has a greatly expanded compute budget that provides clearer signal on automated post-training capabilities. We find that thousands of H100 hours help distinguish methods' performance post-training Qwen3 1.7B-Base models, and that Locus scales best. In this setting, modes trained by Locus collectively surpass the perforamce of the offical human post-trained Qwen3 1.7B model.
In a test of generalization, we ran Locus on all live Kaggle competitions with prize money and public leaderboards. After 16 days, Locus achieved the 4th highest average rank among all participants.
Check out our latest results on automated post-training!
The models are improving the models.
Locus, our automated AI research system, is SOTA on PostTrainBench and post-trains Qwen3 base models that surpass the human post-trained Qwen3 model.
Today, LLMs post-trained end-to-end by Locus are in production to millions. 🧵👇
PostTrainBench evaluates agents' ability to post-train models on various domains given 10 H100 hours.
We extend PostTrainBench via PostTrainBench+, which has a greatly expanded compute budget that provides clearer signal on automated post-training capabilities. We find that thousands of H100 hours help distinguish methods' performance post-training Qwen3 1.7B-Base models, and that Locus scales best. In this setting, modes trained by Locus collectively surpass the perforamce of the offical human post-trained Qwen3 1.7B model.
In a test of generalization, we ran Locus on all live Kaggle competitions with prize money and public leaderboards. After 16 days, Locus achieved the 4th highest average rank among all participants.
Andy Zhou retweeted
Will be interesting to see how agents (without contamination) progress on this benchmark over time
Can coding agents do research?
We release NanoGPT-Bench, an internal eval we’ve used to test agents on an AI R&D problem with months of human progress
Codex, Claude Code, Autoresearch recover only 9.3% of human progress, mostly tuning hyperparams & ignoring algorithmic research
NanoGPT-Bench is built on the NanoGPT Speedrun, a popular LLM pretraining competition to minimize the training time of a GPT-2 style model. Existing human submissions constitute nearly 2 years of work. To control for dependencies and contamination in frontier models, we standardize evaluation to a 5-month window of world records. Evaluation is fully autonomous and end-to-end, with no human intervention or internet access. 🧵
Coding agents struggle to do AI research.
Our analysis on the NanoGPT speedrun show they rarely attempt ambitious ideas, instead opting for hyperparameter tweaks. Releasing our framework as a benchmark here!
Can coding agents do research?
We release NanoGPT-Bench, an internal eval we’ve used to test agents on an AI R&D problem with months of human progress
Codex, Claude Code, Autoresearch recover only 9.3% of human progress, mostly tuning hyperparams & ignoring algorithmic research
NanoGPT-Bench is built on the NanoGPT Speedrun, a popular LLM pretraining competition to minimize the training time of a GPT-2 style model. Existing human submissions constitute nearly 2 years of work. To control for dependencies and contamination in frontier models, we standardize evaluation to a 5-month window of world records. Evaluation is fully autonomous and end-to-end, with no human intervention or internet access. 🧵
Stay tuned 🤫
New NanoGPT Speedrun WR at 105.9s (-1.0s) from @soren_dunn_ , with a triton kernel to fuse the logit softcap and multi-token prediction cross entropy calc. Interestingly, Soren mentioned that their autonomous system Locus at Intology discovered and implemented the improvement. github.com/KellerJordan/modd…
Andy Zhou retweeted
This research is impressive!
Shows the true potential of scaling test-time search for scientific discovery.
It really pushes long-horizon exploration.
"Locus achieves consistent performance boosts up to several days by orchestrating thousands of experiments simultaneously."
Introducing Locus: the first AI system to outperform human experts at AI R&D
Locus conducts research autonomously over multiple days and achieves superhuman results on RE-Bench given the same resources as humans, as well as SOTA performance on GPU kernel & ML engineering tasks.
RE-Bench is a collection of several frontier AI research tasks that typically take human experts (e.g., top ML PhDs and frontier lab researchers) several days. By scaling experimentation to far longer time horizons than previous systems, Locus represents a step change in AI scientist capabilities. 🧵
Andy Zhou retweeted
Autonomous research agents have the potential to solve some of humanity’s biggest problems.
This is the first AI system to outperform humans in R&D - very cool!
Introducing Locus: the first AI system to outperform human experts at AI R&D
Locus conducts research autonomously over multiple days and achieves superhuman results on RE-Bench given the same resources as humans, as well as SOTA performance on GPU kernel & ML engineering tasks.
RE-Bench is a collection of several frontier AI research tasks that typically take human experts (e.g., top ML PhDs and frontier lab researchers) several days. By scaling experimentation to far longer time horizons than previous systems, Locus represents a step change in AI scientist capabilities. 🧵
Super excited about our progress! We've been building out our latest AI scientist system and wanted to share some early results. We were surprised that Locus was not only SOTA on RE-Bench but even surpassed human experts! In the coming months, we'll be releasing novel discoveries made by Locus.
Very proud of our team! We firmly believe AI systems will transform the process of conducting science - if our mission resonates with you, consider joining us: [email protected]
Introducing Locus: the first AI system to outperform human experts at AI R&D
Locus conducts research autonomously over multiple days and achieves superhuman results on RE-Bench given the same resources as humans, as well as SOTA performance on GPU kernel & ML engineering tasks.
RE-Bench is a collection of several frontier AI research tasks that typically take human experts (e.g., top ML PhDs and frontier lab researchers) several days. By scaling experimentation to far longer time horizons than previous systems, Locus represents a step change in AI scientist capabilities. 🧵
Excited to see more groups working towards recursive self-improvement!
Today, I am launching @axiommathai
At Axiom, we are building a self-improving superintelligent reasoner, starting with an AI mathematician.
Andy Zhou retweeted
Excited to be announcing the #AI4Science community, in collaboration w/ @askalphaxiv.
As a part of our speaker series, we are hosting @jeffclune this Friday.
Join 4000+ others passionate about AI-accelerated discovery.
(event link & invite link) 🧵👇
Andy Zhou retweeted
🚨 Great AI talk alert! 🚨
Join us next week for a talk by @jeffclune at the #AI4Science seminar!
sign up link: luma.com/jviy4bz5
Join @intology x @askalphaxiv on 8/8 at 11 am PST to hear @james_y_zou (Stanford) discuss his work on automated scientific research with AI agents! Most recently, The Virtual Lab, which discovered new SARS-CoV-2 nanobodies. 🧑🔬🔥
lu.ma/h6en3gdl?tk=IZJwi3
