SciFM modeling @ Open Athena
Joined March 2021
- Tweets45
- Following217
- Followers65
- Likes408
Eric Czech retweeted
I've been thinking about where the data for the next generation of open models will come from.
The internet gave us a remarkable commons for pretraining. Posttraining is different: it consumes tasks, not documents. And that commons doesn't exist. Not yet. openathena.ai/blog/there-is-…
Eric Czech retweeted
Proud to see our work featured by @GoogleDeepMind 🌱
Google calls us an AI lab, but the result came from biologists and AI researchers working side by side, with guidance from advisors who run plant labs. That pairing is the whole point of @livingmodels .
The setup: @googlegemma prepares and filters the genomic data, and BOTANIC-1, our genomic language model, scores the evolutionary impact of each candidate variant to find the causal mutation.
In a recent test on melon, the pipeline ranked a known yield mutation #1, in an afternoon of compute. Finding variants like this used to take years of physical breeding.
This is the direction we're building toward: models that shorten the path from genome to a climate-resilient crop.
Thanks to the @GoogleDeepMind for the collaboration!
Finding the exact DNA that makes crops climate-resilient used to take years of physical breeding.
To speed this up, AI lab Living Models pairs Gemma 4 E4B with BOTANIC-1, a genomic language model. Gemma prepares the genomic data, feeding filtered candidates to BOTANIC-1 to score evolutionary impact and pinpoint the causal mutation.
In a recent test, they pinpointed a target melon yield mutation in an afternoon of computation, ranking it #1 out of 2,494 possibilities.
Eric Czech retweeted
There’s an enormous amount of open training data on Hugging Face. What does it take to make it work together 🤗?
For Marin’s 535B run, we built on 25T tokens from 152 datasets with licenses permitting training.
Here’s the work between downloading those and training a model 🧵
ALT Flow diagram of the Marin data pipeline. Starting with 25.25 trillion tokens, deduplication removes 2.13 trillion and benchmark decontamination removes another 13.66 billion, leaving 23.11 trillion. Documents are organized into 40 topics and five quality bands, yielding 200 sampling groups. Results from 872 small-model experiments feed a regression used to choose sampling weights.
Eric Czech retweeted
It’s been a few weeks since Marin kicked off our 535B/A23B MoE hero run. So far it’s gone almost boringly well.
It’s also been just over a year since Marin joined Open Athena. I wrote up some thoughts about how far we’ve come, and how we got there. openathena.ai/blog/marin-535…
ALT loss plot showing 535B compared with ladder runs. it's very smooth and the loss is very low
Eric Czech retweeted
marin is the most open model training initiative in the world
you can visualize the exact pre training mixture per domain, look at sampled documents, follow the live training loss, dig into configs, scaling laws
🚢 Marin 535B-A23B started training this week! As usual, the whole process is open.
Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow.
Before kicking off the run, we trained a 4-rung scaling ladder from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens) to debug issues, and to make a forecast of our hero run. This is by far our biggest run, so definitely expecting the unexpected.
Eric Czech retweeted
Percy has long been a pioneer in LLM education. CS336 teaches the end-to-end technical stack. Marin now makes a large-scale training run fully public as it happens. For students and independent researchers without big compute budgets, this will be an invaluable reference.
🚢 Marin 535B-A23B started training this week! As usual, the whole process is open.
Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow.
Before kicking off the run, we trained a 4-rung scaling ladder from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens) to debug issues, and to make a forecast of our hero run. This is by far our biggest run, so definitely expecting the unexpected.
Eric Czech retweeted
🔥 This is genuinely one of the coolest AI experiments happening in public right now.
The Marin team has started training a 535B parameter MoE model - ~23B parameters active per token.
And unlike most frontier-scale training runs, we can actually watch this one being built.
🏗️ The scale:
• 535B total parameters
• ~23B active parameters
• ~18T training tokens
• 11 × GB200 NVL72 racks
• 792 GB200 GPUs
• ~2.7e24 training FLOPs
• roughly 3 months of training
But the hardware isn't even the most interesting part.
🧪 Before launching the 535B run, they trained a whole scaling ladder of smaller models:
61M → 162M → 481M → 1.2B active params
Why?
To predict how the giant model should behave before spending the full compute.
They can estimate the expected loss curve, watch gradient behavior, study MoE token dropping, catch instability and compare the real 535B run against those predictions as training progresses.
The current scaling-law projection lands around 2.04 dropless Paloma macro-loss.
And now the big run is live.
📉 We can follow the training curves
🧠 Inspect the architecture
⚙️ Read the actual JAX training + kernel code
📚 Examine the data pipeline
🧹 See deduplication/decontamination
📊 Watch routing, gradients and token dropping
🐛 See failures and engineering fixes as they happen
🔬 Compare predictions against the real run
They even published details about the training data pipeline: starting from 25.6T raw tokens across 292 sources, then removing ~2.5T tokens through deduplication and benchmark decontamination.
Normally we get:
“Here is our new model.”
Maybe a technical report.
Maybe some benchmarks.
This time we're getting something much rarer:
the process of creating a 500B+ model, while it's actually happening.
For anyone interested in LLM training, scaling laws, MoE architecture or distributed systems, the next few months are going to be fascinating to watch. 👀
GitHub:
github.com/marin-community/m…
Live training / W&B:
wandb.ai/marin-community/mar…
🚢 Marin 535B-A23B started training this week! As usual, the whole process is open.
Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow.
Before kicking off the run, we trained a 4-rung scaling ladder from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens) to debug issues, and to make a forecast of our hero run. This is by far our biggest run, so definitely expecting the unexpected.
Eric Czech retweeted
At @MetaCircle_AI , we are building a "Meta Model" (a model for model) that aims to predict model preformance and design better models automatically.
Here is our prediction for the Marin Hero Run:
meta-circle.com/blog/predict…
Can't wait Marin's final results to be revealed !
🚢 Marin 535B-A23B started training this week! As usual, the whole process is open.
Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow.
Before kicking off the run, we trained a 4-rung scaling ladder from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens) to debug issues, and to make a forecast of our hero run. This is by far our biggest run, so definitely expecting the unexpected.
Eric Czech retweeted
In the fight to defend openness in AI, the Marin project is a precious demonstration of openness in model training, with open code, data, recipes, even experimental results. Releasing AI research openly used to be the norm; I'm grateful for @percyliang's open lab approach.
🚢 Marin 535B-A23B started training this week! As usual, the whole process is open.
Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow.
Before kicking off the run, we trained a 4-rung scaling ladder from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens) to debug issues, and to make a forecast of our hero run. This is by far our biggest run, so definitely expecting the unexpected.
Eric Czech retweeted
At Marin we’ve found that scaling laws let us simulate the entire training trajectory. This became clear in our last 67B MoE run. The run hit the loss target within 1%, but perhaps more interestingly, arbitrary points during training could also be predicted to within 1%. This model is finishing up long context and will soon enter RL. Based on this finding, we’ve used a scaling ladder to simulate the training trajectory of the 535B MoE we kicked off yesterday. Will see how it goes. Now if someone can figure out how to fit a scaling law to each parameter we can just predict our final model and skip this training business.
Something else I've learned from this process is that getting a training run off the ground is way more than just fiddling with the architecture. Lot of herculean efforts from teammates to get trillions of tokens curated, get hardware running on a new GPU stack, improve MFU, and propose great ideas that were tested and integrated.
Given that this is our largest run to date, I anticipate lots of learning and adapting along the way. But thats part of the fun. Details: github.com/marin-community/m…
Eric Czech retweeted
🧬🧬🧬🧬🧬🧬 huge drop for DNA models!!
Variant effect prediction is probably the most important task for DNA foundation models because it directly effects human health outcomes.
And so critical that this team is not trying to reinvent the wheel on the backbone for this model, which means that it will run fast, compatibly, and be easy to develop futher.
You see a lot of people in AI for science making the mistake of trying to do architecture improvements or fundamental changes, which usually means the model doesn't work nearly as well and takes way longer to develop. Such an exemplar release tbh
Excited to share MarinDNA, a 1B gLM that rivals Evo 2 40B on variant effect prediction while being 2,330x faster.
With @eczech0, we built around a standard Transformer so we could reuse LLM infra and methods while focusing on data curation and scaling.
openathena.ai/blog/marin-dna… 🧵
Eric Czech retweeted
Excited to share MarinDNA, a 1B gLM that rivals Evo 2 40B on variant effect prediction while being 2,330x faster.
With @eczech0, we built around a standard Transformer so we could reuse LLM infra and methods while focusing on data curation and scaling.
openathena.ai/blog/marin-dna… 🧵
Eric Czech retweeted
🚨Coming to #ICML2026 🇰🇷
Last year, we introduced Block Diffusion LMs, adopted by dLLMs at Google, NVIDIA, Meta, etc.
🚀Today, we introduce Set Diffusion, a faster, more flexible alternative that combines AR and diffusion by interpolating generation orders.
🧵1/8
Eric Czech retweeted
New OSS diffusion LLM out of a major lab, this time @nvidia.
Shout-out to Cornell students @mariannearr and @SchiffYair whose work on block diffusion and encoder-decoder diffusion is at the core of this model along with MDLM.
Marianne's and Yair's papers are linked below.
Eric Czech retweeted
Building momentum at Marin! Upgrading from Dense -> 129B parameter MoEs -> architecture improvements -> optimizer improvements gives our pretraining recipe an estimated 6x cumulative learning speedup, accounting for MFU. Includes community contributions. openathena.ai/blog/pretraini…
Eric Czech retweeted
Not only do we want to train a good model, we want to know it'll be good before we even start training.
About a month ago, the Marin team launched a 129B (16B active) 1e23 FLOPs MoE run and preregistered a loss of 2.252. The run finished this past week and landed at 2.234.
xitter.cf/percyliang/status/2044…
This week, @classiclarryd kicked off a 129B (16B active) 1e23 FLOPs MoE run. In typical Marin style, we have fit scaling laws and have made a loss projection of 2.252. Stay tuned.
Eric Czech retweeted
To train better open models, we need predictable scaling.
Delphi is Marin’s first step: we pretrained many small models with one recipe, then extrapolated 300× to predict a 25B-param / 600B-token run with just 0.2% error.
Getting there took some work 🧵
Eric Czech retweeted
I've written up a new post, trying to distill the spirit of this essay into something much shorter. It is about the assumptions that interpretability researchers make, and what we might hope to learn about ourselves from studying neural networks:
ericjmichaud.com/interp-and-…
How does scaling up neural networks change what they learn?
Despite its importance, our understanding of this question remains nascent.
I've written a long post reflecting on my model of neural scaling and its relationship to interpretability, etc.:
ericjmichaud.com/quanta
Eric Czech retweeted
Researchers' brilliant ideas often get lost in the sea of endless SOTA claims on weak baselines. At Marin we battle-test ideas in an open arena, where anyone's idea can be promoted to the next hero run. One that recently rose up was @Jianlin_S MoE Quantile Balancing, used in our last 1e22 and ongoing 130B run. Animated visuals of how QB performed are available in the OpenAthena blog. openathena.ai/blog/quantile-…
