Open research for frontier AI. Building models, datasets, infrastructure, and experiments in public. Toward an Open Frontier
Joined September 2026
- Tweets12
- Following16
- Followers9
- Likes8
Marin Community retweeted
I've been thinking about where the data for the next generation of open models will come from.
The internet gave us a remarkable commons for pretraining. Posttraining is different: it consumes tasks, not documents. And that commons doesn't exist. Not yet. openathena.ai/blog/there-is-…
Marin Community retweeted
🚢Marin 535B-A23B training has crossed the halfway mark! Some observations and learnings so far:
Marin Community retweeted
If you're keeping tally of our blogs @ravwojdyla enabled:
- ~All the infra that enabled our data scale
- A kernel that let us ~2x our parameter count while maintaining higher MFU
🐐
43 days ago Marin kicked off the largest live-streamed pretraining run in history: a 535B MoE trained on 18T tokens. A scaling ladder was used to forecast eval loss along the entire trajectory, and the forecast was publicly pre-registered. Its halfway along, and despite being a 300x extrapolation, the run is within 0.3% of the forecast.
The run was originally planned to be 360B parameters. Until @ravwojdyla found a hardware implementation with custom kernels to grow to 535B parameters while also speeding up tokens per second. Learn about how he did it on the Open Athena blog: openathena.ai/blog/expert-pa….
Marin Community retweeted
Marin’s 535B-parameter hero run was originally planned as a 360B model. In openathena.ai/blog/expert-pa… we explain how expert parallelism made room for a roughly 50% larger model, with approximately 23B active parameters per token - and the tradeoffs involved.
Marin Community retweeted
There’s an enormous amount of open training data on Hugging Face. What does it take to make it work together 🤗?
For Marin’s 535B run, we built on 25T tokens from 152 datasets with licenses permitting training.
Here’s the work between downloading those and training a model 🧵
ALT Flow diagram of the Marin data pipeline. Starting with 25.25 trillion tokens, deduplication removes 2.13 trillion and benchmark decontamination removes another 13.66 billion, leaving 23.11 trillion. Documents are organized into 40 topics and five quality bands, yielding 200 sampling groups. Results from 872 small-model experiments feed a regression used to choose sampling weights.
Marin Community retweeted
It’s been a few weeks since Marin kicked off our 535B/A23B MoE hero run. So far it’s gone almost boringly well.
It’s also been just over a year since Marin joined Open Athena. I wrote up some thoughts about how far we’ve come, and how we got there. openathena.ai/blog/marin-535…
ALT loss plot showing 535B compared with ladder runs. it's very smooth and the loss is very low
Marin Community retweeted
I wrote about why more of that process knowledge should become public, and why Marin is organized around open development of frontier AI.
openathena.ai/blog/open-deve…
Marin Community retweeted
Building momentum at Marin! Upgrading from Dense -> 129B parameter MoEs -> architecture improvements -> optimizer improvements gives our pretraining recipe an estimated 6x cumulative learning speedup, accounting for MFU. Includes community contributions. openathena.ai/blog/pretraini…
Marin Community retweeted
To train better open models, we need predictable scaling.
Delphi is Marin’s first step: we pretrained many small models with one recipe, then extrapolated 300× to predict a 25B-param / 600B-token run with just 0.2% error.
Getting there took some work 🧵
Marin Community retweeted
🚢 Marin 535B-A23B started training this week! As usual, the whole process is open.
Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow.
Before kicking off the run, we trained a 4-rung scaling ladder from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens) to debug issues, and to make a forecast of our hero run. This is by far our biggest run, so definitely expecting the unexpected.
Marin Community retweeted
So about a month ago, Percy posted a version of this plot of our Marin 32B pretraining run. We got a lot of feedback, both public and private, that the spikes were bad. (This is a thread about how we fixed the spikes. Bear with me. )
ALT a very spiky training loss curve for marin 32b
