making data loaders go brr (deterministically) | mts @datologyai | prev. Ph.D. @SystemsGroupETH @anaklimovic

Zurich, Switzerland
Joined October 2012
Maximilian Böther retweeted
Awesome to see Zephon, our recently open sourced data loader, integrated with @PyTorch TorchTitan! Check out their blog, and congrats to @MaxiBoether, @josh_wills, and @SGui. We're just getting started with Zephon!
Zephon’s TorchTitan integration preserves identical global batches when training resumes on a different GPU count. In our latest blog’s 1B-parameter model test, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run. Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed copy of the corpus. The data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged when resuming. The lane count must divide evenly across data-parallel replicas, and each optimizer step’s batch must split evenly across lanes. Training losses can still differ because floating-point reduction order depends on GPU count. ✍️ @datologyai @josh_wills @MaxiBoether, Ties Robroek (@datologyai) Read more: pytorch.org/blog/bringing-el…
3
28
1,245
Maximilian Böther retweeted
MosaicML’s Streaming dataloader was one of the most underrated pieces of ML infrastructure ever built imo. It spoiled me. One of my goals when founding Datology was to take all the best things about Streaming and build its torchbearer. Because well-designed dataloaders enable entire domains of data experimentation like real-time curricula, arbitrary mixing schedules, and seamless transitions between pre- and post-training. And even better-designed dataloaders enable productionizing and scaling those experiments. I am so excited that we're finally releasing Zephon. It leveled up Datology and we think it'll level you up, too.
At @datologyai we run thousands of data experiments. Two things keep biting us: - Same run on 16 vs 8 GPUs → different evals - Every pipeline change → re-materialize the dataset Introducing Zephon, the data loader we built to process data on the fly, deterministically. 🧵
2
8
1
85
4,577
Maximilian Böther retweeted

1
4
10
179
Maximilian Böther retweeted
Maybe a bit late, but I wanted to comment on this: I think the AI boom made people forget what a PhD was meant for. Now it's going back to its original purpose: not short-term profit, but long-term research that might not have a practical use for another 10 or 20 years -> Backprop existed decades before the deep learning boom. A PhD isn't for everyone, and that's okay. Industry may suit you better if you want to build something useful now, or academia if you believe in a problem industry doesn't care about yet. The AI boom made a PhD look like preparation for an industry job, but that was never its main purpose.
Students sometimes ask me if it still makes sense, in this accelerating age, to pursue a PhD in AI. Perhaps counterintuitively, I think it's a great time to do so. I wrote up some thoughts on this here: web.mit.edu/phillipi/www/wri…
13
77
6
870
67,068
Maximilian Böther retweeted
Of course, love me some confirmation bias! DCLM was for text, DCVLM is the same for vision: analyzing VLM data mix across scales. Filtering is no bueno, but the "type" mixing matters, with unfortunately (but imo expectedly) the best small-scale mix != the best mid-scale mix.
Replying to @vishaal_urao
Finding 1: Data filtering barely helps for VLMs! 🤯 Tested 60+ filter configs (CLIP-score, DCLM-fasttext, Nemotron classifiers, UniFilter, perplexity). Best case: +0.8pp over no-filtering. Most filters do nothing or actively hurt! @giffmana @tatsu_hashimoto you might like this!
2
18
1
128
19,827
Maximilian Böther retweeted
There are very few things that have radicalized me as much as founding a UG in Germany
22
9
2
330
53,617
The misery around recent decision for EU Frontier AI Grand Challenge reveals again strong deficits in EU decision making and its lack of basic competence in ML/AI. Hard to explain otherwise how an entity without a track of record in NeurIPS/ICLR/ICML gets to do "Frontier AI".
Alexander Doria
@DorialexanderJun 20
So maybe worth a longer note. 1. Domyn is reportedly building a 6k Blackwell cluster in Italy. Meanwhile the call grants about 2.5% of all EuroHPC capacity (yeah it’s bizarre accounting) about 1k h100 cluster for a year, so a tiny fraction of Domyn’s capacity for next year which just… doesn’t make sense. Well capitalized companies hardly ever ask for public compute because it add up frictions, as you cannot use the in house stack (from what I know in France, the allocation of this type went mostly unused). That is if the promised cluster really exists this year and in the promised shape. It has already been postponed multiple times. 2. Domyn (formerly known as iGenius) never pretrained a large model from scratch. The 355b Colossus is a continuous pretrain of Nemotron with not much to show for it (the headline is just an MMLU score). Almost nothing published, hence why hardly anyone in EU AI research knows them. No public deployment story either. Still it does match almost too well the call requirements: an EU company, that "trained or deployed" >100b model, no need for it to be a full pretrain. 3. Financials are fuzzy as well. The 650m headline raise is part debt for cluster buildout, annual revenue seemingly running in the 10-20m range. What we have seems closer to a midsized company creatively rewriting itself to either a lab or an hyperscaler thanks to a sovereign push, and maybe the public compute was actually needed to keep it on. I must say my first reaction when I heard of this call was: who was it made for? At least I have an answer.
3
2
26
3,363
Maximilian Böther retweeted
wild to me that people vibe-generate slides for conference talks they are ugly (for now). they are low info densiry (thanks rlhf) but worse, they don't represent your thoughts, so your presentation of them will be terrible, unless you put in a ton of work (so just write them!)
16
7
2
111
11,178
Maximilian Böther retweeted
A key reason for Zürich's recent robotics and AI boom. There's just incredible talent hidden in ETH, EPFL, UZH and many other places!
DeepMind stayed in London because it is better for talent than Silicon Valley. "I saw London and the UK as having incredible talent from top universities like Cambridge, Oxford, Imperial and UCL. There is a deep heritage of scientific breakthroughs and world-class thinkers. There was less competition for that talent, which made it a huge structural advantage for building DeepMind." @demishassabis What is the single biggest advantage of building in Europe for you @torsten @antonosika @MaxJunestrand @matiii @ChrisParsonson @cjpedregal @matthewclifford @torstenreil @alanchanguk
6
17
5
217
32,882
Maximilian Böther retweeted
Poor one out for the homies
A tragedy is unfolding at AllenAI. Fs in the chat
8
33
5
479
52,359
Maximilian Böther retweeted
Models are typically specialized to new domains by finetuning on small, high-quality datasets. We find that repeating the same dataset 10–50× starting from pretraining leads to substantially better downstream performance, in some cases outperforming larger models. 🧵
19
81
14
623
97,702
Maximilian Böther retweeted
1/ People often think better multilingual models must come at the cost of English performance. Not true. The constraint isn’t capacity, it’s data quality, and we can fix it. Today @datologyAI shares ÜberWeb: a year of multilingual curation lessons, scaled to 20T+ tokens.
7
31
16
154
42,743
Maximilian Böther retweeted
Squirtle
@squirtle_saysJan 27
the world you remember no longer exists
490
4,784
606
74,565
2,611,591
Maximilian Böther retweeted
JosH200
@josh_wills21 Oct 2025
4/ Mixing + Curricula I want the dataloader equivalent of torch.optim.lr_scheduler so I can schedule data mixtures over time with the same flexibility that we have for learning rates. Mixtera, from the systems group at ETH, is the most interesting thing I’ve seen along these lines (and has the bonus for me of being built using our lord and savior @duckdb), and I’m excited to be hosting @MaxiBoether over at @datologyai to explore this space with us!
1
1
2
21
3,175
Maximilian Böther retweeted
ACM SIGOPS ATC. // photo credit: @FrancisYan_
Thrilled to hear USENIX ATC will now officially continue under ACM SIGOPS sponsorship! Same name, agenda, and abbreviation. This is a testament to the community's strength. 👏🏽
3
11
2
84
9,150
Maximilian Böther retweeted
Long in the making, finally released: Apertus-8B and Apertus-70B, trained on 15T tokens of open data from over 1800 languages. Unique opportunity in academia to work on and train LLMs across the full-stack. We managed to pull off a pretraining run with some fun innovations, ...
Replying to @EPFL
@EPFL , @ETH_en and #CSCS today released Apertus, Switzerland's first large-scale, multilingual language model (LLM). As a fully open LLM, it serves as a building block for developers and organizations to create their own applications: cscs.ch/science/computer-sci… #Apertus #AI
9
30
10
227
79,077
Maximilian Böther retweeted
Our paper on Modyn, a data-centric ML pipeline orchestrator, will appear at SIGMOD 2025! See our blog post about Modyn: systems.ethz.ch/research/blo… Paper preprint: anakli.inf.ethz.ch/papers/mo… Open source code: github.com/eth-easl/modyn @MaxiBoether @pinartozun
1
4
42
2,209
Maximilian Böther retweeted
🐍💥Python 3.13.0 has been released! 🎉 This is the first version with 🧵experimental GIL-free mode, an experimental JIT compiler🔧, a slick new REPL 🖌️ and many new cool features! And it's faster, smarter, and more colorful than ever! 🚀 Get it here: python.org/downloads/release…
17
361
61
1,332
163,652
Der 7. Oktober ist ein neues Schreckensdatum im Weltzeitgedächtnis. Es gibt keinen 7. Oktober mehr, an dem man nicht daran denken kann. Es liegt an uns Deutschen, dass sich Juden in Deutschland sicher fühlen. Und wir versagen auf ganzer Linie.
6
26
1
258
5,282
Maximilian Böther retweeted
85
1,014
128
11,155
1,290,797