DatologyAI builds tools to automatically select and optimize the best data on which to train AI models, leading to better, smaller models which train faster.
Redwood City, CA
Joined September 2023
- Tweets299
- Following21
- Followers3.8K
- Likes233
Pinned Tweet
Today, we’re excited to launch the Datology Curation Studio, customizable frontier data curation for every team training its own models.
Data quality is the ultimate compute multiplier, and Curation Studio makes it easy for everyone.
"I'm confident that there's at least another 1000x of compute efficiency gains we can get from data." @arimorcos
On Oct 27, the team demonstrates where that headroom comes from, live: tinyurl.com/datology-webinar
Change your GPU count mid-run and a typical data loader changes your data order, and possibly your results. Zephon keeps every global batch identical, and now works with TorchTitan.
Thanks @PyTorch for having us on the blog.
Zephon’s TorchTitan integration preserves identical global batches when training resumes on a different GPU count.
In our latest blog’s 1B-parameter model test, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run.
Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed copy of the corpus.
The data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged when resuming. The lane count must divide evenly across data-parallel replicas, and each optimizer step’s batch must split evenly across lanes. Training losses can still differ because floating-point reduction order depends on GPU count.
✍️ @datologyai @josh_wills @MaxiBoether, Ties Robroek (@datologyai)
Read more: pytorch.org/blog/bringing-el…
DatologyAI retweeted
What a great opportunity! Get paid to learn about data problems? I wish this existed when I was a PhD!
Applications are open for the January cohort of the Datology Labs Fellows Program.
Funding, compute, and mentorship to work on the science of data.
Paid · One semester · San Mateo/NYC
Apply by Nov 15: labs.datologyai.com/fellowsh…
Applications are open for the January cohort of the Datology Labs Fellows Program.
Funding, compute, and mentorship to work on the science of data.
Paid · One semester · San Mateo/NYC
Apply by Nov 15: labs.datologyai.com/fellowsh…
Today we’re launching Datology Labs: a new commitment to open and collaborative research into the science of data.
Research has been a core pillar of DatologyAI from the start. Now we’re partnering with researchers at universities and across industry to take it further.
The more we learn about data, the more questions we discover. Some of these questions stem from the fundamental relationship between data and learning, while others challenge how we think about safety, alignment, and entirely new scientific domains. Discoveries that demand big swings also demand exceptional ambition, and an appetite for failure.
Read more about our mission: labs.datologyai.com/?v=2
DatologyAI retweeted
Data mixing is one of the open frontiers for efficient training. Great work by @arimorcos and the Datalogy team for this impressive result!
DatologyAI retweeted
In addition to RL as a service, now we have LLM training data curation as a service from @arimorcos
Nice blog showing the large compute efficiency wins during post-training possible by just better data curation during pretraining / midtraining.
Curation Studio puts this process in your hands:
Clean → Calibrate → Create → Compose.
Bring your data, optimize for your tasks, and keep your data inside your VPC.
See how it works:
datologyai.com/blog/introduc…
If you want to own your own intelligence, book a demo with our team, or join us live on October 27th to see Curation Studio in action and chat with the team behind the research.
Save your seat here: 245669869.hs-sites-na2.com/c…
New from DatologyAI: Zephon, an open-source data loader for text and multimodal training.
Resume a run on a different number of GPUs and most loaders quietly change the data. In our tests, GPU count alone moved eval scores by up to 0.82 points, about 4x the gap used to choose between data curation methods.
tl;dr: same training batches on 8, 16, 32 or 64 GPUs, even while tokenizing, packing and mixing on the fly.
For how it works, see @MaxiBoether's thread. For the infra view, see @josh_wills's.
Blog: datologyai.com/blog/zephon
Code, docs and paper: zephon.io
Maxi on the research: xitter.cf/MaxiBoether/status/210…
Josh on the engineering side: xitter.cf/josh_wills/status/2107…
Data Loader (n.): The part of your training stack that is too ML-y for the systems folks and too systems-y for the ML folks.
It's the work I love best, and I had a ton of fun building a new one with the team here at @DatologyAI. My three favorite things about Zephon:
Dumplings & Data at SF Tech Week today, with @Momolicious.
Thanks to everyone who lined up for momos and stayed to talk data. Models are what they eat.
Big week continues....
Tomorrow morning: something new for anyone loading data onto GPUs.
Friday: the big one.
@arimorcos closed his keynote at @AIconference last week with a customer story: Thomson Reuters.
Decades of proprietary data, deep legal expertise, and frontier token bills eating into margins. What they did, in four clips.
They own the weights. When the next model or use case comes along, they run it again for another $450K.
That's within reach of every enterprise. Friday, we put it in your hands.