Generative AI MLE, FOSS toolmaker, innovation catalyst @CoreWeave + @AiEleuther. https://xitter.cf/t.co/z0fpuhlWRs

Seattle, WA
Joined November 2013
David Marx (@digthatdata.bsky.social) retweeted
'we're in this bizarre world where the best way to learn about llms... is to read papers by chinese companies. i do not think this is a good state of the world' - us labs keeping their architectures and algorithms secret is ultimately hurting ai development in the us.
5
25
2
123
19,025
David Marx (@digthatdata.bsky.social) retweeted
Happy Christmas friends
12
6
22
736
David Marx (@digthatdata.bsky.social) retweeted
🔥 Are you ever dissatisfied with the imprecise names in vision-language datasets? 🚀 At #NeurIPS2024, we introduce 𝐑𝐄𝐍𝐎𝐕𝐀𝐓𝐄, showing how better segmentation dataset names lead to 𝐛𝐞𝐭𝐭𝐞𝐫 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 & 𝐞𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧. Let’s dive in! 🧵👇
1
7
2
34
5,816
David Marx (@digthatdata.bsky.social) retweeted
New NanoGPT training speed record: 3.28 FineWeb val loss in 4.66 minutes Previous record: 5.03 minutes Changelog: - FlexAttention blocksize warmup - hyperparameter tweaks
8
21
10
242
75,281
David Marx (@digthatdata.bsky.social) retweeted
This is officially the new record! Congrats @hi_tysam (who is also an OG of CIFAR-10 speedrunning) xitter.cf/hi_tysam/status/186085…
New NanoGPT training speed record: 3.28 FineWeb val loss in 4.66 minutes Previous record: 5.03 minutes Changelog: - FlexAttention blocksize warmup - hyperparameter tweaks
3
11
138
19,812
David Marx (@digthatdata.bsky.social) retweeted
Is KL-regularization the right tool for language model alignment? The χPO algorithm: We show that a one-line change to DPO—moving from KL to chi-squared regularization—is sufficient to achieve state-of-the-art theoretical guarantees, provably alleviating over-optimization.
3
25
2
207
25,979
David Marx (@digthatdata.bsky.social) retweeted
Jeremy Howard
@jeremyphoward21 Nov 2024
Narrative on X: 🦋 has no AI/ML and just talks about itself My actual feed on 🦋:
58
62
20
1,265
241,899
David Marx (@digthatdata.bsky.social) retweeted
Jeremy Howard
@jeremyphoward21 Nov 2024
You can choose what feeds you have on your homepage -- I personally like the "popular with friends" feed (in the above tweet). The default is "following" - which looks like this for me:
4
5
120
21,240
David Marx (@digthatdata.bsky.social) retweeted
Jeremy Howard
@jeremyphoward21 Nov 2024
Because X has tended to censor discussion of social networks I won't link directly, but look for this post to get an instant AI/ML feed thanks to @maosbot
10
14
3
191
25,241
David Marx (@digthatdata.bsky.social) retweeted
We announce LAION-DISCO-12M - a collection of 12 million links to publicly available YouTube samples paired with metadata to support basic machine learning research in foundation models for generic audio and music. laion.ai/blog/laion-disco-12…
44
3
190
14,545
David Marx (@digthatdata.bsky.social) retweeted
[1/7] New paper alert! Heard about the BitNet hype or that Llama-3 is harder to quantize? Our new work studies both! We formulate scaling laws for precision, across both pre and post-training arxiv.org/pdf/2411.04330. TLDR; - Models become harder to post-train quantize as they are overtrained on lots of data, so that eventually more pretraining data can be actively harmful if quantizing post-training! - The effects of putting weights, activations, or attention in varying precisions during pretraining are consistent and predictable, and fitting a scaling law suggests that pretraining at high (BF16) and next-generation (FP4) precisions may both be suboptimal design choices! Joint work with @ZackAnkner @spectorb @blake__bordelon @Muennighoff @mansiege @CPehlevan @HazyResearch @AdtRaghunathan.
23
156
43
835
762,222
David Marx (@digthatdata.bsky.social) retweeted
Excited to finally release our NeurIPS 2024 (spotlight) paper! We introduce Run-Length Tokenization (RLT), a simple way to significantly speed up your vision transformer on video with no loss in performance!
22
167
13
1,416
156,159
David Marx (@digthatdata.bsky.social) retweeted
Every time policymakers or private companies tried to undermine your rights online during the last Trump administration, we were there. In preparation for the next four years, here’s just some of the groundwork we’ve already laid: eff.org/deeplinks/2024/11/20…
1
44
109
7,325
David Marx (@digthatdata.bsky.social) retweeted
Working Talk: Speculations on Test-Time Scaling Slides: github.com/srush/awesome-o1/ (A lot of smart folks thinking about this topic, so I would love any feedback.) Public Lecture (tomorrow): simons.berkeley.edu/events/s…
25
107
8
735
75,485
David Marx (@digthatdata.bsky.social) retweeted
📢 Job Alert 📢 @naverlabseurope My team is looking for a Research Scientist in Visual Representation Learning - More info tinyurl.com/3jz935fk
34
5
167
29,137
David Marx (@digthatdata.bsky.social) retweeted
“Academies call for EU-wide rules on sharing publications.” researchprofessionalnews.com…
2
7
2,707
David Marx (@digthatdata.bsky.social) retweeted
If you insist.
Patrick Ruffini
@PatrickRuffini4 Nov 2024
Simply vote for the candidate with the views closest to yours on taxation.
26
1,365
31
8,840
597,404
David Marx (@digthatdata.bsky.social) retweeted
I will legalize recreational marijuana, break down unjust legal barriers, and create opportunities for all Americans to succeed in this new industry.
14,198
13,201
8,933
114,414
45,207,526
David Marx (@digthatdata.bsky.social) retweeted
Shout this @Miles_Brundage analysis of US-China AI strategy from the rooftops! (Read the whole thing) “While some think that the right approach to the global AI situation is for democratic countries to race against autocratic countries, I think that having and fostering such a zero-sum mentality increases the likelihood of corner-cutting on safety and security, an attack on Taiwan (given its central role in the AI chip supply chain), and other very bad outcomes. I would like to see academics, companies, civil society, and policymakers work collaboratively to find a way to demonstrate that Western AI development is not seen as a threat to other countries’ safety or regime stability, so that we can work across borders to solve the very thorny safety and security challenges ahead. Even if, as I think is very likely, Western countries continue to substantially outcompete China on AI, there is more than enough “gas in the tank” of computing hardware and algorithmic progress in autocratic countries for them to build very sophisticated capabilities, so cooperation will be essential. I realize many people think this sounds naive but I think those people haven’t thought through the situation fully or considered how frequently international cooperation (enabled by foresight, dialogue, and innovation) has been essential to managing catastrophic risks.”
I just sent this message to my colleagues, and elaborate on my decision and next steps in a blog post (see next tweet):
2
2
1
29
3,729