
Generative AI MLE, FOSS toolmaker, innovation catalyst @CoreWeave + @AiEleuther. https://xitter.cf/t.co/z0fpuhlWRs
Seattle, WA
Joined November 2013
- Tweets10.2K
- Following1.9K
- Followers4K
- Likes11.3K
David Marx (@digthatdata.bsky.social) retweeted
'we're in this bizarre world where the best way to learn about llms... is to read papers by chinese companies. i do not think this is a good state of the world' - us labs keeping their architectures and algorithms secret is ultimately hurting ai development in the us.
David Marx (@digthatdata.bsky.social) retweeted
🔥 Are you ever dissatisfied with the imprecise names in vision-language datasets?
🚀 At #NeurIPS2024, we introduce 𝐑𝐄𝐍𝐎𝐕𝐀𝐓𝐄, showing how better segmentation dataset names lead to 𝐛𝐞𝐭𝐭𝐞𝐫 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 & 𝐞𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧.
Let’s dive in! 🧵👇
David Marx (@digthatdata.bsky.social) retweeted
New NanoGPT training speed record: 3.28 FineWeb val loss in 4.66 minutes
Previous record: 5.03 minutes
Changelog:
- FlexAttention blocksize warmup
- hyperparameter tweaks
David Marx (@digthatdata.bsky.social) retweeted
This is officially the new record! Congrats @hi_tysam (who is also an OG of CIFAR-10 speedrunning)
xitter.cf/hi_tysam/status/186085…
David Marx (@digthatdata.bsky.social) retweeted
Is KL-regularization the right tool for language model alignment?
The χPO algorithm: We show that a one-line change to DPO—moving from KL to chi-squared regularization—is sufficient to achieve state-of-the-art theoretical guarantees, provably alleviating over-optimization.
David Marx (@digthatdata.bsky.social) retweeted
Narrative on X: 🦋 has no AI/ML and just talks about itself
My actual feed on 🦋:
David Marx (@digthatdata.bsky.social) retweeted
You can choose what feeds you have on your homepage -- I personally like the "popular with friends" feed (in the above tweet). The default is "following" - which looks like this for me:
David Marx (@digthatdata.bsky.social) retweeted
Because X has tended to censor discussion of social networks I won't link directly, but look for this post to get an instant AI/ML feed thanks to @maosbot
David Marx (@digthatdata.bsky.social) retweeted
We announce LAION-DISCO-12M - a collection of 12 million links to publicly available YouTube samples paired with metadata to support basic machine learning research in foundation models for generic audio and music.
laion.ai/blog/laion-disco-12…
David Marx (@digthatdata.bsky.social) retweeted
[1/7] New paper alert! Heard about the BitNet hype or that Llama-3 is harder to quantize? Our new work studies both! We formulate scaling laws for precision, across both pre and post-training arxiv.org/pdf/2411.04330. TLDR;
- Models become harder to post-train quantize as they are overtrained on lots of data, so that eventually more pretraining data can be actively harmful if quantizing post-training!
- The effects of putting weights, activations, or attention in varying precisions during pretraining are consistent and predictable, and fitting a scaling law suggests that pretraining at high (BF16) and next-generation (FP4) precisions may both be suboptimal design choices!
Joint work with @ZackAnkner @spectorb @blake__bordelon @Muennighoff @mansiege @CPehlevan @HazyResearch @AdtRaghunathan.
David Marx (@digthatdata.bsky.social) retweeted
Excited to finally release our NeurIPS 2024 (spotlight) paper! We introduce Run-Length Tokenization (RLT), a simple way to significantly speed up your vision transformer on video with no loss in performance!
David Marx (@digthatdata.bsky.social) retweeted
Every time policymakers or private companies tried to undermine your rights online during the last Trump administration, we were there. In preparation for the next four years, here’s just some of the groundwork we’ve already laid: eff.org/deeplinks/2024/11/20…
David Marx (@digthatdata.bsky.social) retweeted
Working Talk: Speculations on Test-Time Scaling
Slides: github.com/srush/awesome-o1/
(A lot of smart folks thinking about this topic, so I would love any feedback.)
Public Lecture (tomorrow): simons.berkeley.edu/events/s…
David Marx (@digthatdata.bsky.social) retweeted
Check out our NeurIPS paper on planning in (discrete) MDPs as (a new kind of) inference. We extend to factored models using loopy BP. Joint with @lazarox8, @dileeplearning and Li Yu. arxiv.org/abs/2406.17863
David Marx (@digthatdata.bsky.social) retweeted
📢 Job Alert 📢 @naverlabseurope
My team is looking for a Research Scientist in Visual Representation Learning - More info tinyurl.com/3jz935fk
David Marx (@digthatdata.bsky.social) retweeted
“Academies call for EU-wide rules on sharing publications.”
researchprofessionalnews.com…
David Marx (@digthatdata.bsky.social) retweeted
If you insist.
Patrick Ruffini@PatrickRuffini4 Nov 2024United States
United StatesConnected via United States App StoreAccount-level information, not a live location or per-post device.
Simply vote for the candidate with the views closest to yours on taxation.
David Marx (@digthatdata.bsky.social) retweeted
I will legalize recreational marijuana, break down unjust legal barriers, and create opportunities for all Americans to succeed in this new industry.
David Marx (@digthatdata.bsky.social) retweeted
Shout this @Miles_Brundage analysis of US-China AI strategy from the rooftops!
(Read the whole thing)
“While some think that the right approach to the global AI situation is for democratic countries to race against autocratic countries, I think that having and fostering such a zero-sum mentality increases the likelihood of corner-cutting on safety and security, an attack on Taiwan (given its central role in the AI chip supply chain), and other very bad outcomes. I would like to see academics, companies, civil society, and policymakers work collaboratively to find a way to demonstrate that Western AI development is not seen as a threat to other countries’ safety or regime stability, so that we can work across borders to solve the very thorny safety and security challenges ahead.
Even if, as I think is very likely, Western countries continue to substantially outcompete China on AI, there is more than enough “gas in the tank” of computing hardware and algorithmic progress in autocratic countries for them to build very sophisticated capabilities, so cooperation will be essential. I realize many people think this sounds naive but I think those people haven’t thought through the situation fully or considered how frequently international cooperation (enabled by foresight, dialogue, and innovation) has been essential to managing catastrophic risks.”
Miles Brundage@Miles_Brundage23 Oct 2024United States
United StatesConnected via United States App StoreAccount-level information, not a live location or per-post device.
I just sent this message to my colleagues, and elaborate on my decision and next steps in a blog post (see next tweet):










































