Wow - open-source LLMs for code are so back Just one week after the latest DeepSeek release which brought vibe-coding for free, here is the high profile @allhands_ai team dropping a 32B model with matching performance on software engineering tasks benchmarks like SWE-Bench Verified. This is 20 times smaller than the DeepSeek model! Results obtained by post-training with reinforcement learning in a code environment: remember the 2025 trend of reasoning models in math/code/etc trained with RL - yep, it strikes again Feels like the open-source world has started to crack the secret code generation sauce of closed models like Claude Sonnet. And we're still very much at the beginning, this is so exciting!

This might be one of the most important 45-mn read you could indulge in today if you want to understand the secret behind high performance large language models like Llama3, GPT-4 or Mixtral Inspired by the @distillpub interactive graphics papers, we settled to write the most extensive, enjoyable and in-depth tech report we could draft on the science of creating high quality web-scale datasets, detailing all the steps and learnings that came in our recent 15 trillion tokens 🍷FineWeb release And it's not all, in this article we also introduce 📚FineWeb-Edu a filtered subset of Common Crawl with 1.3T tokens containing only web pages with very high educational content. Up to our knowledge, FineWeb-Edu out-performs all openly release web-scale datasets by a significant margin on knowledge- and reasoning-intensive benchmarks like MMLU, ARC, and OpenBookQA To close, we make a number of surprising observations on the "quality" of the internet it-self which may challenge some of the general assumptions on web data (not saying more, I'll let you draw your conclusions ;) Take a walk through it, you won't be disappointed 👇