8th Neural scaling workshop at @NeurIPSConf is wrapped! Some observations 🧵 1/ @darkproger (OpenAI) shared how gpt-oss was built and how to efficiently run 128 experts on a single h100. This was a significant move from OpenAI in the open-weights community. 2/ @waterluffy (MBZUAI) detailed how to optimize training LLMs and how to fight instabilities in training. As a community we are closing the gaps between open-source and closed source models, which is exciting. 3/ @mciccone_AI (Vector Institute) showed how academia can solve scarce compute problem by federating Europe's scattered public clusters. I personally think this is going to be super impactful as there is a ton of abandoned hardware not being used bc there is no physical connection. 4/ @aurickq (Snowflake) presented Arctic inference and how suffix tree cache can generate 4x speedups (now available on @vllm_project!) without losing the quality. 5/ I shared how we at @cerebras think about MoE models and where the efficiency gains are. MoEs are compute efficient, but not parameter efficient. They introduce a lot of redundancy that post-training needs to deal with. This is why looking at expert merging, pruning and quantization is important. 6/ @JustinLin610 (Qwen) shared how his team is approaching scaling models to higher intelligence. knowledge, code, math, context all matter. Make risky bets to win! Looks like NSA does not work, so don't use that :) Junyang is super down to earth and honest. Really appreciate your insights! 7/ Eric Sather (Cerebras) showed how expert merging leads to collapsed representations in MoE models, which hurts quality. Pruning should be your go-to strategy. 8/ @irinarish (CERC-AAI UdeM Lab/Mila/42.com) shared her students' work on inference optimizations. Looks like GPT-3 moment for time series models is yet to come! 9/ Ayush Kaushal (Nolano AI/Mila) discussed with us how to scale ternary models. What are the similarities with LLMs and how ternary can be more beneficial during the inference time. Was very insightful for me, would def read the paper in detail. Meta highlights on "Sovereign AI" panel with reps from US, UAE, Canada, Armenia, Japan, Korea: "Sovereign" means different things. It's independence from other countries, capability, and benefits for citizens. @sarahookr explained how low-resource languages benefit from popular ones through data scaling, but model capacity limits how many languages fit. Hrant Khachatrian noted Armenia's students now have more compute than MIT students, enabling cross-country collabs. @keunwoochoi shared how Korea's open-source contributions are critical for their government strategy. Meta highlights on "AI: Show me the money" panel with reps across academia and industry. Is AI profitable? looks like it can be. @dylan522p and @swyx both agree that we already see profit in coding apps if you're #1. @tri_dao advises hw companies to make risky bets to compete with nvidia. @irinarish noted AGI has many definitions and is hard to optimize for, so it makes more sense to optimize software while developing AI capabilities to generate profits (@swyx).