Sung Jae Bae retweeted
one of the fun implications of swarm scaling laws is that companies are literally superintelligences
Sung Jae Bae retweeted
1/ I trained an AI to be top 1% in the world at Super Smash Bros. Melee by watching 80k hours of human replay.
Meet HAL.
Sung Jae Bae retweeted
I've been thinking about where the data for the next generation of open models will come from.
The internet gave us a remarkable commons for pretraining. Posttraining is different: it consumes tasks, not documents. And that commons doesn't exist. Not yet. openathena.ai/blog/there-is-…
Sung Jae Bae retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Sung Jae Bae retweeted
For people who aren’t sure whether these research innovations actually matter, here’s a simple explanation of why labs like DeepSeek and Kimi keep producing them:
Within a year, once LLMs can handle most everyday tasks, no one will choose an expensive model when a comparable one costs half as much. (and, separately, cannot access your data.)
The research we have seen so far has already reduced costs by more than 10x compared with models from a year ago, and these gains will only compound further.
Setting aside hardware differences, model architecture is the only factor that mathematically determines how much it costs to generate a million tokens. The rest depends on better hardware and infrastructure design, which are also continually being improved through new research.
And doing research actually doesn't require a crazy amount of compute. Coming up with an idea costs nothing, while the cost of validating ideas at <10b level scales is also negligible compared to frontier training runs. Only when a idea survives the initial smaller scale validation will it be brought onto the scaling ladder. Talent and taste matter more here.
The closed-source labs are likely doing something similar too. This is another inevitable path, alongside capability hill climbing.
Sung Jae Bae retweeted
The what now?!
arxiv.org/abs/2610.06783v1
Sung Jae Bae retweeted
Happy to share that Interleaved Head Attention (IHA) has been accepted at #NeurIPS2026! 🎉
In standard multi-head attention, heads remain isolated during attention. As a result, combining multiple relationships often has to happen across successive layers. 🧠🔗
IHA lets heads communicate before attention by constructing learned mixtures of their queries, keys, and values, creating richer within-layer interactions while preserving the standard attention operation. ⚙️✨
Across long-context retrieval and mathematical reasoning, IHA consistently improves over standard attention. 🚀
Huge thanks to my co-lead author @dvsaisurya and our amazing coauthors @rach_it_, @rish2k1, @Devvrit_Khatri, @brandfonbrener, and brilliant mentors @pliang279, Inderjit Dhillon, and Manzil Zaheer!
🎥 Quick overview below
📄 Paper: arxiv.org/abs/2602.21371
🧵 [1/14]: Talking-Heads Attention by @NoamShazeer et al. showed something interesting: maybe attention heads shouldn’t be fully isolated. 🧠
That got us thinking:
If communication across heads matters, what is the right way for heads to communicate, especially from a one-layer reasoning perspective? 🔗⚙️
That question led us to Interleaved Head Attention (IHA) ✨
📄 Paper link: arxiv.org/pdf/2602.21371
Sung Jae Bae retweeted
Paper thread! I recently read this paper from some FAIR colleagues and NYU folks in depth, so making a thread while it's fresh in my head.
The main point is seeing how far on the SMALL side we can go in scaling laws without losing fit or predictability, and what it takes?
Sung Jae Bae retweeted
I often see people mic drop with "Bitter Lesson" without maybe understanding its implications.
The best interpretation I like is from @hwchung27 "We need scalable methods that can better leverage computation."
For example, LLMs are amazing technology but their current training itself adds in a lot of inductive bias, e.g., data distribution being good enough to simply clone.
RL faces it's own version of bitter lesson: where does that reward come from? Well today, it's mostly human ingenuity.
Sung Jae Bae retweeted
wow insane, they are literally livestreaming the RL training run of Mimo V2.6 Pro (1T, 42B active) and Flash (309B, 15B active) with per batch data/harness composition and a ton of internal training metrics
mimo.xiaomi.com/rl/
Nearly half a year of silence. We spent it studying one problem: how far RL can scale.
MiMo-V2.6 is in the middle of its RL run right now. Three things we scaled: compute (~2B tokens per step, 1568 prompts × 16 rollouts, fully async), environments and harnesses (multi-task agentic RL, mixed across multiple harnesses in one run), and grader compute (agentic in-group credit assignment, with test-case and rubric-based rewards). We'll open-source the details piece by piece over the coming weeks.
Streaming the run: mimo.xiaomi.com/rl/
Sung Jae Bae retweeted
Some news: 5 months ago I joined @coreauto, a small research lab focused on new ways for models to learn. I’m also hiring four interns to do something my younger self would've loved.
It started with a pretty unusual project. We set up a handful of small online businesses, and put an agent in charge. We gave each business some initial funding and a seed idea, and off they went, trying to sell real things to real customers while we watched. It’s about as open-ended and unforgiving a task as you can probably give to an agent, with countless decisions to make and effects that take time to become clear.
Each business got its own bank account, a debit card to pay suppliers, a web browser, an email inbox, a domain name, cloud infrastructure and a schedule. Every few hours, the agent woke up and decided what to do. It might update a landing page, adjust an ad campaign, reply to a customer, and then go back to sleep.
The agents built actual storefronts and apps. They bought ads, attracted visitors, and even made some sales. The agents acted as operators that kept a business moving, but we noticed they never really stepped back to strategize about what might move the needle. If people weren’t buying, why? Was the landing page unappealing? Or maybe the offer itself was the problem?
One amusing example was the very first order a customer placed. Our supplier had run out of stock. But it’s a dropshipping business, and thankfully tens of other suppliers carried the same product. Instead of trying another supplier, the agent almost immediately sent the customer a polite apology and attempted to issue a refund. Gah! So close!
The agents could handle much of the work of getting a business up and running. What they struggled with was deciding what to change when it wasn’t working.
That’s what we want to understand. We also want to see what changes when people take the lead. What does a business operator actually do all day when agents can do so much? What new work does this allow them to do? Which decisions can’t be delegated?
—
So this winter we're hiring 4 interns as Associate General Managers. Associate Product Manager internships are programs that give early career generalists a product to own. We want to try something similar, but with entire businesses.
You’ll create and grow businesses, with agents handling the routine work so you can focus on figuring out what matters. Each AGM gets a budget and our agent infrastructure that’s designed for this. Which ideas to try is up to you. Put more money behind what finds customers and keep refining.
Where you step in tells us what agents can't do yet. You'll be in our SF office with me and people I’m so fortunate to work with, like @_arohan_, @joannejang, @MillionInt, @marksaroufim and @juliacvillagra
We don't care much where you went to school or what your resume looks like. We’re looking for enterprising people with a drive to create, who’ve made money on the internet before and already use AI extensively to enable their ideas. If that sounds like you: coreauto.com/AGM
Sung Jae Bae retweeted
We just broke 2 NanoGPT world records
On track 3 optimization, we set 2 new world records on both the Muon and MuonH optimizers by 90 (-3%) and 60 (-2%) steps.
We implement a simple variant of single EMA momentum - using multiple EMA buffers and anneal momentum during training
Sung Jae Bae retweeted
Big labs are moving away very slowly from transformers because they’re making small gradient steps in each iteration and iteration takes them multiple months.
You can just front run it by… taking big steps 😀
Sung Jae Bae retweeted
Super Smash Bros Melee in MR.
Where the players fight on your own table as a platform. Even hang off the table!
On the Meta Quest.
super smash bros melee is fully decompiled, this project started in july 2020, so it took over 6 years. the progress was hugely accelerated by llms and finished off with astra. it's a very large game with 3.88mb of compiled code.
github.com/doldecomp/melee
Sung Jae Bae retweeted
Deepseek V4.1 Flash 552B total, 8/16B active with a new arch trained on 45T tokens, there are different active parameters for input/output tokens with the encoder/decoder arch, engram, new sparse attention, new mHC, native vision
very high benchmarks (beating K3), insane efficiency, and as always amazing tech report
this is probably the most novel arch i've seen in a while, pretty insane
Sung Jae Bae retweeted
The world of R&D is forking into two paths: the token-abundant research, and the token-starved research. The future is in the former - evidentially, the progress by today's top AI industry teams and neolabs is breathtaking, where researchers’ human brilliance is super charged by AI’s assistance. Every research university president should be reading this report and reflecting on what the future of higher education research should be.
openai.com/index/research-ac…


































