CS PhD Student @ Kerner Lab @hannah_kerner @SCAI_ASU. I am interested in foundational topics in machine learning.
Tempe, AZ
Joined March 2023
- Tweets5.7K
- Following97
- Followers280
- Likes338
Pinned Tweet
Introducing Multi-Head LatentMoE 🚀
Turns out, making NVIDIA's LatentMoE [1] multi-head further unlocks O(1), balanced, and deterministic communication.
Our insight: Head Parallel; Move routing from before all-to-all to after. Token duplication happens locally. Always uniform, always deterministic.
It works orthogonally to EP as a new dimension of parallelism. For example, use HP for intra-cluster all-to-all as a highway, then use EP locally.
We propose FlashAttention-like routing and expert computation, both exact, IO-aware, and constant memory. This is to handle the increased number of sub-tokens.
Results:
- We replicate LatentMoE and confirm it is indeed faster than MoE, with matching model performance. (See Design Principle IV in [1])
- Up to 1.61x faster training than MoE+EP with identical model performance.
- Higher model performance while still 1.11x faster with doubled granularity.
📄 Paper: arxiv.org/abs/2602.04870v1
💻 Code: github.com/kerner-lab/Sparse…
[1] Elango et al., "LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts", 2026. arxiv.org/abs/2601.18089
Chenwei Cui retweeted
Late to the party, but this blog post is excellent. Arguably the way people should first learn neural networks.
Chenwei Cui retweeted
Continuous Softmax Attention
kexue.fm/archives/11980
Chenwei Cui retweeted
In July we open-sourced a dynamic graph of everything a child learns in primary school. 1,590 concepts, 3,221 dependencies.
Now we’re turning every concept into interactive experiments kids can play and learn from.
First one, age 7: heat and water.
Everything that used to be a boring textbook can now be an engaging experience, potentially tailored to every child.
Kids education just got a massive upgrade.
Chenwei Cui retweeted
A 3B-active model can now write long-term memory as well as Claude Opus 5.
Tencent Hunyuan trained Hy-Memory-30B-A3B for one job. It decides what an agent stores, how precisely, and how new evidence updates what is already there.
The same model runs inside Mem0, MemOS, and Hy-Memory. On fixed 20% benchmark subsamples, it ranks first on all four evaluated harnesses, including held-out EverMemOS, with a 55.49% macro average against 54.62% for Opus 5.
Training has two stages. On-policy distillation teaches each framework's write protocol. RL then rewards the writer when later questions can be answered from what it stored, with a penalty for storing too much.
arxiv.org/abs/2610.22196
Chat with Paper: academy.dair.ai/papers/hy-me…
Chenwei Cui retweeted
And the innovation is not incompatible with compute and data. Creativity could come from restriction, but at the same time resources leads to innovation and taste.
🙌
mHC, iHC, oHC, xHC, and a zoo of AttnRes variants... An LLM architect with FOMO from the daily explosion of new papers? 🫠
Fear not! You can't afford to miss ScalingDepth — our community hub for keeping up with the ever-evolving world of residual connections!
We'll keep tracking and sharing the latest research, methods, and papers from the community 🚀
And it's an OPEN community! Missed something? PRs and DMs are always welcome! Got a cool blog, post, idea, or unexpected finding? We'd love to hear it! ScalingDepth hosts a dedicated space to showcase community insights and discoveries.🌍
Built by the community. Driven by curiosity. Pushing the limits of depth! 🔥
🌐 keyu-wang-2002.github.io/Sca…
Chenwei Cui retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Chenwei Cui retweeted
What is TMLR doing under a 300% increase in submissions?
- Suspensions of >150 authors
- Publicly naming major violations
- Submission quotas distinguishing slop patterns
- Desk rejections 6% → 53%; focus on clarity
- AI reviews evaluating soundness
blog.tmlr.org/2026/adapting-…
Chenwei Cui retweeted
AI for academic research:
2 free paper reviews per week from Google's Deep Review, powered by new Gemini 4 Argon.
Google just launched Deep Review for reviewing draft papers.
You can pay for more. BTW, If you're revising more than 2 papers a week--how?
research.google/deepreview/
Chenwei Cui retweeted
Introducing Depth Memory 🧠
One framework that unifies residual methods like mHC and AttnRes. They turn out to be special cases of a single view: how layers write to and read from a memory across depth.
Our new depth mixer, Linear Depth Memory (LDM), consistently beats both at minimal overhead. 👇
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
Chenwei Cui retweeted
wow, amazing technical blog post about microsoft's new 3T pre-training with very rare and clear insights on how to train models of this size
first let's get an overview of the shape of this model, it's a sparser model than Kimi K3 (80B active vs 104B), same sliding window hybrid pattern, with this time more kv heads for the sliding window layers (a bit different from stepfun and poolside that add query head). they added a shared expert and also removed the interleaved MoE/Dense pattern of MAI 1
the scaling ladder is very similar to MAI 1 with the same mechanism, each rung depends only on the number of layers. they mention that one of the key improvements is how they handle the smaller rung data distribution where they make sure to have the same repetition as the final corpus. they already did that for MAI-1 but only for data ablation iirc
their prediction is almost identical to the final run loss, very impressive
they compare the model with different base models such as deepseek v4 pro and MAI 1, tho in the case of deepseek v4 pro i'd say it's quite hard to distinguish how much the data mixture "overfitting" to the scaling ladder has an impact here (they mention it's both), need to always be a bit careful about EG metrics for data in general
they have VERY NICE scaling laws for weight decay and learning rate, for the learning rate they express it as a function of tokens per parameter (N/D) and steps (hence depending on the batch size). they also do the same for dropout where it's a function of active params only, they also find the usual sqrt(k) scaling for learning rate and batch size (other scaling laws often also converge to that)
a bunch of other stability stuff, they use Muon this time and activation gradient clipping, the efficiency gain here is actually pretty crazy at larger scale. they also increase the training horizon to ~40T text (20T unique) + a new 10T (4.5T unique) multimodal since the model is natively multimodal
imo this is a safe but very clean scaling! very exciting for the tech report and MAI-base-3, congrats!!
We now have state-of-the-art pre-trained base models in the 3T-parameter weight class, built entirely in-house at MAI.
Post-training results coming soon -- stay tuned!
Lots of details in our technical blog post.
microsoft.ai/news/frontier-p…
Chenwei Cui retweeted
New voices. Better accents. And less "checking..."
GPT-Live-1.1 brings these updates to Voice in ChatGPT, Codex, and dots, with new voices in French, German, Hindi, Mexican Spanish, Korean, and Japanese.
Rolling out today to all users. Coming soon to the API.
Chenwei Cui retweeted
John has shaped a lot of the way i think about research. specifically, (1) how to adapt quickly to new research tides, and (2) the type of research that is still worth doing today.
1) Part of the magic was seeing John being ahead of the crew in terms of AI coding agent adoption. As soon as Claude became encouraged in MSR, his retro-looking, briefcase-sized laptop immediately had 16 tmux windows of Claude terminals plastered on the screen. Despite being a senior researcher, he probably knows more about model training pipelines than many new-age researchers (me included). He was helping MSR pre-train 4B models rivaling the frontier. I mean, how often do you see senior authors putting out first-author works these days?! (see arxiv.org/abs/2609.03807) "Speed is a quality" as he puts it—being able to react quickly to new technologies (agents, libraries, etc.), discoveries, and research makes your life much easier. Investing in setting up a good research pipeline/scaffold/codebase also helps amortize the cost of implementing your next research idea.
2) When I first joined MSR as an intern, a strong emphasis John had on research was on capabilities. And I frankly think that's really the biggest priority academia should focus on these days. He likes to ask "what can't current frontier models do" and work around that. While there is some value in pursuing another row of bolded numbers at the bottom of a results table with a new architecture or optimizer tweak, many of these gains do not pan out at scale, or simply give too much of a implementation headache that the frontier would not even consider ablating. A novel capability, on the other hand, is hard to dismiss. In fact, it could be the only number in the results table because there are simply no baselines/competition! When I first came up with NextLat, I was pretty occupied with the directions of belief state learning and training RNNs with transformer. While those turned out to be interesting scientific questions, John was a strong proponent in suggesting the capabilities of world modeling and variable-length speculative decoding, which brings the method further in the space of frontier-relevance.
I really enjoyed working at MSR and I am glad I spent my formative years of research working with John!
The COLM Keynote talk on Research at AI Speed has slides here: hunch.net/~jl/Research_at_AI… .
Here we try to answer the conversation I keep having with folks around "How do I do something useful?"
Chenwei Cui retweeted
New Carnegie Mellon and Microsoft paper finds that the best AI code reviewer caught 71.9% of known bugs in its top 5 comments, still too few to remove human reviewers.
Agents now write code faster than people can review it, tempting teams to let AI review it too. Older review benchmarks scored AI against human comments, which can be nitpicks or miss real bugs.
They built CRBugBench, 1,800 code review tasks where each pull request introduced a bug that humans confirmed and fixed. The agent reviews the original change in a sandbox, and AI judges check whether its top findings name that bug.
The best agent, claude-opus-5, caught 71.9% of bugs in its top 5 comments, up from 47.7% for opus-4.6 5 months earlier. Performance dropped up to 25.5% on 1 company's private repos, where bugs more often need project context.
Use review agents alongside human reviewers, and give them your repo's own knowledge, especially on large or context-heavy changes.
Chenwei Cui retweeted
Today we are rolling out Gemini 4 Argon globally in the Gemini API, to Pro + Ultra in Gemini App, and Ultra subs in Antigravity!
Argon is a huge step forward for Gemini, and a labor of love. This model has been battle tested by over 100,000 people at Google over the last month, with so much new innovation, ambition, and hard work baked into this release. The team has been heads down over the last 6 months working to make this possible across the stack (research, infra, safety, eng, etc).
We rolled the model out to cyber defenders as part of our fairwind program and early access customers over the last 2 weeks, it has been amazing to see and talk to folks who are now using this model daily. Seeing the models writing capability, raw knowledge and persistence for coding, paired with new features like 1M token output when the model needs to work more has gotten me even more excited about our path ahead.
This is just the start of the Gemini 4 era. As you saw with 3.5, 3.6, 3.7, and 3.8 Flash, our iteration speed with Gemini is fast : ) Now with Gemini 4 Argon to work from, I am cannot wait to see the progress continue, and most importantly see what this new model enables for the world.
This is such a special moment in the arc of technology, as millions of people everyday tackle problems they could never have thought possible before. Gemini 4 Argon was built to help you tackle the hardest problems. I have personally felt this spark with Argon and hope you do too. Onwards!!
Chenwei Cui retweeted
I made a Codex-native app that can transfer highlights from my paperbacks to my ebook copies.
Osmo Pocket takes photos and sees the highlight. Takes a pic, uses OCR, then uses computer use to highlight my Kindle copy.
Everything syncs on @readwise ✨
Chenwei Cui retweeted
Lately I’ve been learning math with ChatGPT (Astra) as my tutor, using this document scanning camera and a vibecoded tool that pastes the captured image right into the chat.
This way, I just work problems on paper and ChatGPT gives feedback. It’s really nice.
Chenwei Cui retweeted
arxiv.org/abs/2610.13630
Multi-head attention residuals. Uses multi-scale depthwise conv to expand dimensions.
Chenwei Cui retweeted
Probably! I see it as two issues: architectural and numerical. QK-clip keeps logits out of the bad regime, so it helps, but the kernel error is still there once keys grow. We'll investigate, thanks for the pointer! 🙏
