Excited to share our work on a model for mapping molecules to olfactory receptors to perception, out today! Check out the link in thread ⬇️

Hardware yearns for block sparse attention, yet it seems largely absent from open weight LLMs. DeepSeek developed NSA, and people speculated DeepSeek v4 would integrate it, yet it was never utilized. We have a hypothesis as to why. We found that replacing dense attention with NSA significantly degraded its ability on synthetic retrieval tasks, even when finetuned on them. On our 32k context benchmark, it scored 0.300 compared to dense attention’s 0.904. We found the reason, and how to fix most of it. Introducing COBS🌽(Cumulant Order Block Sparse Attention), with @AdiGhai18 @sanjitneelam @ZVasania @tensorpro: • Raises NSA’s 0.300 → 0.820, closing ~86% of the gap to dense • 15.15x less KV-cache read traffic than dense (just 1.21x the NSA baseline) • Lower position-wise NLL than dense in our comparison The key insight: block selection is the keystone to block sparse attention, and existing methods are mathematically stuck in storing a first-order approximation of a cumulant generating function. COBS caches a compressed second cumulant and escapes this ceiling. Details in the paper.

Deciding when to jump in and help someone—and when to hold back and let them work through it—is something humans navigate constantly. How do AI assistants handle this tradeoff? We introduce Int-Bench, a framework for evaluating interventions during problem-solving tasks.

This week, I wanted to see if we could get the smallest possible Mixture of Experts going that takes a few hours to pretrain on a single GPU, is fast for experiments, but also somehow performs well compared to larger MoEs. For reference, we have Nanochat for transformer experimentation that's 561M params. But Qwen's smallest MoE is still 14 billion total params which is still +++ GPUs. It's been very fun to work on nanochat version of that 🐸⚡️ nanoMoE is a 500M total param MoE that you can train for <5 hours on a single H100 using 3.5-4 less OOMs than the smallest MoEs. And packs a punch for its size (reaches 87% accuracy of OLMoE-1B-7B.). Also tried out the new Quantile Balancing from kimi so no hyperparam sweeping. More numbers on performance here: github.com/chloechiaw/nanomo…

This is Google’s new diffusion LLM, DiffusionGemma’s denoising canvas over time. Diffusion LLMs can generate tokens in flexible order. But in practice, do they just become autoregressive anyway? 1/ 🧵

We'll be closing out this semester's Berkeley BioML Seminar on 4/28 with a talk from @antoinekoehl on PEINT, a powerful deep learning framework for both phylogenetic inference and protein engineering. Sign up below! luma.com/3fbbftwy

Wrote a deep dive on implementing a language model from scratch in JAX and scaling it with distributed training! If you’re coming from PyTorch and want to see how the same ideas look in JAX, or just want a hands-on intro to distributed training, check out this blog post: chuyishang.com/blog/2026/jax… Comes with code + an assignment and test cases so you can follow along!