nikiv.dev retweeted
Here's the best blog post I've read this week, by the @dottxt
"The next generation of LLM systems won't be judged by how well they format output. They'll be judged by whether they can be trusted to act."
We spent two years getting LLMs to speak valid JSON.
That was the easy part.
Now we're solving what actually matters: making sure the model can't even express an action it's not allowed to take.
Not "will not." Cannot. 👇
blog.dottxt.ai/control-layer…
nikiv.dev retweeted
Finally release one of my works done during my time at @GoogleDeepMind: gemma_penzai: github.com/google-deepmind/g…. It can be used to visualize and debug (Multimodal) LLMs.
In gemma_penzai, every layer is like a building block. You can also easily insert/delete/manipulate any layer you want!!! With it, you can attempt to find interesting phenemenon in LLMs, such as attention sink, and then propose new model architectures.
nikiv.dev retweeted
Last week I traced Splash Attention through the TPU compiler stack.
But before that, I had to figure out how JAX actually compiles to TPU machine code.
Google's TPU compiler is closed-source. The IRs are undocumented. So I rented a v6e for $1 and traced every layer: JAX → HLO → LLO → VLIW.
8 lines of code → 250 VLIW bundles. Here's the full breakdown:
xitter.cf/PatrickToulme/status/2…
nikiv.dev retweeted
One day we’ll be able to decompose the loss curve of a neural net into all of the quanta it learns along the way - this is one of my fav streams of fundamental research. Really promising line of work
How does scaling up neural networks change what they learn?
Despite its importance, our understanding of this question remains nascent.
I've written a long post reflecting on my model of neural scaling and its relationship to interpretability, etc.:
ericjmichaud.com/quanta
nikiv.dev retweeted
For the last 5 months, I’ve been working on Hazel, and I’m super excited to finally share a first open beta version: hazel.sh, try it out now!
Fully open source as always of course github.com/HazelChat/hazel
I started building Hazel because I’ve been super unhappy with work chat apps for quite some time. Slack feels lacking in a lot of areas, customization support feels limited, Discord is amazing but sadly feels way too unprofessional, and I don’t even want to talk about Teams…
I knew it would be a fool’s task to try to build something to replace these tools, but I wanted to try anyway and quickly got obsessed. I kept grinding on it more and more, slowly chipping away at one thing after another, and I can finally say I’m happy enough to use Hazel over any of the other solutions for my day‑to‑day communication, it's far from perfect but in a state I would love to get some feedback to make this truly the best communication platform.
I built Hazel from the ground up with @EffectTS_ , @ElectricSQL , and @tanstack db to not only make it really, really fun to develop, but also great to use and maintain.
For now, some core features include:
- Public / Private / DM channels
- Threads & replies
- Reactions & file uploads
-A media explorer
- A smart notification system
…and much, much more.
Hazel is far from done, but I hope I can inspire some of you to join me on this journey to do the impossible and build the Slack of the modern era.
nikiv.dev retweeted
Wrote a primer on @a1zhang's & @lateinteraction's important work on Recursive Language Models. Enjoy! (link in comments).
nikiv.dev retweeted
Have you ever done a dense grid search over neural network hyperparameters? Like a *really dense* grid search? It looks like this (!!). Blueish colors correspond to hyperparameters for which training converges, redish colors to hyperparameters for which training diverges.
nikiv.dev retweeted
this is how Noam and the pre-training team at c .ai did knowledge distillation!!
This, and few more tricks are covered in Today's @character_ai blogpost blog.character.ai/squinch/.
nikiv.dev retweeted
Hot topics in RL
On-policy RL
Everyone faces training rollout mismatch
- Truncated importance sampling: fengyao.notion.site/off-poli…
- IcePop: doubled-ended importance ratio clipping
- Rollout Routing Replay: arxiv.org/abs/2510.11370
Efficient rollout systems design
PipelineRL: continuous batching, in-flight weight updates, not invalidating the KV cache
Everyone in the open source community and their grandmas uses this
My favorite visualization: storage.googleapis.com/intel…
Does RL improve pass @ k?
Limit of RLVR (limit-of-rlvr.github.io) claims RL helps a base model more quickly answer questions that it already knows (sample efficiency), but doesn't improve pass @ k.
Interplay of pre-training, mid-training, and RL (xitter.cf/xiangyue96/status/1998…) roughly agrees with limit of RLVR, and did extensive experiments comparing pass @ 1 vs. pass @ 128. Main catch imo is they only did GRPO.
Pass @ k Training (arxiv.org/abs/2508.10751)
Relaxing the reward assignment constraint, improves the exploration ability of the model and the downstream pass @ k performance (k = 1, 4, 8, 16)
OLMo 3 DPO training improved pass @ k (They only showed experiments to k=32 though)
There are competing views on whether RL can genuinely improve base model's performance (e.g., pass@128). The answer is both yes and no, largely depending on the interplay between pre-training, mid-training, and RL. We trained a few hundreds of GPT-2 scale LMs on synthetic GSM-like reasoning data from scratch. Here are what we found: 🧵
nikiv.dev retweeted
🧵Five pretraining tricks from CAI.
Before the Google deal, @character_ai was running pretraining on GCP H100-TCPX which has 1/4 of bandwidth as IB (!). @NoamShazeer invented a gradient compression algorithm called "Squinch" maintaining SOTA MFU despite the poor networking.
nikiv.dev retweeted
⚠️ Limited by GPU compute? You shouldn’t need a local cluster to train reinforcement learning agents!
Meet 🔥 OpenTinker 🛠️ — our open-source RL-as-a-Service that lets you design agents locally while training and inference run seamlessly on remote GPU servers ☁️. No infrastructure headaches 🤯, no coupling between agent logic and execution 🔓 — just clean, scalable, agentic RL 🚀.
#Agent #LLM #ReinforcementLearning #AgenticAI
@thinkymachines
nikiv.dev retweeted
This might be the most @huggingface pilled paper ever
Awesome work!! And they built on top of the scripts I wrote 🥺
nikiv.dev retweeted
🔥CAD: Efficient Long-context Language Model Training by Core Attention Disaggregation
Repo: github.com/hao-ai-lab/distca
Blog: hao-ai-lab.github.io/blogs/d…
Training a long-context LLM model can suffer from severe workload imbalance caused by core-attention - the softmax(QK^T)V part.
Core-attention disaggregation (CAD) fundamentally eliminates workload imbalance by disaggregating core-attention from the rest of the model.
nikiv.dev retweeted
I was thinking of writing a blog that takes you from 0 knowledge of GPU to implementing ~CUBLAS level matmul on H100
then I realized I was not the first one to think of this and there's awesome stuff by @gordic_aleksa
aleksagordic.com/blog/matmul
nikiv.dev retweeted
Scaling MoE inference is often communication + KV-cache bound: once you push expert parallelism, decode can become dominated by collectives and imbalance, and prefill stragglers can stall an entire EP group.
New community benchmark results for vLLM wide-EP on multi-node H200 (Coreweave, Infiniband + ConnectX-7):
- Sustained ~2.2k tokens/s per H200 GPU (up from earlier ~1.5k tokens/s per GPU)
In the post we share the key pieces that enable this:
- Wide-EP (`--enable-expert-parallel`) for DeepSeek-style MoE + MLA KV efficiency
- DeepEP all-to-all, Dual-batch Overlap (DBO), and Expert Parallel Load Balancing (EPLB)
- Prefill/Decode disaggregation and deployment paths via llm-d, NVIDIA Dynamo, and Ray Serve LLM
blog.vllm.ai/2025/12/17/larg…
nikiv.dev retweeted
Very very interesting. As I've said, BD Seed is the only lab that's trying to compete with DeepMind's Gemini vision (though Qwen, Tencent HY have potential):
«Seed1.8 is developed to support generalized real-world agency»
And it's not just cute GUI VLA agency. They mean it.
