all things toward agi
Valhalla
Joined December 2022
- Tweets1.7K
- Following7.5K
- Followers251
- Likes1.9K
Program Counter retweeted
My friends from the former DeepMind Alberta have gone on to do RL for financial trading with great success!
We are looking for more amazing researchers and engineers to join our team. EquiLibre Technologies is a frontier AI trading research lab. We train our own foundational models, use our own RL algorithms, and trade our own money.
The proof is in the pudding. [THREAD]
equilibre.ai
techcrunch.com/2026/06/30/th…
Program Counter retweeted
This is a very good teaser but the details behind the scenes are even cooler!
For some intuition on CSM, Cerebras made everyone understand that if you can pool SRAM together on a big wafer you are going to go very fast. Etched can pool the SRAM without needing to keep the dies attached. This means they can push from 8x to 128x to 1000x+ chips all working together in ultra low-latency scale up domains, and then at the same time they can use LVI to pack the die with tons of flops for throughput. These are beautiful ideas and it’s great to see them coming to life!
Program Counter retweeted
It literally doesn't matter? CXMT cannot satisfy China demand this decade. So there's 0nreason besides the small price arbitrage. It's like bannin Iranian oil. India and China buy it anyways, but price diff is less – at San Francisco, CA
Program Counter retweeted
🚨 Why does Self-Play RL for LLMs keep collapsing? Most fixes focus on the reward signal. In our new paper "Survive or Collapse", we show that's the wrong lever. The true binding constraint is actually Data Gating: deciding which generated tasks enter the training pool. 🧵 1/n
Program Counter retweeted
I will be presenting our recent work on writing efficient GDN kernels on B200 GPUs at MLSys 2026 today (11 am–1 pm PDT)!
FlashInfer ran a kernel competition for B200 GPUs. Our team (@thepushkarp + me) won 🥇 1st place on the Gated Delta Net track (more details here: github.com/romitjain/kachua-…)
Do join if you are around.
#MLSys2026
Program Counter retweeted
highly recommend joining and learn from the eval master
i'll be talking about llm benchmarks, the infra behind it, the challenges and learnings later today at @tngtech :)
will be live streamed and recorded, link in replies :)
Program Counter retweeted
We built a kernel abstraction to rewrite the entire transformer stack as GEMM + Epilogue kernels!
Neural net architectures such as transformers consist entirely of matrix multiplications and elementwise nonlinearities such as RMSNorm, log sum exp, and gated activations. Fusing these elementwise nonlinearities into GEMMs in both the forward and backward passes allows us to make training and prefill as compute-bound as possible!
Our kernel abstraction CODA is implemented in CuTeDSL, and by abstracting away the fixed prologue and main loop of the GEMM kernel, we expose an epilogue function where LLMs like Claude can easily implement elementwise nonlinearities in fusions approaching speed-of-light!
Program Counter retweeted
Sold. This guy rocks
🚨 New Paper 🚨
ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models
A few modifications to Schedule-Free Learning make it completely LR tuning free, and allow it to greatly outperform schedules for long duration training!
arxiv.org/abs/2605.19095v1
Program Counter retweeted
Your drifting model is secretly a fixed point for the Wasserstein gradient flow on...
...the KL?
...an approximation to the Sinkhorn?
...Is it even a Wasserstein gradient flow at all?
arxiv.org/abs/2605.05118
@liwenliang @agalashov @JamesTThorn @ValentinDeBort1 @ArnaudDoucet1
Program Counter retweeted
next we need a lecture with @willcb 🗣️
New blackboard lecture w @reinerpope
How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do.
0:00:00 – Building a multiply-accumulate from logic gates
0:16:20 – Muxes and the cost of data movement
0:25:59 – How systolic arrays work
0:39:00 – Clock cycles and pipeline registers
0:51:40 – FPGAs vs ASICs
1:03:14 – Cache vs scratchpad
1:07:16 – Why CPU cores are much bigger than GPU cores
1:11:49 – Brains vs chips
1:15:22 – A GPU is just a bunch of tiny TPUs
Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!
Program Counter retweeted
awesome!
New in-depth blog post: "Dissecting ThunderKittens: Anatomy of a Compact DSL for High-Performance AI Kernels"
This post is my attempt to dissect ThunderKittens from the bottom up. I approached TK by asking what each abstraction is really buying us: which hardware detail it corresponds to, how it maps onto the underlying layouts the GPU actually wants, what boilerplate it removes, and which parts of the GPU programming model still remain visible to us as kernel authors.
The post walks through the tile abstractions TK provides: register, shared, and tensor memory tiles, global layouts, vector abstractions, warp/warpgroup compute, TMA, swizzling, Hopper WGMMA, Blackwell tcgen05, 2xSM MMA, tensor memory, Cluster Launch Control, TK’s pipeline templates, and static persistent tile scheduling.
At the end, I demonstrate TK’s lcf pipeline template by implementing a non-causal attention prefill kernel and benchmarking it against FlashAttention-2 and FlashAttention-3 on an H100 PCIe across different sequence lengths. The kernel beats FA2 across the sweep by ~1.55x on average, and closely tracks FA3, where FA3 is only ~1.05x-1.17x faster on the longer sequence lengths.
Blog link: hamzaelshafie.bearblog.dev/d…
Repo: github.com/HamzaElshafie/tk_…
I also put an extensive list of resources at the end, which I found very useful for interested readers.
Please note: this is my own independent writeup. I’m not affiliated with @HazyResearch, and any mistakes in the post are mine. If you spot any please reach out!
1 / xx
Program Counter retweeted
My NeurIPS 2025 paper looked into the numerical errors present in synthetic data used for neural surrogates... and uncovered a counter-intuitive finding. Since the paper can be hard to digest, here is a lightweight introduction: youtu.be/dy9vMyHlcLA
Program Counter retweeted
Really elegant idea to use symmetry as a guiding principle to derive several popular optimizers, incuding Muon. Looking forward to reading this more deeply!
1/4
New paper with @weijie444!
We introduce a symmetry-compatible principle for LLM optimizer design and, as a byproduct, get an end-to-end layerwise optimizer stack where every major matrix-valued parameter (embeddings, LM heads, SwiGLU MLPs, MoE routers) has its own principled update!
📝 arxiv.org/abs/2605.18106
💻 github.com/timlautk/equivari…
Program Counter retweeted
Why do you need to talk about CUDA streams and CUDA events for a blog post on Continuous Batching?
I have recently started reading more about LLM inference optimization. Upon asking around, I was quickly greeted by the term "Continuos Batching" (CB).
> batch your input prompts
> some generations finish ahead of others
> swap out completed generations with new prompts
The CPU needs to prepare the batch and swap out completed generations while also scheduling it to the GPU. This is a classic case of computation and communication parity.
If the communication is blocking, the GPUs are kept idle and it hurts the performance of CB. How do we work on this?
@remi_or_ in his article takes us through the CPU and GPU decoupling methods in CB.
> What are CUDA streams
> Why need CUDA events
> Forcing synchronizations to mitigate race conditions
> Bringing it all together
PS: It is one of the best blog posts I have ever read, and I request you all to give it the love it deserves.
Program Counter retweeted
The most underrated math theorem but Google secretly used it to change the world.
Perron-Frobenius Theorem (positive matrix version):
Let A be an n×n matrix with every entry a_{ij} > 0. Then there exists a unique positive real number λ > 0 (the Perron root) and a unique (up to scaling) positive vector x > 0 such that:
A x = λ x
Moreover:
λ = ρ(A) (spectral radius of A)
λ > |μ| for every other eigenvalue μ
λ is simple (algebraic multiplicity = 1)
Your search results aren't just a list; they are the coordinates of a high-dimensional vector pointing toward the most authoritative nodes on the web.
Program Counter retweeted
Today in continuous diffusion language models, we have:
- Spherical flows arxiv.org/abs/2605.05629
- Hyperspherical flows arxiv.org/abs/2605.11125
Another case of convergent evolution! Two different takes on the same core idea, published within days of each other.
🔥 New paper: Language Modeling with Hyperspherical Flows
Recent flow language models (FLMs) all use Gaussian noise. Makes sense for images, but not necessarily for text 🫠 We propose to add noise by rotating embeddings on 𝕊^{d−1} instead 🌐
w/ @caglarml
(1/9)
Program Counter retweeted
In my next blog post, I hope to share my experience of reading the CPU and GPU trace of a Large Language model from scratch.
While working on it, I wanted to share my traces with other colleagues of mine and was quite amazed that there was no good way to do it.
I have devised a simple way to share traces now:
- Build a HF bucket
- Sync the HF bucket with your local traces
- Use `ui.perfetto.dev/#!/?url=` link to share
If you need a GIST, let me know. 🤗
Program Counter retweeted
The most exciting project I’ve ever involved about ML algorithms!
Program Counter retweeted
Accepted to ICML. Excited to discuss with the community in Korea!
