10X Builder | AI Performance Engineer | Co-founded @mybridgecard(YC S22)
Making GPUs Go Brrr
Joined July 2018
- Tweets3.8K
- Following1.7K
- Followers567
- Likes5.9K
Pinned Tweet
I recently got a PR merged into @nvidia's Dynamo (A Datacenter Scale Distributed Inference Serving Framework) and the process was a great A/B test of KV data transfer over TCP vs KV data transfer over RDMA.
PR link: github.com/ai-dynamo/dynamo/… 🧵
One thing I've found really interesting about efficient LLM serving is how many of the optimizations we use today build on ideas that have existed in computer science for decades.
Some examples:
1. PagedAttention and Virtual Memory
2. Pipeline Parallelism and CPU Instruction Pipelining
3. Speculative Decoding and Speculative Execution
🧵
A smaller draft model generates several candidate tokens, and the larger target model verifies them in a single forward pass.
If enough of those tokens are accepted, we can advance multiple tokens without repeatedly running the expensive target model for each one.
We often talk about scaling AI infrastructure as if it just means adding more GPUs.
This conversation between @dylan522p and @sorcer, CTO of @CoreWeave, was a great reminder of how much more there is underneath.
At large bare-metal scale, you’re thinking about hardware failures and automated recovery, security and isolation, networking, power infrastructure, and liquid cooling.
I loved the discussion around how CoreWeave approaches full liquid cooling and how former crypto-mining sites have become valuable for AI infrastructure because much of the hard electrical work is already there: large grid connections, substations and power distribution.
Really great conversation on what it actually takes to operate AI infrastructure at scale.
youtu.be/5kerW51jdgk?is=zKx4…
I'll be speaking at 15:40 GMT+1 today. Reaally excited to be giving this talk.
Tune in. #SysConf #SysConf26
We are excited to announce Festus Owumi (@_enfinity) as a speaker at SysConf 2026.
His talk is titled “Distributed LLM Inference From First Principles: When One GPU Isn’t Enough.”
Distributed inference solves at least two important problems: memory capacity limits on GPUs and a need for higher serving throughput. But how do we build distributed inference?
In this talk, Festus walks us through building a distributed inference runtime from first principles, based on a learning project. It starts by reviewing the “boring” case of one model, one process and one [CPU] device. Then the talk will introduce ranks, world size and point-to-point communication.
After the above foundation, Festus will introduce more complex concepts to distribute the model such as Pipeline Parallelism, Tensor Parallelism and the techniques used to achieve them including partitioning and collectives such as AllGather or AllReduce.
#SysConf #SysConf26
ALT “Distributed LLM Inference From First Principles: When One GPU Isn’t Enough” is the title of Festus Owumi’s talk at SysConf 2026.
ALT Festus Owumi is an engineering leader and ML systems engineer building production distributed systems and cloud-native infrastructure, with a focus on LLM inference and GPU serving. He's contributed to NVIDIA Dynamo, an open-source inference framework.
youtu.be/N1rjtDs8blY?is=55Tr…
When we think about LLMs today, we usually think of autoregressive, decoder-based Transformers which generate one token, feed it back in, generate the next token, and repeat.
That sequential dependency is one of the fundamental limitations of LLM inference today.🧵
In this great @NoPriorsPod episode, @StefanoErmon, CEO of @_inception_ai, talks about how diffusion language models could change the economics of text generation by processing many tokens in parallel rather than decoding strictly one at a time.
The ecosystem however is still very early. Most of today’s training and inference tooling has been built around autoregressive LLMs.
Stefano mentions that @_inception_ai had to build its own serving engine because existing stacks such as vLLM and SGLang weren’t built to serve their diffusion LMs.
Really worth the watch!