365 days of GPU programming ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░░░░ 286/365
out of memory
Joined June 2025
- Tweets900
- Following814
- Followers7.4K
- Likes4.9K
Pinned Tweet
Day 83/365 of GPU Programming
Looking at DeepSeek's Multi-Head Latent Attention today. The last part of the AMD challenge series is to optimize an MLA decode kernel for MI355X where the absorbed Q and compressed KV cache are given and your task is to do the attention computation.
A resource that really helped internalize what MLA does was @rasbt's incredible visual guide to attention variants in LLMs (luckily he posted that last week!), which covers everything from MHA to GQA to MLA to SWA, et cetera. If there's one place to get a visual intuition for recent attention mechanisms, it's this blog post.
@jbhuang0604's video on MQA, GQA,MLA and DSA was the best conceptual intro I found on the topic and progressively builds up the ideas from first principles.
The Welch Labs analysis of MLA is a great watch as well. Beautiful visualization of the changes DeepSeek made for MLA.
Tried out a few kernels once I had a basic understanding of MLA and I think I'm slowly getting more comfortable with at least analyzing kernels.
Day 82/365 of GPU Programming
Taking a closer look at Mixture of Experts today, so I can write better MoE kernels. Specifically, to optimize an MXFP4 MoE fused kernel for the GPU Mode challenge.
I haven't had much prior exposure to MoEs, so lots of new concepts I learned today. Luckily I found the best intro to MoEs thanks to @MaartenGr visual overview of the topic.
I then watched @tatsu_hashimoto's amazing Stanford CS336 lecture on MoEs, which added deeper context around why MoEs are gaining popularity, FLOPs, OLMoE, infra complexity, routing functions (mindblown this works so well...), expert sizes, training objectives, top k routing and DeepSeek variations.
Once I had a basic understanding I started playing around with the some AITER kernels but progress there is tbd.
Also had a nice chat with @juscallmevyom (who was kind enough to reach out!) about the AMD kernels and the challenge of materialization overhead.
Day 289/365 of GPU Programming
I really enjoyed @neuralkian's last freecodecamp course on the semiconductor supply chain and I just saw he released another 8 hour intro to the physical infrastructure behind data centers, so will be spending the next couple of days going through the course. Some of the topics like networking and interconnects are more directly related to GPU programming than other chapters but I think knowing the physical stack better can only help.
Day 288/365 of GPU Programming
Finishing up the efficientml.ai assignment I started yesterday. Learning about using evolutionary search via random mutation and crossover operations to search over the space of subnetworks that satisfy specific efficiency constraints (i.e. for this example, 250 KB, 60M MACs).
levi retweeted
first part of a blogseries to understand tensor core arithmetic better.
simonveitner.substack.com/p/…
Day 288/365 of GPU Programming
Finishing up the efficientml.ai assignment I started yesterday. Learning about using evolutionary search via random mutation and crossover operations to search over the space of subnetworks that satisfy specific efficiency constraints (i.e. for this example, 250 KB, 60M MACs).
Day 287/365 of GPU Programming
Going through assignment/lab 3 today which is about efficiency based Neural Architecture Search (NAS). In this case, searching for a tiny neural network that can run efficiently on a very resource-constrained microcontroller. Should be a fun one to tackle.
Day 286/365 of GPU Programming
Time to catch up on lecture 10 of the EfficientML.ai course that was released yesterday. Depending on whether assignment 3 comes out today, I'll probably spend the weekend going through the next Lab/homework of the course. I also want to review some of the NCCL material I've been studying since I still have a lot of concepts I'm unsure about.
Day 286/365 of GPU Programming
Time to catch up on lecture 10 of the EfficientML.ai course that was released yesterday. Depending on whether assignment 3 comes out today, I'll probably spend the weekend going through the next Lab/homework of the course. I also want to review some of the NCCL material I've been studying since I still have a lot of concepts I'm unsure about.
Day 285/365 of GPU Programming
Earlier this week I really enjoyed listening to the first author's talk of their corresponding paper Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithms, so want to take a closer look at the actual paper today and read through any of the details that might have been left out in the talk.
Day 285/365 of GPU Programming
Earlier this week I really enjoyed listening to the first author's talk of their corresponding paper Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithms, so want to take a closer look at the actual paper today and read through any of the details that might have been left out in the talk.
Day 284/365 of GPU Programming
The Efficient AI computing course I'm following just released lecture 9 on knowledge distillation. So going to temporarily switch back from my NCCL tangent to go through the new lecture and study related material. Also still eagerly waiting the release of Stanford's CS 312 lectures which should hopefully be out soon.
Day 284/365 of GPU Programming
The Efficient AI computing course I'm following just released lecture 9 on knowledge distillation. So going to temporarily switch back from my NCCL tangent to go through the new lecture and study related material. Also still eagerly waiting the release of Stanford's CS 312 lectures which should hopefully be out soon.
Day 283/365 of GPU Programming
Another NCCL resource I'm going through today is from a couple weeks ago at PyTorchCon 2026: Advanced NCCL Tutorial: Beyond Collectives by Jeff Hammond, Hanyue He & Ke Wen from NVIDIA.
Day 282/365 of GPU Programming
Continuing my NCCL studies with another GPU Model talk. This time on Multi-GPU Programming in NCCL and NVSHMEM by Jeff Hammond (a principal engineer at Nvidia).
Day 281/365 of GPU Programming
I also really enjoyed this talk
Demystifying NCCL An In depth Analysis of GPU Communication Protocols and Algorithms by Zhiyi Hu from ETH Zurich at the Hot Interconnects Symposium. Would love to see more resources like this that explore important but under-documented tech.
Day 281/365 of GPU Programming
I also really enjoyed this talk
Demystifying NCCL An In depth Analysis of GPU Communication Protocols and Algorithms by Zhiyi Hu from ETH Zurich at the Hot Interconnects Symposium. Would love to see more resources like this that explore important but under-documented tech.
Day 280/365 of GPU Programming
GPU Mode has several lectures on NCCL, so will be going through one or two of those today starting with the one on the basics of NCCL. The talk is by Daniel Johnson (a RS at Meta), so one part I'm really curious learning more about is his explainer on how PyTorch uses NCCL.
Day 280/365 of GPU Programming
GPU Mode has several lectures on NCCL, so will be going through one or two of those today starting with the one on the basics of NCCL. The talk is by Daniel Johnson (a RS at Meta), so one part I'm really curious learning more about is his explainer on how PyTorch uses NCCL.
