Intelligence per picojoule, with @itsclivetime and @dylan522p (0:00) Intro (1:22) What is codesign? (2:49) Codesign example: Swish vs ReLU (4:22) Are DeepSeek papers codesign? (6:45) Predicting where ML research will go (8:06) Should researchers hate your chips? (9:34) Can you codesign too much? (13:23) Picking the right grain size for specialization (16:22) How much hardware flexibility for The Age of Research? (20:05) Did reasoning and RL disrupt hardware roadmaps? (23:09) Cerebras/Groq: unexpected wins on reasoning and RL (25:34) Disaggregating MLP and attention (29:06) The right metrics for quantization and codesign papers

I chatted with @ysmulki about MatX, chip design and where silicon designed for LLMs is headed (8:17) Tightly coupling SRAM and HBM on one chip (14:03) More MoE FLOPS, smaller KV cache load (16:08) Numerics: from 32-bit to 4-bit (19:02) Targeting both training and inference (22:14) Chip timelines (27:15) Logic and memory scarcity (29:42) Compute costs (32:07) Latency: from 20ms to 1ms as the new table stakes (40:50) Programming the chip (43:00) Starting MatX (47:11) Codesign without seeing the models (51:57) Interconnect design (55:44) Performance modeling philosophy (1:07:02) Prefill vs. decode (1:13:47) What's next

Put your KVs in HBM! Contexts are growing ~infinitely long, and HBM is the best place to fit them. Definitely won't be underutilized: *everyone* is running out of HBM capacity.

We’re building an LLM chip that delivers much higher throughput than any other chip while also achieving the lowest latency. We call it the MatX One. The MatX One chip is based on a splittable systolic array, which has the energy and area efficiency that large systolic arrays are famous for, while also getting high utilization on smaller matrices with flexible shapes. The chip combines the low latency of SRAM-first designs with the long-context support of HBM. These elements, plus a fresh take on numerics, deliver higher throughput on LLMs than any announced system, while simultaneously matching the latency of SRAM-first designs. Higher throughput and lower latency give you smarter and faster models for your subscription dollar. We’ve raised a $500M Series B to wrap up development and quickly scale manufacturing, with tapeout in under a year. The round was led by Jane Street, one of the most tech-savvy Wall Street firms, and Situational Awareness LP, whose founder @leopoldasch wrote the definitive memo on AGI. Participants include @sparkcapital, @danielgross and @natfriedman’s fund, @patrickc and @collision, @TriatomicCap, @HarpoonVentures, @karpathy, @dwarkesh_sp, and others. We’re also welcoming investors across the supply chain, including Marvell and Alchip. @MikeGunter_ and I started MatX because we felt that the best chip for LLMs should be designed from first principles with a deep understanding of what LLMs need and how they will evolve. We are willing to give up on small-model performance, low-volume workloads, and even ease of programming to deliver on such a chip. We’re now a 100-person team with people who think about everything from learning rate schedules, to Swing Modulo Scheduling, to guard/round/sticky bits, to blind-mated connections—all in the same building. If you’d like to help us architect, design, and deploy many generations of chips in large volume, consider joining us.

MatX hardware will maximize intelligence per dollar for the world’s largest models. We are a team of 50+ and growing quickly. If you are passionate about building the best chips for LLMs, consider joining us. matx.com/jobs

MatX is designing chips and systems to 10x the computing power for the world’s largest AI workloads. Today, we are pleased to announce the closing of a >$100M Series A funding round led by @sparkcapital, with participation from @JaneStreetGroup, @danielgross and @natfriedman, @TriatomicCap, @HarpoonVentures, and @adamdangelo. In two years, we proved out all our technical bets across ML numerics, chip design and implementation, software, and system design—and secured all the necessary partnerships—to develop our chip. With this round of investment, we are now sufficiently funded to bring our systems to market.

1. Breakdown of DeepSeek V3 efficiency vs Llama 3: - Better: 11x fewer FLOPs per token, thanks to MoE [37B vs 405B activated params] - Better: 2x faster numerics [fp8 vs bf16 training] - Worse: 0.5x flops utilization [16% vs 33% end-to-end MFU*] - Neutral: similar hardware platform [H800 and H100 both have 2Pflops/s dense fp8] - Neutral: same training data volume [14.8T vs 15T tokens] Llama 3’s design was obviously and intentionally conservative: dense model (not MoE), bf16 training (not fp8), GQA attention (not cheaper alternatives). DeepSeek benefited by being aggressive on all these fronts, at the cost of being later to market. 2. The core algorithmic improvements were already known; the closed source LLM labs were probably already doing similar things. DeepSeek’s improvements are real, but far more modest than the Llama comparison would suggest; my wild guess is closer to 1.5x improvement. MoE was published in 2017; in 2021 Switch Transformer reported 7x speedups vs dense models, similar to DeepSeek’s 11x. OpenAI is widely rumored to have been using MoE models for years. NVIDIA published their fp8 training paper in 2022. 3. NVIDIA’s stock price is down 15% after DeepSeek. Should it be? LLM compute is like a gas: it expands to fill the available budget. Over the last 3 years the labs have grown their budgets, despite algorithms and hardware improving. There’s no reason to expect this to change now: you win by making the best model, not by shrinking your budget. The more meaningful question: do algorithmic improvements like DeepSeek’s mean that margins will shift from hardware vendors to labs? Hard to see why. Algorithmic improvements are quickly copied from one lab to another, making it hard for them to maintain technological differentiation. Hardware improvements take much longer to copy.

MatX will be at MLSys. Come join us at our After Hours in Santa Clara to talk about chips, compilers, partitioning, and optimizing ML models for future hardware. Many of us will be there, including me and @mikegunter_. Tuesday May 14th at 4pm, see matx.com/meetmatx.