Building TIRx until agents take my job. https://xitter.cf/t.co/pZw7abwags CS Ph.D. student @ CMU
Pittsburgh, PA
Joined August 2020
- Tweets93
- Following130
- Followers1.3K
- Likes207
Bohan Hou retweeted
Kernel agents can already beat hand-tuned kernels. Spinning up more agents is easy, but scaling GPU evaluation is still hard and expensive.
What happens when the agents scale faster than the GPUs?
Meet KCoral 🪸: a lightweight shared GPU benchmark environment for agentic kernel development.
🧑💻 Agents keep writing and revising code in their own workspace.
⚡ Evaluation is sent to KCoral, which schedules a shared GPU pool and returns correctness and timing.
🌐 Scale across multiple GPUs and nodes through a unified API endpoint.
🔒 Per-request isolation with only milliseconds of execution overhead.
By overlapping CPU compilation with GPU work across requests, KCoral further speeds up kernel evaluation. On a single NVIDIA B200, this translates to 2.58× the throughput of sequential local execution.
Bohan Hou retweeted
Today, LithosAI is launching a public preview of LithosBox: ultra fast versioned sandboxes that your agents can snapshot, fork, and restore in milliseconds.
So far, we've been focused on providing ultra-low latency inference for open models like Kimi K3, DeepSeek V4.1, and GLM 5.3 Flash (stay tuned!). Now, we're giving agents runtime and post-training environments with Git-like control.
With LithosBox, agents can:
→ spin up, branch, and snapshot thousands of environments as they explore multiple paths forward
→ decide on a winner and rebuild the winning state
→ then throw away the environments they don't need, all in milliseconds
@dimitriosSkar, @JiaZhihao, and the rest of our team put together this agent vs. agent showdown with Doom so you can see how LithosAI helps your agents run 5x faster with AI serving infrastructure co-designed for ultra-low latency across inference and execution environments.
In our demo, the agent completes its entire run 5.29x faster with LithosBox and Lithos DeepSeek V4.1 Flash inference than it does with leading sandbox and inference providers.
Try LithosBox for free for up to two weeks via our public API.
2.94× KDA fwd, 6.84× KDA bwd, 2.59× MSA prefill, 3.99× MSA decode, 1.33× KDA decode, 1.71× MLA, 1.68× VSA — just by pressing Enter.
After years of working on ML compilers and designing many abstractions, the most useful compiler thing in TIRx I’ve ever built might just be a PTX tablegen.
AI agents are becoming GPU kernel engineers. We're rethinking the compiler stack around them.
Meet TIRx Harness: an open compiler harness that gives agents an environment to explore, debug, and optimize GPU kernels.
🧱 Minimal stable low-level compiler foundation for predictable GPU programming
🔍 Domain-specific compiler analysis to guide debugging and optimization
📚 Kernel zoo of 60+ kernels and hardware specs to reuse strategies and discover new ones
📊 Remote GPU evaluation for reliable benchmarks and profiling
Agent-built Kimi Delta Attention kernels achieved geometric-mean speedups of 2.94x over FlashKDA (forward) and 6.84x over FLA (backward).
Our bet: the next leap in agentic GPU programming will come from engineering the environment agents optimize in.
Check out our blog: blog.mlc.ai/2026/09/29/tirx-…
Bohan Hou retweeted
🚀 Kernel Design Agents to Optimize Kimi Delta Attention (upto 2.96x)
When the KDA (Kernel Design Agents) project was initiated, its name happened to coincide with another KDA (Kimi Delta Attention). More than once, people asked us , "Can we use KDA to write a version of KDA?" Today, right at a moment with a full moon good for shooting, we are thrilled to share the latest results of KDA(gent)-v0.6: using KDA(gent) to optimize KDA(ttn)!
KDA-v0.6 achieves this through
* Humanize2 Flow & LLM Collaboration Optimization: Introducing multi-round iterative evolution with Flame Chase, GPT-5.6-Sol, and Fable-5, paired with an automated Workspace cleanup mechanism to significantly broaden the operator search space.
* Multilingual Domain-Specific Skill Stack Support: Fully supporting CuTe-DSL, CUDA C++, as well as recent Agent-Native CAKE IR and TIRx. Combined with customized diagnostic tools (such as CuTe's dedicated IKET pipeline analyzer and TIRx's CPU-side numerical simulation and static checks), achieving deep performance bottleneck diagnostics.
* Self-Evolving Kernel Wiki: Building a dynamically updated knowledge base that automatically cleans up invalid/erroneous implementations, corrects classification tags, and streamlines search indexing, significantly improving the Agent's knowledge retrieval quality and code generation accuracy.
KDAgent ultimately optimized the KDAttn kernel to 2.96x of FlashKDA, and we face a lot of unexpected hacks during the optimization. More details shared in our blogs nvlabs.github.io/kda/blog/20…
Release results of KDA for KDA: github.com/NVlabs/kda/tree/2…
Community & Production Deployment:
All kernels are open-sourced for community verification! For deploying NVIDIA Agentic CUDA acceleration in production environments, we recommend using the production-level version rigorously verified by FlashInfer.
By the Team: Dongyun Zou, Yixin Dong, Hongyi Jin, Junxian Guo, Yahui Cui, Avery Huang, Zihao Ye, Junru Shao, Changye Li, Zijian Zhang, Sihao Liu, Song Bian, Ligeng Zhu.
Bohan Hou retweeted
🚀We’ve been pushing agentic inference toward the physical limits of the hardware.
Announcing LithosAI’s first pricing tiers, with early-access pricing ahead of the September 1 API launch.
Kimi K3 is live now at 800+ tokens/sec/user on standard GPUs, with full model quality.
Try the live demo at lithosai.com and sign up for early access.
🚀Tired of slow agents? We’re launching LithosAI’s ultra-fast Kimi K3 pricing tiers.
If you’re building agents where latency matters, we can give you the fastest inference available across models, through a hosted API or on-prem.
Kimi K3 chat is live today at 800+ tokens/sec/user. Try it now.
Early sign-ups are open with an early-access discount ahead of the public API launch on September 1. Our Base tier is also coming to OpenRouter.
The countdown to the LithosAI API starts now.
Sign up today at lithosai.com
Bohan Hou retweeted
Do we have Kimi K3 Artificial Analysis @ArtificialAnlys results? Yes, 295 tok/sec.
Do we use funky quantization to get there? No. Native precision, full model quality.
Is the pricing based on unrealistic batch-size-1 assumptions? No. Ping us for pricing.
Is this a public API? Not yet, we are ramping capacity. Stay tuned.
Is there more coming? You bet 🚀
Bohan Hou retweeted
Our paper "XGBoost: A Scalable Tree Boosting System" received the Test of Time Award at #KDD2026! The honor belongs to the entire community that made @XGBoostProject today. I will be giving a talk tomorrow at the award session on the paper, history, and lessons learned
Bohan Hou retweeted
🍰 and another CAKE kernel merged into FlashInfer: SM100/SM103 tinygemm2, up to 1.79x faster on large shapes, ~18-23% geomean kernel-time reduction, e2e verified.
translates to up to 7.6% end-to-end serving speedup in SGLang.
github.com/flashinfer-ai/fla…
BTW, I noticed that the CuTe DSL 4.7 PR includes a static analysis tool for catching certain synchronization bugs. I expect we’ll see many more tools like this going forward. :)
Very interesting post. One optimistic thought:
Models will keep getting better—GPT-6, 7, 8, whatever comes next.
But if humans still need to review every GPU kernel they generate, the real bottleneck remains.
I’ve seen PRs add a single PTX instruction like fence.proxy.async to fix bugs that only appear after hours or days on hundreds of GPUs.
For example, FlashKDA relased last week: github.com/MoonshotAI/FlashK…
I already run 3–5 agents in parallel.
The slowest part is still me.
I still have to read the code.
So how far are we from a truly human-free GPU kernel development loop?
And if we still need tooling around generated code—even at the PTX level—then abstractions haven’t disappeared.
They’ve just moved from the programming interface into the verification infrastructure.
Very interesting post. One optimistic thought:
Models will keep getting better—GPT-6, 7, 8, whatever comes next.
But if humans still need to review every GPU kernel they generate, the real bottleneck remains.
I’ve seen PRs add a single PTX instruction like fence.proxy.async to fix bugs that only appear after hours or days on hundreds of GPUs.
For example, FlashKDA relased last week: github.com/MoonshotAI/FlashK…
I already run 3–5 agents in parallel.
The slowest part is still me.
I still have to read the code.
So how far are we from a truly human-free GPU kernel development loop?
And if we still need tooling around generated code—even at the PTX level—then abstractions haven’t disappeared.
They’ve just moved from the programming interface into the verification infrastructure.
Don’t feel bad about causing infra folks pain—NVIDIA does that every 2–3 years anyway :)
Presenting my grand unified theory of ML researcher impact: Your impact is directly proportional to how much pain you cause to infra.
Fundamentally, you can only inflict pain upon infra if your approach actually works. And the better your approach works the more pain infra is forced to endure.
So, to give some examples:
- MoE's add a ton of data-dependent computation => pain (shazeer++)
- GDN/KDA are the most complex architecture I've been forced to care about and a very annoying matrix inversion => pain (sonta++)
- Muon is much more annoying than Adam and causes annoying restrictions on parallelism => pain (keller/jeremy++)
- RL scaling forced many researchers to care about LLM inference and RL infra as a category => pain (tworek++)
Even papers like Attention Is All You Need have lead to significant pain! Before transformers were invented everyone was running small jobs and I never needed to think about kv-caches or 6D parallelism.
Bohan Hou retweeted
Presenting my grand unified theory of ML researcher impact: Your impact is directly proportional to how much pain you cause to infra.
Fundamentally, you can only inflict pain upon infra if your approach actually works. And the better your approach works the more pain infra is forced to endure.
So, to give some examples:
- MoE's add a ton of data-dependent computation => pain (shazeer++)
- GDN/KDA are the most complex architecture I've been forced to care about and a very annoying matrix inversion => pain (sonta++)
- Muon is much more annoying than Adam and causes annoying restrictions on parallelism => pain (keller/jeremy++)
- RL scaling forced many researchers to care about LLM inference and RL infra as a category => pain (tworek++)
Even papers like Attention Is All You Need have lead to significant pain! Before transformers were invented everyone was running small jobs and I never needed to think about kv-caches or 6D parallelism.
Bohan Hou retweeted
Today, SkyPilot is out of stealth.
Building custom intelligence is now existential. We help frontier AI teams build intelligence faster by removing their biggest bottleneck: AI compute fragmentation.
Frontier teams like @appliedcompute, @AbridgeHQ, @hippocraticai, @hcompany_ai, and @nubank already run on SkyPilot, with 10x faster time-to-intelligence and double-digit increase in GPU utilization.
AI teams today get compute anywhere they can. They then firefight compute fragmentation across providers. Researchers burn time on workload setup. Infra gets paged when GPUs go down. Frontier teams build slowly even on the fastest compute.
@skypilot_org turns your fragmented compute into one AI supercomputer, so you run frontier workloads faster. Many users manage 10,000+ GPUs across providers with SkyPilot. GPU hours consumption has grown 6x in the last 6 months.
1/ We're launching SkyPilot Platform — the AI compute platform for frontier AI teams to manage large GPU fleets and accelerate building custom intelligence.
Optimized for fleet management, team governance, and frontier workloads — pretraining, post-training, multi-cluster serving, and sandboxes. SkyPilot open source users can switch to the platform with a server URL change.
2/ We've raised over $20M led by @Lux_Capital (@breeves08), with participation from @AmplifyPartners (@dauber, @lennypruss), @coatuemgmt, @FoundationCap (@ashugarg , @JayaGup10), @RaceCapital, @thehousefund, and top operators like @alighodsi (CEO, Databricks), @JeffDean (Chief Scientist, Google), @rauchg (CEO, Vercel), @amasad (CEO, Replit), @ClemDelangue (CEO, @huggingface) and more.
We're hiring across Engineering and GTM to deliver the platform for the next decade of AI.
Above all, I'm excited to be building with the incredible team we've assembled, along with my cofounders Zhanghao @Michaelvll1, Romil @bromil101, Scott, and Ion @istoica05.
If you firefight AI compute, let's build.
ALT Today, SkyPilot is out of stealth. We help frontier AI teams build intelligence faster by removing their biggest bottleneck: AI compute fragmentation.
Bohan Hou retweeted
Today we're launching Lithos’ agentic inference engine starting with Kimi K2.7 Code, a frontier open-source 1T parameter coding model.
On a single standard 8xB200 GPU node, performance peaks above 1,000 tok/s per user on coding workloads at model-native precision, with no approximation and full model quality.
Raw speed is the agent differentiator now.
Check it out at lithosai.com