ML Systems @thinkymachines | PhD CS @CarnegieMellon
Seattle, WA
Joined November 2016
- Tweets217
- Following398
- Followers990
- Likes399
Aurick Qiao retweeted
Tinker lets customers turn their expertise into cutting-edge models that they own. One year after launch, we look back at what we’ve shipped and what our users are building.
thinkingmachines.ai/news/one…
Aurick Qiao retweeted
Tinkerers have been busy scaling up long-context RL! We’ve made significant improvements to Tinker’s efficiency to support those, and are passing these on with price cuts up to 70%.
GLM-5.3-Flash and DeepSeek-v4.1-Flash are also live for cost-efficient long-context work.
Aurick Qiao retweeted
So… I have it working now. The core of an inference engine in Rust is about 3k lines (scheduling, kv/swa/recurrent state, prefix cache, multimodal, dynamic batching)
I then asked Codex to implement (with MTP, cudagraphs, TP on Blackwell):
- DeepSeekV4 Flash
- Gemma 3.8 27B
- GPT-OSS
- Inkling
It turns out that most models take a lot of code to run with fast, persistent graphable kernels and so on… on the order of 2.5k for a simple model (gpt-oss) and up to 7k for a monster like DeepSeekV4 Flash that has mHC, CSA, HCA and DSpark
Not sure if I’ll continue with this, but it was fun! The simple core is reusable, and startup times are crazy fast for an inference engine with no tech debt and not having to serve a ton of dynamic configurations - just one model forward pass in nvfp4 :’) - maybe one day SGLang/vLLM will be modular like this.
See interactivity-throughput below for Inkling-Small TP=1 after a few sconv kernel fusions. (This is without overlap scheduling btw! Don’t even really need it when your core + scheduler is small and Rust.)
dumb idea: I wonder if it could be useful to have smaller LLM inference libraries: not a full "inference engine" but an "inference engine engine" of sorts
It does scheduling, metadata planning, parallelism. But you write the forward pass, weight load, cuda graph, + control startup time.
And for the high-level parts (tokenizer, multimodal, scheduler admission, retraction heuristics), feels like we should just let folks write code instead of configuring 100 CLI arguments
Aurick Qiao retweeted
Great to see continued progress on advanced CUDA Graph techniques in SGLang!
The innovative Breakable CUDA Graph, pioneering full CUDA Graph support for prefill, efficient CUDA Graph memory management, and more.
🚀 New blog: Advanced CUDA Graph Techniques in Inference and the design story behind them.
CUDA Graph removes kernel-launch overhead by recording GPU work once and replaying it. But bringing it into a real inference engine introduces challenges in ops compatibility, graph management, and memory overhead.
Together with the @AIatMeta team, we refactored SGLang's CUDA Graph support around a common runner/backend interface to tackle these problems systematically. On top of this architecture, we highlight several techniques first developed in SGLang:
1️⃣Breakable CUDA Graph (BCG), our new default prefill CUDA Graph backend, removes the dependency on torch.compile for faster setup and broader compatibility.
2️⃣Experimental Full CUDA Graph for prefill, first introduced in SGLang for FA4 and FlashInfer, using request padding to enable full graph capture for dynamic prefill workloads.
3️⃣Graph memory reuse across shapes and segments, keeping the CUDA Graph memory footprint fixed and making memory management more predictable as graph coverage expands.
Aurick Qiao retweeted
🚀 New blog: Unified Radix Cache: One Tree for Hybrid Model Prefix Caching
Hybrid models complicate prefix caching: each attention type has its own cache reuse semantics, and specialized cache classes multiply combinatorially, duplicating tree logic as caching features grow.
Unified Radix Cache brings everything into one shared tree:
1️⃣ One token-keyed radix topology for canonical prefix identity, with composable components for architecture-specific reuse semantics
2️⃣ HiCache and session-aware eviction are natively supported in the shared tree, built for agentic workloads
3️⃣ A clean separation between the controller and tree core makes the core replaceable, including an experimental Rust tree core implementation
One tree, composable cache semantics, and a unified foundation for hybrid model prefix caching.
Aurick Qiao retweeted
Releasing weights indiscriminately isn't safe. Neither is keeping capable models inside a few labs.
We think there's a path between them. We haven't mapped all of it. Our new post covers the part we can see: how we assessed Inkling, and why access should widen in stages.
thinkingmachines.ai/blog/a-s…
Aurick Qiao retweeted
Inkling Small was the first model I got to test end to end at ARC Prize. Its cost-to-performance ratio was so good that I thought I made a mistake at first 😂.
Inkling Small from @thinkymachines on ARC-AGI (Verified):
- ARC-AGI-2: 40.1%, $0.23/task
- ARC-AGI-1: 84%, $0.11/task
Inkling Small is the highest-scoring open-weight model evaluated by ARC Prize on both ARC-AGI-1 and ARC-AGI-2, setting a new cost-performance frontier.
Aurick Qiao retweeted
Day 0 SGLang support for Inkling-small and we got 648 tok/s decode with DSpark!
Cookbook and HF link below ⬇️
Inkling-small is out today! With SGLang, you can get 648 tok/s decode with DSpark (simulated acc len=4) and 288 tok/s w/o DSpark, under the same setup (8x @NVIDIAAI B200, TP 8, NVFP4, bs=1).
What makes this model different is the size. 276B total with 12B active is a sweet spot for RL, and both LoRA and full-parameter training become well within reach. Miles is ready and verified for multimodal RL on Inkling-small, so you can turn your multimodal data into real capability gains.
At ~1/4 the size, Inkling-small matches the bigger version in capability and even wins on some benchmarks.
Run Inkling-small with SGLang, and customize it with Miles.
Aurick Qiao retweeted
Whereas I felt like it took a village to release inkling, inkling-small felt much more routine 😆 We just took the pipeline used for Inkling, passed in a smaller model, and voila - new model! Inkling small benefited quite a bit vs Inkling from some minor improvements, but there's still so much more left in the tank...
Today, we are releasing Inkling-Small.
Inkling-Small achieves comparable performance to Inkling at a quarter of its size. It features 276B total parameters, 12B active. We are making the full weights available.
thinkingmachines.ai/news/ink…
Fine-tune it on Tinker today, or chat with it in text, image, and audio on Tinker Playground.
Aurick Qiao retweeted
Inkling-Small is comparable to Inkling at a quarter the size. Weights are open, fine-tunable on Tinker today. Look forward to seeing what people make with it.
Today, we are releasing Inkling-Small.
Inkling-Small achieves comparable performance to Inkling at a quarter of its size. It features 276B total parameters, 12B active. We are making the full weights available.
thinkingmachines.ai/news/ink…
Fine-tune it on Tinker today, or chat with it in text, image, and audio on Tinker Playground.
Aurick Qiao retweeted
Today, we are releasing Inkling-Small.
Inkling-Small achieves comparable performance to Inkling at a quarter of its size. It features 276B total parameters, 12B active. We are making the full weights available.
thinkingmachines.ai/news/ink…
Fine-tune it on Tinker today, or chat with it in text, image, and audio on Tinker Playground.
Aurick Qiao retweeted
If you've wondered where I've been recently, I gave an interview. This is a very difficult time for me. I'm trying to bring my mom home, and I could use your help spreading the word. Please watch, and please share.
interview: youtube.com/watch?v=JlK2a6yv…
website: bringmymomhome.com/
Aurick Qiao retweeted
We trained a DSpark speculator for Inkling NVFP4, built end-to-end with SpecForge on a live SGLang target engine.
- 1.89x decode throughput over non-spec at bs=64 (8xB200, TP8)
- 7–14% faster than Inkling's built-in MTP at the same batch size
- 3.66 mean accept length, up to 4.76 on GSM8K
Under the hood:
1️⃣ 5-layer Qwen3-style DFlash parallel-draft backbone
2️⃣ Rank-256 Markov logit-bias head for intra-block dependencies
3️⃣ Per-position confidence head predicts acceptance
4️⃣ Distilled online from 400K Inkling regenerations
Try it out now! Serving command in the comment.
Aurick Qiao retweeted
Built a podcast clipping app with Inkling from @thinkymachines ✨
The model is exceptional at reasoning over long-form audio - so I have it listen to full episodes and direct FFMPEG on which clips to cut.
You can have it choose the best moments or search for specific topics 👇
Aurick Qiao retweeted
Inkling from @thinkymachines on ARC-AGI (Verified)
- ARC-AGI-2: 36.5%, $0.64/task
- ARC-AGI-1: 79.5%, $0.30/task
As of today, Inkling is the highest-scoring open-weight model evaluated by ARC Prize on both ARC-AGI-1 and ARC-AGI-2.
Aurick Qiao retweeted
Introducing Inkling from Thinking Machines Lab on Together AI.
A multimodal MoE model built for token-efficient reasoning, with native text, image, and audio input support. Supports controllable inference effort, optimized with Together’s FlashAttention-4-based kernel.
Aurick Qiao retweeted
Inkling, @thinkymachines' first open model, dropped today: 975B total / 41B active MoE, up to 1M context, reasoning natively over text, images, and audio.
Serving and RL support are already live: you can run and shape it on an open stack, starting now.
Day 0 support on SGLang @sgl_project and Miles @radixark👇
- Inkling's new architecture (ShortConv, attention with relative positional embedding, shared expert sink MoE) is natively implemented and deeply optimized, with prefill full CUDA graph and MXFP8 KV cache
- Full parameter and LoRA RL in a customized Megatron backend, train inference consistency via customized kernels, routing replay, and cross-runtime parameter synchronization
- DFlash speculative decoding from @modal for low-latency serving
Launching now, blog and cookbook in the comments ⬇️
Aurick Qiao retweeted
We trained a custom DFlash speculator for Inkling, running on SGLang. Many thanks to the team there for their optimization work.
Inkling, @thinkymachines' first open model, dropped today: 975B total / 41B active MoE, up to 1M context, reasoning natively over text, images, and audio.
Serving and RL support are already live: you can run and shape it on an open stack, starting now.
Day 0 support on SGLang @sgl_project and Miles @radixark👇
- Inkling's new architecture (ShortConv, attention with relative positional embedding, shared expert sink MoE) is natively implemented and deeply optimized, with prefill full CUDA graph and MXFP8 KV cache
- Full parameter and LoRA RL in a customized Megatron backend, train inference consistency via customized kernels, routing replay, and cross-runtime parameter synchronization
- DFlash speculative decoding from @modal for low-latency serving
Launching now, blog and cookbook in the comments ⬇️
Aurick Qiao retweeted
The best base model for training isn't about someone else's benchmarks.
Excited to offer @thinkymachines' first open-weights model: Inkling. 975B MoE, multimodal, with Apache 2.0. A great foundation for fine-tuning. Available day-zero.
fireworks.ai/models/firework…
Today, we are introducing Inkling.
Inkling reasons efficiently across text, image, and audio modalities. We are making the full weights available.
thinkingmachines.ai/news/int…
Available today for fine-tuning on Tinker. Play with it in the Inkling Playground. 🧵
