built an LLM inference engine from scratch. vLLM / SGLang / mistral-rs contributor. I post serving internals: KV cache, batching, quant, kernels.
Cayman Islands
Joined February 2015
- Tweets1.7K
- Following52
- Followers94
- Likes1.1K
Pinned Tweet
I'm building an LLM inference engine from scratch, in public. Not using vLLM or TGI. The goal: re-derive every serving trick until I understand how modern inference squeezes throughput from a GPU. I call it nanoserve: github.com/pjdurden/nanoserv…
jevkit-calibrate is at 501 downloads across npm and PyPI.
unofficial. reliability diagrams and ECE on your data.
npm npmjs.com/package/jevkit-cal…
pypi pypi.org/project/jevkit-cali…
checks @typesafeai's jev calibration.
Nvidia's DSX POD could cut AI data center setup from six months to as little as 14 days 🤯
Six months' lead time leaves GPUs in crates. Two weeks moves the bottleneck to filling the room with hardware serving tokens cheaply.
'As little as' does lots of work.
Webhooks is down right now 🚨
GitHub status page: "Delays for Pull Requests and Webhook deliveries" since 10:52 AM ET.
@github
Inception's Mercury 3 claims it beats GPT-6 Luna and Claude Haiku 5.5 at a fraction of the wait.
Diffusion decoding writes many tokens per pass, so its latency curve doesn't match autoregressive serving.
Speed claims are cheap. Show me p99 under batch load.
OpenAI models are coming to Google Cloud 👀
Same models, one more place to run them. Serving cost per token is about to get very interesting.
claude-skills-live, 21 Oct
+32 added, 4,266 indexed.
New: CCometixLine (3.5k stars), AutoSci (1.7k stars).
github.com/pjdurden/claude-s…
Day 88 of building an LLM inference engine from scratch.
My benchmark joined a booking's sweeps on the p50 speedup only. vLLM's benchmarks report p99 next to p50 for a reason: CUDA graphs earn their keep in the tail. Now the join takes a column.
github.com/pjdurden/nanoserv…
The gotcha: a two-arm sweep has no split column. Printing - there would say "this sweep did not run this load", which is not why the cell is empty.
So the column is refused, naming the sweep that lacks it, before a single table prints.
Also: three of my "refused" tests passed before I wrote any code. --join was an unknown flag, so argparse already exited 2 with --join in its usage line.
Green for the wrong reason. Each now asserts the exact sentence. 2531 green. #buildinpublic
Put two benchmark sweeps in one table and the rows stop meaning one thing.
nanoserve Day 87: with more than one sweep, --table joins rows by load, one itl_p50_speedup column per sweep. 2520 green.
Small diff. The table is the part people actually read.
SpaceX's AI business is beating expectations.
Revenue is the easy number. The one I'd watch is what it costs to serve each token. That's where the margin lives.
Mistral just dropped Voxtral 3.
- A full voice agent stack
- Listens, thinks, and talks in real time
Real-time voice lives or dies on the latency budget, not the model. Curious how they hit it end to end.
DeepSeek just open-sourced the inference engine behind its fastest, cheapest model.
If you serve models yourself, this is the part I'd read first. The scheduler and KV cache decisions are where per-token cost actually lives.
Weights were never the whole story.
jevkit-bench: 503 downloads across npm and PyPI.
scores accuracy and cost, diffs two runs.
npm npmjs.com/package/jevkit-ben…
pypi pypi.org/project/jevkit-benc…
github github.com/pjdurden/jevkit-j…
built on @typesafeai's jev
OpenAI says GPT-6 is much better with legacy languages like COBOL and Fortran. 😅
Somewhere a mainframe team is going to have a very interesting quarter.
For anyone serving that on their own GPUs: legacy codebases are long context. Plan for the KV cache bill.
NVIDIA just dropped cosmos3-nano-libero10-sdg-export-S5_2_s43 on Hugging Face 👀
> 15B parameters
> robotics
> license: other
@NVIDIAAI @huggingface
BREAKING: US orders federal agencies to remove DeepSeek AI from their systems.
Open weights mean the model doesn't vanish when one vendor gets banned. It just moves to whoever has the GPUs.
Banning the endpoint is easy. Banning the weights is not.
Anthropic is reportedly in talks for yet another $45B compute deal with Nscale.
Training gets the headlines. Serving every token on that much compute is where the money actually burns.
claude-skills-live, 20 Oct
+28 added, 4,234 indexed.
New: hyperframes (68k stars), xerj (10.8k stars).
github.com/pjdurden/claude-s…