Comparing MiMo-V2.6-Flash in DS4 to the Xiaomi API. More work to be done on long context :) Still cool that a single M3 Ultra can compete on decode performance. Also, prefill on M5 Ultra should be much better.

And here's MiMo-V2.6-Pro-RL in DS4 on 2x M3 Ultra

Here's MiMo-V2.6-Flash-RL in DS4 on M3 Ultra

Pre-orders for M5 Ultra 256GB already at 16-18 weeks shipping. Hopefully this is not bad news for the 512GB version...

You can run the full DeepSeek V4 Pro 0813 in native precision at a fairly decent speed, locally on 2x Mac Studios using DS4. Also worth noting that this model beats Mythos Preview at CyberGym (83.3 vs 83.1)

Some throughput benchmarks on one and two machines. Tensor parallelism via RDMA over Thunderbolt improves generation 1.5x and prefill 1.8x.

Per-token divergence from the full model, by span. It picks the same next token 95.5% of the time on tool calls and 87.4% on the final written answer, with the thinking spans at 78.1% carrying most of the cost.

Relative perplexity on 8 codebases across 5 languages, against the full 1.56TB release. The Linux kernel stays within 5% of the baseline.

@antirez @ivanfioravanti some tweaks to the DS4 TP support in case its useful github.com/kernelpool/ds4/tr…

Benchmarks for the Kimi K3 2bit quant thanks to @ivanfioravanti's llm_context_benchmarks tool. Performance holds up nicely over longer context thanks to KDA/MLA 🚀