Professor at IST Austria
Vienna
Joined May 2022
- Tweets179
- Following311
- Followers1.9K
- Likes96
Dan Alistarh retweeted
Disaggregated quantization for LLMs 🧵
LLM prefill and decode can be processed by separate weights in different precisions. Boosts speed AND accuracy. Offloading makes it zero-overhead on long sequences for dense models. Applicable to existing quantized checkpoints.
Paper: arxiv.org/abs/2609.26333
Code: github.com/IST-DASLab/disagg…
Qwen3.8 NVFP4 prefillers: huggingface.co/ISTA-DASLab/Q…
Dan Alistarh retweeted
We’re excited to share our #ICML2026 paper on optimizer robustness in Async Pipeline Parallel LLM pretraining
Main message: with the right PP schedule, optimizer choice, and Error Feedback correction, Async PP can match sync training quality even at 10B MoE scale
[1/10]
ALT Training loss on a 10B MoE model trained for 200B FineWeb-Edu tokens. Both asynchronous runs remain stable throughout training, with Async PP + Error Feedback closing the final gap to the synchronous baseline entirely.
Weight-only quantization powers local LLMs like llama.cpp or Ollama. But SOTA quantized accuracy requires complex kernels that are notoriously hard to implement.
Can we get SOTA accuracy and keep things simple? Our new GSQ (Gumbel-Softmax Quantization) method says yes. 🧵
🚀 Scalar structure means GSQ can use highly-optimized low-precision GEMM kernels:
Using vLLM + the excellent Humming kernels (github.com/inclusionAI/hummi…) on L40s GPUs, 2-bit GSQ-quantized Llama-3.1-70B hits up to 6.2× throughput vs BF16!
As always, our release is fully open:
💻 Paper and code: arxiv.org/pdf/2604.18556
@huggingface : huggingface.co/collections/I…
Credits: Alireza Dadgarnia, Soroush Tabesh, Mahdi Nikdan, Michael Helcig, Eldar Kurtić, and Max Kleinegger
We thank @RedHat_AI and @verdacloud for the support!
Dan Alistarh retweeted
TurboQuant has drawn a lot of attention recently, but the accompanying evals didn't tell the full story.
So we ran what I believe is the first comprehensive study of TurboQuant: where it helps, where it falls short, and how it impacts accuracy, latency, and throughput.
Findings:
Dan Alistarh retweeted
🚀 Our #ICLR2026 paper: The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm
We show GPTQ is exactly Babai's nearest plane algorithm, giving a geometric view of LLM quantization and inspiring improved PTQ methods.
Efficient GPTQ Triton kernels included!
Speedrunning GPT-2 is now routine thanks to @karpathy.
But can we speedrun GPT3-175B?
We attempted to match accuracy on a <$10K budget; while we didn't quite reach it, our first results show that quality data, engineering, and native FP4 can get close.
Details in 🧵
Credit goes to Erik Schultheis, Matin Ansaripour @matin_asp, Andrei Panferov @black_samorez, and George Vlassis @gvlassis98. Thanks to @verdacloud (particularly Paul Chang) for compute support, and Jen Iofinova for safety testing. This work was supported by FWF BilAI and SwissAI.
As usual, our release is fully open:
- Full report: github.com/IST-DASLab/Clover…
- QuartetII paper: arxiv.org/abs/2601.22813
- Tokenized data: huggingface.co/datasets/dasl…
- Model: huggingface.co/daslab-testin…
- Codebase: github.com/IST-DASLab/Clover…
We’re releasing MatGPTQ (Matryoshka GPTQ) an accurate and efficient post-training quantization (PTQ) method that jointly optimizes a single model across multiple bit-widths, producing a sliceable checkpoint that can be deployed across diverse hardware and memory budgets. [1/4]
On the systems side, we provide custom CUDA kernels for Ampere GPUs and vLLM integration (see below for OSS code) to fully leverage nested models:
- 3x-5.6x kernel speedups when memory-bound
- 1.5x-3.5x end-to-end speedups in vLLM
Credits go to @maxkleinegger and Elvir Crnčević!
- Paper: arxiv.org/abs/2602.03537
- Models: huggingface.co/collections/I…
- Code: github.com/IST-DASLab/MatGPT…
