DeepSeek just published DSpark, a speculative decoding system that boosts live DeepSeek V4 serving throughput by 51% to 406% under stricter latency targets. With how most speculative decoding methods draft more tokens, but waste verification compute when those tokens get rejected, DSpark fixes this with a semi autoregressive drafter for more coherent long drafts, plus a confidence scheduler that only verifies prefixes likely to survive. It also gives 60% to 85% faster per user generation at matched throughput.

Speculative execution for LLMs is an excellent inference-time optimization. It hinges on the following unintuitive observation: forwarding an LLM on a single input token takes about as much time as forwarding an LLM on K input tokens in a batch (for larger K than you might think). This unintuitive fact is because sampling is heavily memory bound: most of the "work" is not doing compute, it is reading in the weights of the transformer from VRAM into on-chip cache for processing. So if you're going to do all that work of reading in all those weights, you might as well apply them to a whole batch of input vectors. I went into more detail in an earlier thread: xitter.cf/karpathy/status/… The reason we can't naively use this fact to sample in chunks of K tokens at a time is that every N-th token depends on what token we sample at time at step N-1. There is a serial dependency, so the baseline implementation just goes one by one left to right. Now the clever idea is to use a small and cheap draft model to first generate a candidate sequence of K tokens - a "draft". Then we feed all of these together through the big model in a batch. This is almost as fast as feeding in just one token, per the above. Then we go from left to right over the logits predicted by the model and sample tokens. Any sample that agrees with the draft allows us to immediately skip forward to the next token. If there is a disagreement then we throw the draft away and eat the cost of doing some throwaway work (sampling the draft and the forward passing for all the later tokens). The reason this works in practice is that most of the time the draft tokens get accepted, because they are easy, so even a much smaller draft model gets them. As these easy tokens get accepted, we skip through those parts in leaps. The hard tokens where the big model disagrees "fall back" to original speed, but actually a bit slower because of all the extra work. So TLDR: this one weird trick works because LLMs are memory bound at inference time, in the "batch size 1" setting of sampling a single sequence of interest, that a large fraction of "local LLM" use cases fall into. And because most tokens are "easy". References arxiv.org/abs/2302.01318 arxiv.org/abs/1811.03115 arxiv.org/abs/2211.17192

MidJourney just announced... a full body ultrasound! Yup... read on because it's as crazy as it sounds. "As powerful as MRI and as casual as a trip to the spa" They are calling it "the @midjourney scanner" Insane details: - First, the scale. The device uses 8,960 individual transducers arranged in a ring around your body - The precision is the most jaw-dropping part: it resolves motion at the picometer range. It can image internal tissues finer than the width of an atom. We are talking sub-atomic level diagnostic capability - The compute requirement is massive. The system processes 17 gigabytes of data per second. It takes 40GB of raw data to reconstruct just one cross-sectional slice. And they are planning to scan 100 slices? - Midjourney claims that fewer than 12 of these machines could perform more full-body scans than every MRI machine on Earth combined. Welcome to the future of healthcare! Not only these scanners are announced, they will exist in a "Midjourney SPA" - with hot tubs, saunas, cold plunges, and 9-10 whole body scanners.