I love Zstd. It is my default goto for anything compression related: it has a sweet spot between availability (it’s now standardized!! RFC8478), complexity, performance and compression ratio. My favorite thing about it are the skippable frames. Just a killer feature. It’s not even a Zstd thing – it’s inherited from LZ4 and compatible with those frames. But it’s immensely powerful to build stuff on top of the compression algorithm. Here’s the idea: Zstd is a framed format, and Zstd files are a collection of one or more frames. Normal zstd frames begin with a magic number and contain compressed data, but the standard also defines the concept of a “skippable frame” that can be inserted between any two zstd frames and it’ll always be ignored by a compliant decoder. All you need to create a skippable frame is a 8 byte header with a magic number (0x184D2A5?) followed by the size of the frame. After that, you can insert anything you want in the middle of the stream! Pictures of cats, cooking recipes… huh. Something interesting I’ve been doing with these frames: at Cursor we use a lot of tarballs because those never go out of fashion. Pods, training, evals… you always have files you gotta move around, and 80s technology cannot be beaten. But of course, we don’t go for the classic `.tar.gz` because, why would you do that when zstd is so widely available? If you do `.tar.zstd`, the resulting tarball is smaller and decompresses faster. It’s just better. But is “faster” fast enough? If you try to fetch and extract a tarball in a modern machine (i.e. one with a good uplink and a NVME drive), you’ll see it’s actually _very_ slow. Do a `curl tarball | zstd -d | tar xf -` and take a look at the system metrics. It’s not even close to saturating neither the available bandwidth nor the IOPS for the host! Can we improve on this? Skippable frames to the rescue. Whenever we build a tarball, we add a 4kb footer at the end of it, which is just a skippable frame containing an index of valid segments for the tarball (i.e. chunks of the tarball that you could download and extract separately). With this, we can do as many `Range` HTTP requests in parallel as we want, and each request can also be extracted in parallel. It’s trivial to saturate bandwidth and IOPS this way, and make fetching tarballs at least 10x faster. And the best thing? The segment index at the end IS an skippable frame after all. If you just fetch the tarball as usual with `curl` and pipe it into `zstd`, it extracts the same. So anywhere in the infra that doesn’t have specialized code to download & extract tarballs in parallel just continues working.