Zero copies. One copy. Two copies… Which one is better? Zero, of course. Zero-copy is an optimization that has to do with how we copy data in the OS. The problem is simple: Many applications, Kafka included, have the job of basically: 🔸1. reading files from the disk 🔸2. serving that data over the network as a response. When we get into the internals of it all, we see that the actual bytes get copied a bunch of times: 1. disk → OS read buffer 2. OS read buffer → application buffer 3. application buffer → socket buffer 4. socket buffer → NIC buffer Four copies! That being said, some of these copies are more efficient than others because they use the DMA engine. DMA stands for Direct Memory Access. It’s a feature in computers memory controllers which allows certain hardware (sound cards, graphics cards, network cards, etc.) to access the memory (RAM) without the CPU’s involvement. Additionally, we have a bunch of user <->kernel mode switches which aren’t free either: 1. user -> kernel mode 2. kernel -> user mode 3. user -> kernel mode 4. kernel -> user mode after response is written out So in total - 4 context switches and 4 copies of the data. Do we need all of this? 💡 Step 1: Disk -> OS Read Buffer. The OS read buffer may seem unnecessary here - we could just copy directly into the application. In a single isolated operation, the OS read buffer does not make sense - but throughout the continuous functioning of the system it shines. Why? Caching. Readahead caching is when the OS caches more parts of a file that you actually wanted to read. This is why linear reads on HDDs can be so efficient - the OS typically reads more than requested amount, and the rest is left in the cache. On the next read, you read from memory and not the disk, resulting in a big performance gain. 🤖 Translated into Kafka - the next fetch request will directly hit the cache and avoid accessing the disk. So let's keep that buffer. 🤝 As for the first context switch associated with it, it is fundamentally unavoidable. We will always need to go to kernel mode to read from the disk. 💡 Step 2: OS Read Buffer -> Application Buffer. If your application is not doing anything with the data, why do we need to copy the bytes there? Exactly. We don’t. This is what zero-copy is. Kafka stores its data in the same binary format it responds to requests with. Zero-copy, misleadingly named, is the act of NOT copying data back to the application but rather directly copying it into the NIC buffer. This avoids copies 2) and 3), and avoids the two context switches associated with them! This is a very efficient optimization, especially when you're reading more than what the OS read buffer can take. 🙈Imagine the worst case we had - reading multiples of what the disk read cache can store. This meant that: 💥1. the extra cached copy in the OS read buffer wouldn't get used on further reads (entirely useless copy) 💥2. we had to do steps 1) and 2) in a loop until the wohle data is read by the application. The new flow, thanks in part to scatter-gather operations, is such that: 1. the DMA engine reads from disk into the OS read buffer. 2. the CPU copies very small amounts - file descriptors (references) into the OS socket buffer. 3. scatter-gather operations have the DMA engine use the socket buffer references to read directly from the OS read buffer into the NIC buffer. Misleadingly named - it's not quite *zero* copies, but it's much less than we started with! Four context switches and four copies down to just two of each! It's a great optimization that has been benchmarked to be up to 65% more efficient in certain cases. I am not certain of its exact efficiency gains in Kafka in particular, as it depends a lot on the application that's being benchmarked.