Scalability! But at what cost? This paper is an absolute classic because it explores the underappreciated tradeoffs of distributing systems. It asks about the COST of distributed systems--the Configuration that Outscales a Single Thread. The question is, how many cores does a big distributed system need to outperform some moderately-optimized single-threaded code running on your laptop? As it turns out, scalability often comes with an extremely high COST. The authors examine several graph processing systems--including some big names like Spark--and find that they need dozens to hundreds of cores to outperform a single-threaded program. Why is this the case? It's not because these distributed systems are badly designed, but because distributing computation is inherently inefficient for many problems. Fundamentally, a distributed system cannot rely on all processors sharing state, at least not efficiently. This is a big issue! In graph algorithms, it means servers need to expensively exchange data and eliminates a wide swathe of algorithms and optimizations that rely on shared state. In distributed databases, it means expensive coordination is required to distribute transactions to ensure participating servers have consistent views of data. Does this mean we shouldn't build scalable systems? Of course not! Many problems are well beyond the capability of a single server, no matter how optimized. But it does mean we should be mindful of the efficiency costs of scaling. As an aside, I think this kind of thinking is why Postgres is so popular, despite not being distributed. A large Postgres server can handle a vast amount of traffic (especially with read replicas, which can be cheaply maintained). You need a huge company or incredibly heavy workload to outscale that single server, and when you do, the alternatives come with huge tradeoffs!

What happens if your CPU gets something wrong? If it wakes up one day and decides 2+2=5? Well, most of us will never have to worry about that. But if you work at a company the size of Google, you do, which is why this paper on "mercurial cores" is so fascinating. What the authors report--and supposedly this is common knowledge at the hyperscalers--is that a couple cores per several thousand machines are "mercurial." Due to subtle manufacturing defects or old age, they give wrong answers for certain instructions. These can cause all sorts of impossible-to-diagnose issues. Some rare problems at Google that were traced back to bad CPUs include: - Mutexes not working, causing application crashes - Silent data corruption - Garbage collectors targeting live memory, causing application crashes - Kernel state corruption causing kernel panics What makes CPUs go bad? It's very hard to tell. The authors posit that issues are becoming more frequent as CPUs get more complex, but there aren't solid numbers behind that. There are certainly strong relationships between frequency, temperature, voltage, and bad CPU behavior--most mercurial CPUs only cause problems under very specific conditions, but those conditions vary from CPU to CPU. Age is another source of problems, as older CPUs are more likely to exhibit problems. Bad CPUs are an especially serious problem because they're very hard to detect. If cosmic rays flip bits in storage or on the network, that can be detected through error coding. But there's no analogy for a CPU that allows cheap online verification of its correctness. Instead, the best detection techniques involve monitoring for symptoms. If a core exhibits exceptionally high rates of process crashes or kernel panics relative to its fellows, that's a strong indication something is wrong with it. For the most critical applications, the authors propose triple modular redundancy--redoing each of its computations on three cores and majority-voting a reliable result. More than anything, this paper is a call to action--letting everyone know that CPUs can fail. So now, if you ever find a bug you can't diagnose, you can blame the CPU! 🙂