Eventual consistency bugs start before "eventually" When you read "Eventually consistent", it usually sounds like it will happen with a small delay. And this means that when writes stop, replicas will catch up with the data. But the question is what happens before this? Here is what happens: 𝟭. 𝗥𝗲𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝗹𝗮𝗴 The primary confirms a write before every replica has it. A read from the slow replica returns the old value. 𝟮. 𝗥𝗲𝗮𝗱 𝘆𝗼𝘂𝗿 𝘄𝗿𝗶𝘁𝗲𝘀 After a write, the read carries that version. A replica that is behind waits until it catches up. 𝟯. 𝗖𝗼𝗻𝗳𝗹𝗶𝗰𝘁 𝗿𝗲𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻 When more than one node takes writes, two of them can accept different writes before they sync. Last write wins keeps one and 𝗱𝗿𝗼𝗽𝘀 𝘁𝗵𝗲 𝗼𝘁𝗵𝗲𝗿 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗮𝗻 𝗲𝗿𝗿𝗼𝗿. A merge keeps both. 𝟰. 𝗤𝘂𝗼𝗿𝘂𝗺 With 3 replicas, we wait for 2 on every write and ask 2 on every read. Because 2 + 2 > 3, every read reaches at least one replica that has the confirmed write. 𝟱. 𝗛𝗶𝗻𝘁𝗲𝗱 𝗵𝗮𝗻𝗱𝗼𝗳𝗳 When a replica is down, another node keeps its write and delivers it when the replica is back. 𝟲. 𝗥𝗲𝗮𝗱 𝗿𝗲𝗽𝗮𝗶𝗿 A read notices that one replica is stale and writes the newest value back to it. Before we add a read replica, we should go feature by feature and decide if that read is allowed to be old. A product list can be a little behind. The profile a user just saved cannot. Write that decision down for every screen before the first replica goes live.