It's the final day of mirrord Failure Week! We've spent the week looking at failures that are easy to miss because they're partial, slow, or only show up under sustained load. So for the last one, we're going with the most obvious failure of all: 💀 Your dependency is completely down Most teams probably think they've already handled this one. The dependency is unreachable, the call fails, and your error handling kicks in. But the interesting question isn't whether the call fails, it's whether that failure stays contained. Say your checkout page also pulls recommendations from a separate service. Recommendations are nice to have, but they have nothing to do with a user actually completing a purchase. If the recommendations service goes down, checkout should keep working. But an unhandled exception, or simply awaiting that request alongside critical checkout logic without isolating its failure, can mean an optional feature takes down the thing the customer actually came for. This is exactly the kind of assumption that's worth testing against the real system. With mirrord Chaos Testing, you can make your service's connection to the real recommendations service fail completely, while the dependency itself keeps working normally for everyone else in the cluster. ✅ What to check: Does your application still function when a non-critical dependency is unreachable, or can one optional feature take down the whole request? Check the thread for the mirrord Chaos Testing rule you can use to reproduce this 🧵

It's the penultimate day of mirrord Failure Week, where we're highlighting 5 failures your service should survive, but that often go untested because reproducing them realistically is hard. Here's daily failure #4: 📈 Your dependency slows down under sustained load So far all the failures we mentioned are visible pretty quickly once you send a single request. This one is different because it requires continuous load. Say your shopping cart publishes an event to RabbitMQ after every checkout, and a consumer picks those events up and writes them to Postgres. Normally, those database writes take 100ms. But what happens if they suddenly start taking a full second? The checkout service itself can keep working perfectly because it isn't waiting for those writes. But the consumer can no longer keep up with incoming messages, so the queue starts growing... and growing... while everything the customer sees still looks healthy. That's what makes this failure easy to miss. Testing one request tells you almost nothing. You need sustained traffic while the dependency is degraded to see whether your system can still keep up. With mirrord Chaos Testing, you can add latency to the consumer's real Postgres writes while sending a steady stream of messages through the system. Instead of just checking whether individual requests succeed, you can watch whether the slowdown creates a backlog over time. ✅ What to check: Watch your queue depth, not the response from checkout. If it keeps climbing, you've found a throughput problem that normal request-level testing probably wouldn't reveal. Then you need to decide what should happen before that backlog becomes dangerous: scale or speed up the consumer, bound the queue, or alert before it gets there. Check the thread for the mirrord Chaos Testing rule you can use to reproduce this 🧵

It's day 3 of mirrord Failure Week, where we're highlighting 5 failures your service should survive, but that often go untested because reproducing them realistically is hard. Here's daily failure #3: 👻 Your request failed, but the work still happened This is a particularly nasty one because, as far as your client is concerned, everything is pretty clear: the request timed out, so it failed. Except the server might have already done the work. Say your shopping cart app calls an inventory service to reserve an item. The inventory service successfully reserves it, but its response takes longer than your client's timeout. Your client gives up, reports a failure, and retries. Meanwhile, the inventory service finishes the original request. Now you've potentially reserved the same item twice. This is why simply testing that your client handles a timeout isn't enough. You need to know what happened on the other side of that timeout. And this is where chaos testing gets interesting. Any fault that outlasts your client's timeout produces this. Latency is simply the easiest one to arrange, so set it above whatever timeout your client uses. That lets you reproduce the exact ambiguity you're trying to protect against: the client thinks it failed, while the server succeeded. ✅ What to check: Don't just look at the error your client reports. Check what actually happened in the dependency: reservations created, rows written, charges made, etc. If one failed request and a retry result in the work happening twice, you probably need idempotency, not a longer timeout. Check the thread for the mirrord Chaos Testing rule you can use to reproduce this 🧵

It's day 2 of mirrord Failure Week, where we're highlighting 5 failures your service should survive, but that often go untested because reproducing them realistically is hard. Here's daily failure #2: 🎲 Your dependency is flaky, not fully down A dependency that's completely unavailable is relatively straightforward to handle. Every request fails, your error handling kicks in, and you know exactly what state you're dealing with. But a dependency that only fails some of the time is much more interesting. Say it's the search service behind your app's product search bar. Maybe 25% of connections are failing while the other 75% are perfectly healthy. Your application should ideally keep using the healthy connections and handle the failures gracefully. But retries, connection pools, and circuit breakers can actually turn that partial failure into a much bigger problem. For example, retries might hammer an already struggling dependency, your connection pool might keep reusing unhealthy connections, or your circuit breaker might start rejecting requests that would otherwise have succeeded. This is a scenario that's difficult to test realistically with mocks. You don't just want to know whether your code handles a connection error. You want to see what happens when some real connections fail while others continue working normally, all while your service remains connected to other real dependencies in the cluster. With mirrord Chaos Testing, you can reset only a percentage of your service's connections to the real dependency, without changing the dependency itself or affecting anyone else using the cluster. ✅ What to check: Does the partial failure stay partial, or does your application amplify it? Watch for retry storms, unhealthy connections being reused, or a circuit breaker that starts rejecting otherwise healthy traffic. Check the thread for the mirrord Chaos Testing rule you can use to reproduce this 🧵