Picked up another on my ride home today.

One of my favorites

@Twitter Could you use the local time zone for login warnings? My network packets are indeed coming from LA; Pacific time (PDT) would be a better choice. (This was a legit warning, it was a new device on a new IP, but it was actually me.) Thanks!

Woooooo!!! I am very excited.

Putting this all together, the lesson is clear. Rolling deploys and kafka consumer groups with a shuffling partition assignment algorithm are not friends. Other assignment algorithms exist, but for us the simplest change was our deployment strategy. 7/7 <fin>

We were forced to reconsider our long-held adherence to rolling deploys for everything. This isn’t a web service responding to continuous requests, it’s a stream processor. It can pick up where it left off. We tried it out. 6/7

Though this sort of spiked delay and slow recovery over 20 minutes was annoying, it was actually ok during normal times. But when the system underwent any other stresses, they compounded and the effect could be overwhelming. 5/7

This had a consistent, dramatic, and painful affect on the cluster’s ability to keep up with traffic! Throughout the deploy, the constant shuffling spun our poor servers dizzy until finally they could climb off the peak of delay once the whole ordeal was over. 4/7

A deploy in the standard style rolls out over 10 minutes and there is a significant increase in the number of reports - this shows a constant reshuffling of kafka partitions to consumers as each consumer goes offline and then comes back with the new version. 3/7

Insisting on rolling deploy windows (rather than yearning for the myth that you can have atomic deploys) has always served me well. It encourages healthy development and deployment sensibilities. This is still one of my favorite @honeycombio graphs ever. 1/7