#ApacheKafka tech lead at @Shopify. Views are my own 🇪🇨🇨🇦
Canada
Joined February 2010
- Tweets328
- Following142
- Followers208
- Likes1.4K
Vanessa Vuibert retweeted
Our Apache Kafka fleet smashed records this BFCM, handling over 66 million messages per second! That's 120% more streaming than 2023 🛍️📈
A few more fun #apacheKafka peak stats during Black Friday for @ShopifyEng:
- 120% growth from 2023 with 66.3 million messages per second
- 4.89 PB of data
- 6.3 ms latency
Yesterday was an incredible day for our merchants and for Shopify:
* Merchants drove a record-breaking $5 billion Black Friday in sales (GMV)
* Peak sales / minute for yesterday = $4.6 million
* Shopify's egress processed more than 173B requests
* Peak requests per minute was at 284 million on edge, with more than 80 million on app servers pushing 12TB a minute
* 45 million peak database queries per second, peak database writes was 7.6 million per second
* We rely on Kafka. At peak we hit 66 million messages per second
Let's keep going 🔥
#apacheKafka at Shopify #BlackFriday peak growth through the years:
- 2021 12.9M msg/sec
- 2022 52% growth with 19.6M msg/sec
- 2023 54% growth with 30.2M msg/sec
- 2024 120% growth with 66.3M msg/sec
Looking forward to co-hosting a table talk tomorrow at #StaffPlusNewYork on:
How do you effectively mentor junior engineers as a staff+ leader?
See you tomorrow NYC! 🍎 For more information about #StaffPlusNewYork, including some helpful FAQs, click the link below ⬇️ leaddev.com/staffplus-new-yo…
2
434
A few more fun #apacheKafka average stats during Black Friday Cyber Monday at Shopify:
- 1.5 trillion messages per day
- 2.9 PB of data
- 4.8 ms latency
Our #ApacheKafka streaming infrastructure served 29 million messages per second at peak. That’s 45% growth from 2022 – and we did it more efficiently this year with 14% fewer brokers.
The recording for my #current23 talk about #apacheKafka traffic management is now available!
events.bizzabo.com/468544/ag…
7
504
I enjoyed discussing the different solutions to improve #apachekafka resiliency at #current23 the last couple of days!
The recording to my talk should be available next week. In the meantime, here's a link to my slides: bit.ly/46uvJza
Great presentation from @V3_XD about increasing @apachekafka resilience with DNS traffic management
#Current23
6
422
I am once again a speaker this year at #Current23! I'll be talking about how to improve the resiliency of #ApacheKafka clusters with DNS traffic management and client wrappers. Thanks for having me back and I look forward to seeing you next week!
175
Vanessa Vuibert retweeted
📈Our #ApacheKafka streaming infrastructure served 20 Million messages per second at peak, that’s a 52% growth from 2021!
The recording for my recent talk at #current22 on #Kubernetes StatefulSet migrations is now available: confluent.io/events/current-…
Enjoyed talking about how @ShopifyEngineering's #ApacheKafka team upgrades #Kubernetes at #Current22!
Slides: slideshare.net/VanessaVuiber…
Vanessa Vuibert retweeted
👋🏼 The streaming platform team @ShopifyEng had an absolute blast participating in #Current22 the past couple of days! With the event wrapped up, here are some of our talks that you should definitely check out when the recordings are available 👇🏼🧵:
Excited to talk about how @ShopifyEng does #Kubernetes upgrades for #ApacheKafka clusters. Looking forward to seeing you at #Current22!
Vanessa Vuibert retweeted
Neisi Dajomes of #ECU wins #Gold in the women’s weightlifting 76 kg!
It’s the first time Ecuador have won two medals at the same Olympic Games and Dajomes becomes the first Ecuadorian woman to win an Olympic gold medal!
@ IWFNet #StrongerTogether | #Tokyo2020 | #Weightlifting
Vanessa Vuibert retweeted
I wonder what Space Aliens would say, if told that Earthlings shift clocks by an hour in the summertime to fool themselves into thinking there's more sunlight in the day.
Estamos emocionados de anunciar el tema que tenemos para este mes. Conversaremos sobre el manejo del tráfico en #Kafka. Para esta charla hemos invitado a @V3_XD, Senior product engineer para Shopify (Canadá). Regístrate para estar al tanto de las novedades meetup.com/Quito-Lambda-Meet…
2
Vanessa Vuibert retweeted
It's Double-Post Wednesday on the blog! ✍️✍️
@raphaelbeamonte takes us on a journey through @Shopify’s DNS traffic management evolution:
engineering.shopify.com/blog…
Need a DNS traffic management primer? Check out this intro first (with sweet gifs 😎):
engineering.shopify.com/blog…