Last week I spent some time to understand the #PySpark applyInPandasWithState. This week I'm refactoring the code, hoping to still understand it 2 months later ;) 👉 waitingforcode.com/apache-sp…

In the previous release #PySpark has got an interesting streaming feature -> the arbitrary stateful processing. It has a different API than the Scala version but is more adapted to the Python world. More 👉 waitingforcode.com/apache-sp…

It's not a rebranding but more a regrouping 😉 All my additional #dataengineering content is now available from there waitingforcode.com/better (planning to add some stream processing materials soon)

A new module of the Become a Data Engineer V2 is online. You can now enroll in and learn stream processing. Advanced stateful processing concepts are will be released next 👉 becomedataengineer.com/#modu…

Next module of the reworked Become a Data Engineer V2 is online. You can now enroll in and learn about data cleansing. The streaming modules are next to be released, soon! 👉 becomedataengineer.com/#deta…

Arbitrary stateful processing is no more Scala-reserved! It's one of the 3 new features available in #ApacheSpark Structured Streaming 3.4.0 and presented in the new blog post 👉 waitingforcode.com/apache-sp…

Writing this #ApacheSpark 3.4.0 blog post has taken me the most time so far 🥵 But it helped understand the global watermark issue and the fixes reducing its impact added in the recent release 👉 waitingforcode.com/apache-sp…

I'm slowly but regularly writing updates on the #ApacheSpark 3.4.0. To start, there will be 3 blog posts about new features in Structured Streaming. The first of them about async progress tracking is already there 👉 waitingforcode.com/apache-sp… Enjoy!

Amazon Kinesis confidences, part 2 aka how I learned the difference between #ApacheKafka offset and Amazon Kinesis SequenceNumber 👉 waitingforcode.com/data-engi…

Despite some high-level similarities, #ApacheKafka and #AWS Kinesis Data Streams are different! Despite both being streaming brokers, they have different scaling policies, supported delivery semantics, ... and many others available in the new blog post 👉 waitingforcode.com/data-engi…

As promised, I'm continuing with a new series called "Worth reading for data engineers". It summarizes interesting blog posts I've read. This time you'll find a lot of streaming and a bit of project organization 👉 waitingforcode.com/general-d…