I help companies optimize data streaming systems.
Vancouver, Canada
Joined December 2009
- Tweets5.5K
- Following273
- Followers1.4K
- Likes7K
One of my biggest learnings of the past 12 months as a consultant was ditching hourly pricing.
I no longer have an hourly rate.
You shouldn't either: independentengineer.co/p/hou…
Cool project that's worth checking out if you're running Flink at scale: github.com/datafusion-contri…
So far, when data streaming and AI are mentioned together, it typically describes the problem of maintaining a fresh, relevant context: e.g. use stream-processing to join/aggregate/enrich data streams, and pass them to an agent as context for inference. And honestly, I'm sold.
But is there anything else? E.g. using Kafka for agent-to-agent communication? Streaming long agent sessions? Workflow orchestration?
I really like what S2 is doing: s2.dev/blog/agent-sessions , looking for more examples like this!
I'm considering adding a few updates to my Advanced Apache Flink course. What would you like to see?
streamacademy.io/course/adva…
If you enjoy working with large-scale data processing systems and you want to stay relevant, you have to learn Apache DataFusion. Slowly but surely, it's replacing many other tools. The RAD stack (Rust, Arrow, DataFusion) is the future.
One example: datadoghq.com/blog/engineeri…
I attended the Data Streaming Summit last week. Just published a post with a few highlights from the event: streamingdata.tech/p/data-st…
Watermarks can be a really scary concept when you start working with stateful streaming systems. And all popular frameworks don't offer anything better than "bounded out-of-orderness": pick a number of seconds and hope that all data arrives within that interval.
But when you do control the producers (e.g. application events), the producers themselves can be the source of watermarks! The idea is not new, and it's covered in several famous papers like Google's MillWheel (see the screenshot).
Earlier today at the Data Streaming Summit. Thanks Tun Shwe for the picture!
#DSS26
Does anyone use IVM systems like Materialize or Feldera in production? What's your experience?
Feel free to DM.
A very cool FLIP (Flink Improvement Proposal) has recently been accepted: Running Flink jobs in MiniCluster using the Kubernetes Operator.
cwiki.apache.org/confluence/…
The FLIP addresses the problem by running both components (JobManager and TaskManager) in the same JVM process on a single pod. It's a big step in making Flink simpler and more efficient to operate.
It's fascinating how many projects now offer a native CDC (Change Data Capture) experience. A few years ago, your choice was to use Debezium or ... *checks notes* that's it.
Postgres logical replication for CDC has become almost... boring?