Tableflow is a new Confluent feature which turns your Kafka Topic into an Iceberg or Delta Lake table, all at the click of a button. 🔥 Here’s how it works 👇 It consists of three components: • Materializer - reads Kafka log segments, converts them to Parquet, saves the Parquet in S3 and sends off a CommitTask, to commit the file to Iceberg. 👋 • Compactor (not pictured) - reads small Parquet files, compacts them into larger ones, saves the larger files in S3 and sends off a CommitTask, to commit the changes in Iceberg. 🎁 • Committer - processes CommitTasks by writing metadata to the Iceberg table and committing it. ✅ The materializer is optimized to read directly from object storage, bypassing Kafka entirely. This avoids placing extra load on the Kafka cluster - like connections and request processing - as a regular consumer would. 👌 The Kafka broker would usually itself fetch the data from S3 and return it to the Materializer. Instead, the materializes fetches the coordinates for the segments in S3 and reads them from S3 directly. 👍 Tableflow also performs table maintenance - it compacts small Parquet files into larger ones. Small files are bad because they wreck query performance. 🛑 Column-optimized engines are optimized to read large columns/files at a time, and engines like Flink parallelize their tasks per Parquet file, which can result in a lot of task overhead if there are too many files. The compaction is done via a single-pass TumblingWindow Binpack Compaction algorithm inside a Compactor component not pictured here. When done, it sends a CompactionTask to the Committer to commit the changes to the table when it can. 🫡 If you know anything about Iceberg - you’ll know it does NOT perform well when there are a lot of concurrent writes happening to it. This is because the series of operations for any single writer are: 1. Scan the table 2. Write the data 3. Refresh its metadata 4. Check against conflicts 5. Write the metadata 6. Commit the snapshot via an atomic compare-and-set operation. If any other write happens between steps 3 and 6, the steps have to be repeated all over. This can spiral into a lot of wasted cycles for each concurrent writer. To avoid this, the Committer is a singleton service. It also accumulates commit tasks so as to commit more of them in a larger batch, reducing overhead and concurrent writes. It does two commits in a row - the first one consists of the compaction tasks and the second of the newly-materialized data. This ordering helps avoid potential metadata conflicts. 👌