Rex

How does 𝗗𝗮𝘁𝗮 𝗥𝗲𝘁𝗲𝗻𝘁𝗶𝗼𝗻 work in 𝗞𝗮𝗳𝗸𝗮? Kafka is an extremely important Distributed Messaging System to understand as other similar systems are borrowing fundamental concepts from it. 𝗦𝗼𝗺𝗲 𝗿𝗲𝗳𝗿𝗲𝘀𝗵𝗲𝗿𝘀: ➡️ Data is stored in Topics that can be compared to tables in Databases. ➡️ Messages in the Topics are called Records. ➡️ Topics are composed of Partitions. ➡️ Each Record has an offset assigned to it which denotes its place in the Partition. ➡️ Clients writing Data to Kafka are called Producers. ➡️ Each Partition is and behaves as a Write Ahead Log - Producers always write to the end of a Partition. Why do you need to understand Data Retention and how it might impact your Stream Processing Application? ➡️ Unlike a database - Kafka is usually used to pipe enormous amounts of Data hence the Data is not stored indefinitely. ➡️ Since Data is eventually Deleted, given a restart of your Stream Processing Application - it might not find all of the previously seen data. There are generally two strategies to implement Data Retention in Kafka - Deletion and Log Compaction, 𝗹𝗲𝘁’𝘀 𝘁𝗮𝗸𝗲 𝗮 𝗰𝗹𝗼𝘀𝗲𝗿 𝗹𝗼𝗼𝗸: ➡️ Data Retention Strategies are enabled by setting 𝗹𝗼𝗴.𝗰𝗹𝗲𝗮𝗻𝘂𝗽.𝗽𝗼𝗹𝗶𝗰𝘆 to 𝗱𝗲𝗹𝗲𝘁𝗲 or 𝗰𝗼𝗺𝗽𝗮𝗰𝘁. ➡️ This is set on a Broker level but can later be configured per Topic. 𝗗𝗲𝗹𝗲𝘁𝗶𝗼𝗻: ➡️ Each Partition in a Topic is made of Log Segments. ➡️ Log Segments are closed on a certain condition. It can be on a segment reaching a certain size or it being open for a certain time. ➡️ Closed Log Segments are marked for deletion if the difference between current time and when the segment was closed is more than one of: 𝗹𝗼𝗴.𝗿𝗲𝘁𝗲𝗻𝘁𝗶𝗼𝗻.𝗺𝘀, 𝗹𝗼𝗴.𝗿𝗲𝘁𝗲𝗻𝘁𝗶𝗼𝗻.𝗺𝗶𝗻𝘂𝘁𝗲𝘀, 𝗹𝗼𝗴.𝗿𝗲𝘁𝗲𝗻𝘁𝗶𝗼𝗻.𝗵𝗼𝘂𝗿𝘀. If all of them are set, a more granular option will be used. ➡️ After being marked for deletion the segments will be cleaned up by background processes. ❗️If a Topic is configured to not close Log Segments in short enough time - the data could be never cleaned up as the segments would not close. 𝗟𝗼𝗴 𝗖𝗼𝗺𝗽𝗮𝗰𝘁𝗶𝗼𝗻: ➡️ This Strategy only works on Keyed Partitions. ➡️ Topics are compacted by retaining only the latest written record per key. ➡️ Compaction is performed by the background process consisting of a pool of threads called Log Cleaner. Any thoughts, leave them in the comment section 👇 -------- Follow me to upskill in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space. Also hit 🔔to stay notified about new content. 𝗗𝗼𝗻’𝘁 𝗳𝗼𝗿𝗴𝗲𝘁 𝘁𝗼 𝗹𝗶𝗸𝗲 💙, 𝘀𝗵𝗮𝗿𝗲 𝗮𝗻𝗱 𝗰𝗼𝗺𝗺𝗲𝗻𝘁! Join a growing community of Data Professionals by subscribing to my 𝗡𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿.

Some thoughts around Writing Data to Kafka. Kafka is an extremely important 𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝗱 𝗠𝗲𝘀𝘀𝗮𝗴𝗶𝗻𝗴 𝗦𝘆𝘀𝘁𝗲𝗺 to understand as it was the first of its kind and most of the new products are built on the ideas of Kafka. 𝗦𝗼𝗺𝗲 𝗴𝗲𝗻𝗲𝗿𝗮𝗹 𝗱𝗲𝗳𝗶𝗻𝗶𝘁𝗶𝗼𝗻𝘀: ➡️ Clients writing to Kafka are called 𝗣𝗿𝗼𝗱𝘂𝗰𝗲𝗿𝘀,  ➡️ Clients reading the Data are called 𝗖𝗼𝗻𝘀𝘂𝗺𝗲𝗿𝘀. ➡️ Data is written into 𝗧𝗼𝗽𝗶𝗰𝘀 that can be compared to 𝗧𝗮𝗯𝗹𝗲𝘀 𝗶𝗻 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲𝘀. ➡️ Messages sent to Topics are called 𝗥𝗲𝗰𝗼𝗿𝗱𝘀. ➡️ Topics are composed of 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝘀. ➡️ Each Partition behaves like and is a set of 𝗪𝗿𝗶𝘁𝗲 𝗔𝗵𝗲𝗮𝗱 𝗟𝗼𝗴𝘀. 𝗪𝗿𝗶𝘁𝗶𝗻𝗴 𝗗𝗮𝘁𝗮: ➡️ There are two types of records that can be sent to a Topic - 𝗖𝗼𝗻𝘁𝗮𝗶𝗻𝗶𝗻𝗴 𝗮 𝗞𝗲𝘆 𝗮𝗻𝗱 𝗪𝗶𝘁𝗵𝗼𝘂𝘁 𝗮 𝗞𝗲𝘆. ➡️ If there is no key, then records are written into Partitions in a 𝗥𝗼𝘂𝗻𝗱 𝗥𝗼𝗯𝗶𝗻 𝗳𝗮𝘀𝗵𝗶𝗼𝗻. ➡️ If there is a key, then records with the same keys will always be written to the 𝗦𝗮𝗺𝗲 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻. ➡️ Data is always written to the 𝗘𝗻𝗱 𝗼𝗳 𝘁𝗵𝗲 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻. ➡️ When written, a record gets an 𝗢𝗳𝗳𝘀𝗲𝘁 assigned to it which denotes its 𝗢𝗿𝗱𝗲𝗿/𝗣𝗹𝗮𝗰𝗲 𝗶𝗻 𝘁𝗵𝗲 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻. ➡️ Partitions have separate sets of Offsets starting from 1. ➡️ Offsets are incremented sequentially when new records are written. -------- Follow me to upskill in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space. Also hit 🔔to stay notified about new content. 𝗗𝗼𝗻’𝘁 𝗳𝗼𝗿𝗴𝗲𝘁 𝘁𝗼 𝗹𝗶𝗸𝗲 💙, 𝘀𝗵𝗮𝗿𝗲 𝗮𝗻𝗱 𝗰𝗼𝗺𝗺𝗲𝗻𝘁! Join a growing community of Data Professionals by subscribing to my 𝗡𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿.