What are the most common 𝗨𝘀𝗲 𝗖𝗮𝘀𝗲𝘀 𝗳𝗼𝗿 𝗞𝗮𝗳𝗸𝗮? We have covered lots of concepts around Kafka already. But what are the most common use cases for The System that you are very likely to run into as a Data Engineer? 𝗟𝗲𝘁’𝘀 𝘁𝗮𝗸𝗲 𝗮 𝗰𝗹𝗼𝘀𝗲𝗿 𝗹𝗼𝗼𝗸: 𝗪𝗲𝗯𝘀𝗶𝘁𝗲 𝗔𝗰𝘁𝗶𝘃𝗶𝘁𝘆 𝗧𝗿𝗮𝗰𝗸𝗶𝗻𝗴. ➡️ The Original use case for Kafka by LinkedIn. ➡️ Events happening in the website like page views, conversions etc. are sent via a Gateway and piped to Kafka Topics. ➡️ These events are forwarded to the downstream Analytical systems or processed in Real Time. ➡️ Kafka is used as an initial buffer as the Data amounts are usually big and Kafka guarantees no message loss due to its replication mechanisms. 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲 𝗥𝗲𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻. ➡️ Database Commit log is piped to a Kafka topic. ➡️ The committed messages are executed against a new Database in the same order. ➡️ Database replica is created. 𝗟𝗼𝗴/𝗠𝗲𝘁𝗿𝗶𝗰𝘀 𝗔𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗶𝗼𝗻. ➡️ Kafka is used for centralized Log and Metrics collection. ➡️ Daemons like FluentD are deployed in servers or containers together with the Applications to be monitored. ➡️ Applications send their Logs/Metrics to the Daemons. ➡️ The Daemons pipe Logs/Metrics to a Kafka Topic. ➡️ Logs/Metrics are delivered downstream to storages like ElasticSearch or InfluxDB for Log/Metrics discovery respectively. ➡️ This is also how you would track your IoT Fleets. 𝗦𝘁𝗿𝗲𝗮𝗺 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴. ➡️ This is usually coupled with ingestion mechanisms already covered. ➡️ Instead of piping Data to a certain storage downstream we mount a Stream Processing Framework on top of Kafka Topics. ➡️ The Data is filtered, enriched and then piped to the downstream systems to be further used according to the use case. ➡️ This is also where one would be running Machine Learning Models embedded into a Stream Processing Application. 𝗠𝗲𝘀𝘀𝗮𝗴𝗶𝗻𝗴. ➡️ Kafka can be used as a replacement for more traditional messaging brokers like RabbitMQ. ➡️ Kafka has better durability guarantees and is easier to configure for several separate Consumer Groups to consume from the same Topic. ❗️Having said this - always consider the complexity you are bringing with introduction of a Distributed System. Sometimes it is better to just use traditional frameworks. -------- Follow me to upskill in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space. Also hit 🔔to stay notified about new content. 𝗗𝗼𝗻’𝘁 𝗳𝗼𝗿𝗴𝗲𝘁 𝘁𝗼 𝗹𝗶𝗸𝗲 💙, 𝘀𝗵𝗮𝗿𝗲 𝗮𝗻𝗱 𝗰𝗼𝗺𝗺𝗲𝗻𝘁! Join a growing community of Data Professionals by subscribing to my 𝗡𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿: newsletter.swirlai.com

Switching to a 𝗠𝗟𝗢𝗽𝘀 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿 role might be easier than you think. Usually MLOps Engineers are professionals tasked with building out the ML Platform in the organization.  This means that the skill set required is very broad - naturally very few people start off with the full set of skills you would need to brand yourself as a MLOps Engineer. This is why I would not choose this role if you are just entering the market. However, you can get into the role from various sides once you already have some experience. Here are some examples - If you currently are: 𝗠𝗟 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿: this is the easiest one since if you are currently successful in your role you might already have skills that are needed to become a successful MLOps Engineer. What you will need to do though is switch from an execution to a service role. So the main shift is mental rather than technical. 𝗜𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲/𝗖𝗹𝗼𝘂𝗱 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿: you are most likely good with infrastructure architecture, IaaC, Cloud Services etc. These are all crucial skills to have in the ML platform team. 𝗗𝗲𝘃𝗢𝗽𝘀 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿: you have probably mastered CI/CD infrastructure and very well know how to template and automate things, how to increase developer velocity - each of these being a necessity to become MLOps engineer. 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿: There are a number of areas that software engineering skills could be leveraged in ML platform team, e.g. coding up clean interfaces, backend services, UIs to be used by platform adopters. On top of that, you are probably as good with CI/CD infrastructure as most DevOps engineers are. I personally have seen that most successful MLOps Engineers have Machine Learning Engineering background as they best understand the needs of users of the ML Platform they would be building.  In any case - be prepared to learn a lot as MLOps Engineers are generalists and it usually takes a lot of time to acquire the full-stack. What Data Role are you currently occupying? What are your future plans? -------- Follow me to upskill in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space. Also hit 🔔to stay notified about new content. 𝗗𝗼𝗻’𝘁 𝗳𝗼𝗿𝗴𝗲𝘁 𝘁𝗼 𝗹𝗶𝗸𝗲 💙, 𝘀𝗵𝗮𝗿𝗲 𝗮𝗻𝗱 𝗰𝗼𝗺𝗺𝗲𝗻𝘁! Join a growing community of Data Professionals by subscribing to my 𝗡𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿: newsletter.swirlai.com