Never trust the source of your data. I spent 2 months building a data pipeline to train a model. A single word broke it in 24 hours. A pipeline that doesn't validate the schema of the data will invariably fail and take your entire system down with it. I was building a churn model for a subscription application. A single change in a database table turned "session_minutes" into a string feature. My code started sorting the field lexicographically, so "100 became less than "20" overnight. I took the whole thing down, and it was 100% preventable. I allowed data to come in without first verifying its schema. I was reacting to any potential failures instead of preventing them. Always enforce a contract on all incoming data. This is the difference between a system that's brittle and reactive and a system that's resilient and predictable. Many people I know use Kafka for data streaming. Unfortunately, Kafka is overkill, notoriously complex, and expensive for 80% of the problems it solves. Bufstream is a great drop-in alternative (and the only one validated by Jepsen). 8x cheaper than Kafka, and it's a simple binary you can run on your local machine, AWS, or GCP, capable of scaling from 0 to 100 GiB/s of writes with no operator intervention. The best part about Bufstream is its built-in schema validation and focus on data quality and privacy. Here is Bufstream's GitHub repository: fnf.dev/3TZ4jgo And here is an article covering how Bufstream makes Schema-driven development a must-have for anyone building data streaming pipelines: fnf.dev/3Gx9wc9 Thanks to the @bufbuild team for collaborating with me on this post.