AI makes some reasonable guesses, but always verify the results. I’m working on an article to streamline pipeline creation. We still need humans in the loop. The image below recommends an ID-based delete-insert for a fact table, which will take forever to run if your data is large. Add these to your skills.md file. #dataengineering #llm

Working on a project that teaches you how to build data pipelines following standard design practices and speeding up development with agent harness & skills! #dataengineering #dataproject

I’ve spent way too much time thinking about my brand colors. Ultimately, I decided to go with what would make my readers' lives easier. A bright screen makes it hard to concentrate for long intervals. So I switched my website to use the Gruvbox theme. Does it help? Check it out at startdataengineering.com/

Updating my “Python Essentials for Data Engineers” post and came across this amazing data quality tool. pointblank from posit. Easy to set up and use, as any tool should be. No need to wrestle with 100s of configs/settings. #dataengineering #python #dataquality

Which one should I use; Gruvbox or standard? Gruvbox is easier on the eyes But the standard theme (Bootswatch, Flatly) is familiar Let me know in the poll below 👇 #ux #readability

When choosing a tool/framework, always pick the one with the simplest implementation. But that could be extended with libraries, Plugins, Ecosystem, etc. Here are some of my favorite tools 1. DuckDB: SQL, enhanced with extensions 2. Python: English-like syntax, enhanced with libraries 3. Quarto: Markdown-like syntax, enhanced with extensions 4. nvim: vim syntax, enhanced with extensions The fancier/esoteric the tool, the harder it is to get productive quickly. What are some of your favorite tools? #tools #dataengineering

90% of all facts and dimensions follow a similar design pattern, with some differences in business logic. This messy middle (fact & dims) is where DEs can lay the foundations for reliable data. If you mess this up, your team will always face issues/questions/mismatches, etc. Here is a V1 of this design

Whenever I design a workshop, I use @robfitz ’s “The Workshop Survival Guide”. I see how great user design increases engagement, recommendations, and leads to higher ratings. Here is a very rough version of a workshop that I am working on (coming soon). Focus on 1. Takeaways for the audience. Not just watch and forget. 2. Increase play. Interweave code exercises and visuals to help understanding. #dataengineering #userexperience

AI creates code fast, but is it really saving you time? Spend a few hours learning the system to save days in bug fixing. But, in the age of performative work (quick implementation + heroic bug fixes), this is rewarded. So engineers are incentivized for this pattern. I'm curious to hear how other engineers approach this. LMK in the comments below. #dataengineering #agenticengineering

LLMs made code cheap, but they cost you your understanding. Don’t yeet code into production or learn new stuff without a deep understanding of what LLMs generate. Or, better yet, practice without AI; once you get a feel for the system, implement with AI. You still got to think! arxiv.org/html/2601.20245v1 #llm

My first pipelines used Python, MySQL, and Task Scheduler. Then came a Cambrian explosion of tools like Spark, S3, etc. Causing a mad dash towards “upskilling” in these tools just to keep up. This was a tough time for me, filled with anxiety and imposter syndrome (still exists). New & mid-level DEs are facing a much more FUD-driven environment today, with fewer opportunities, LLMs, and so many more tools/platforms to learn. But not all is lost. True mastery is when you can distill complex ideas into fundamental principles. Distilling data engineering using this, it becomes. “Data engineers get easy-to-use & reliable data to end-users” Take a look at the image to see what Easy-to-use, reliable, and end-users are.