Joseph Machado retweeted
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
Joseph Machado retweeted
An orchestration tool that I've been impressed with is @dagsterio. Easy setup, powerful features and great docs.
Use 👇🏽 to play around with a pipeline on dagster
startdataengineering.com/pos…
#data #dataengineering #Python #Database #DataAnalytics
Joseph Machado retweeted
If you are interested in using "Change Data Capture" pattern for streaming ETL, check out startdataengineering.com/pos…
#ETL #changedatacapture #dataengineering #debezium #BigData
Joseph Machado retweeted
Preparing for SQL interviews?
Do Leetcode SQL hard, sort by freq, and do the first 40
#data
#dataengineering
#Software
#SQL
Joseph Machado retweeted
Starting a data project is a lot of work! It can be overwhelming to define the problem, set up systems, and then code!
Use this DE project as a blueprint to build your own:
startdataengineering.com/pos…
#data #dataengineering #Database #DataAnalytics
#dataviz #Python #datapipeline
Joseph Machado retweeted
Backfilling is an inevitable part of data projects. When designing your data pipelines take some time to answer the following questions
1. Does multiple backfill runs cause duplicate data?
2. Can multiple backfills be parallelized?
#data #DataEngineering #datapipeline #datasets
Joseph Machado retweeted
When data to process is larger than memory, try to stream with python generators, before jumping to distributed systems!
#data #dataengineering #Python #pythonlearning #Generator
E.g. Stream a file(note () and not []), get diff between date cols
Joseph Machado retweeted
Learning data engineering? Build a pipeline locally.
1. Python to pull data from an API (e.g. Coincap)
2. Load data into a local Postgres container
3. Automate it with cron/task scheduler
Start small, build, improve, & repeat.
#data #dataengineering #pythonlearning #Python
Joseph Machado retweeted
Starting as a DE? 90% of what you will need
is SQL (OLAP), python, & distributed system basics
Don't overcomplicate!
#data
#dataengineering
#SQL
#Database
#Python
Joseph Machado retweeted
Exercise project for anyone starting in data engineering startdataengineering.com/pos…
#dataengineering #bigdata #ETL #ApacheAirflow #AWS #ApacheSpark
Joseph Machado retweeted
It can be overwhelming to start learning data engineering. I'd recommend starting with the basics of python, sql, UNIX commands, building a simple data project, update Github, Linkedin. Landing a DE job is 60% part learning and 40% marketing. See reply 👇🏽 for helpful links.
Joseph Machado retweeted
Data engineers write the most complex piece of code to Upsert into tables.
Here's THE command you need to know
MERGE INTO/INSERT ON CONFLICT
#data
#dataengineering
#SQL
Joseph Machado retweeted
Over the past 10 years I've seen multiple data engineers do this mistake when it comes to pulling data from an API!
I'll show you how to avoid it in 2 minutes:
#data #dataengineering #datapull
Joseph Machado retweeted
If you’ve worked with business users, you know they love creating pivot charts in Excel—ever wonder how to do it in SQL?
Transforming rows into columns for easy analysis doesn’t have to be limited to spreadsheets.
Let me show you how:
startdataengineering.com/pos…
#data #SQL
Joseph Machado retweeted
Generate a range of number/date rows in SQL with GENERATE_SERIES
Use cases:
1. Reporting on all dates: If your data does not have every date, but you want to report on them, use this.
2. Looping over series of values like numbers or dates
#data
#SQLtips
#dataengineering
Joseph Machado retweeted
I've spent over 4,000 hours in stakeholder requirement-gathering meetings!
Save hours of your life by asking these questions: 👇
#data
#dataengineering
#datastakeholder
Joseph Machado retweeted
Are you a solo Data Engineer feeling overwhelmed, not knowing where to start?
I'll teach you how to excel in just 10 minutes: 👇
#data #dataengineering #solodev
Joseph Machado retweeted
I've been interviewing data candidates in system design for 10 years!
Here are the key concepts you need to master – and I’ll break it down in just 5 minutes: 👇
#data #dataengineering








