Data stuff Free Data Engineering Fundamentals Course: https://xitter.cf/t.co/Sp4qDLRwEB

New york
Joined April 2020
Joseph Machado retweeted
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out. I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really). It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely. The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture. We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying. I worry.
501
1,891
394
15,239
1,624,498
Joseph Machado retweeted
Left anti-join is cool! Get all the data from the left table that has no matching data in the right table select t1.* from t1 left join t2 on t1.id=https://xitter.cf/t.co/TilIGFtVGj where t2.id is null; #data #dataengineering #SQL #Database
6
19
3
141
If you are interested in using "Change Data Capture" pattern for streaming ETL, check out startdataengineering.com/pos… #ETL #changedatacapture #dataengineering #debezium #BigData
4
61
2
286
Joseph Machado retweeted
Preparing for SQL interviews? Do Leetcode SQL hard, sort by freq, and do the first 40 #data #dataengineering #Software #SQL
11
70
2
374
Starting a data project is a lot of work! It can be overwhelming to define the problem, set up systems, and then code! Use this DE project as a blueprint to build your own: startdataengineering.com/pos… #data #dataengineering #Database #DataAnalytics #dataviz #Python #datapipeline
5
75
2
363
Backfilling is an inevitable part of data projects. When designing your data pipelines take some time to answer the following questions 1. Does multiple backfill runs cause duplicate data? 2. Can multiple backfills be parallelized? #data #DataEngineering #datapipeline #datasets
4
19
3
205
When data to process is larger than memory, try to stream with python generators, before jumping to distributed systems! #data #dataengineering #Python #pythonlearning #Generator E.g. Stream a file(note () and not []), get diff between date cols
4
71
376
Learning data engineering? Build a pipeline locally. 1. Python to pull data from an API (e.g. Coincap) 2. Load data into a local Postgres container 3. Automate it with cron/task scheduler Start small, build, improve, & repeat. #data #dataengineering #pythonlearning #Python
8
102
2
521
Joseph Machado retweeted
Starting as a DE? 90% of what you will need is SQL (OLAP), python, & distributed system basics Don't overcomplicate! #data #dataengineering #SQL #Database #Python
21
64
1
451
Exercise project for anyone starting in data engineering startdataengineering.com/pos… #dataengineering #bigdata #ETL #ApacheAirflow #AWS #ApacheSpark
14
89
1
475
It can be overwhelming to start learning data engineering. I'd recommend starting with the basics of python, sql, UNIX commands, building a simple data project, update Github, Linkedin. Landing a DE job is 60% part learning and 40% marketing. See reply 👇🏽 for helpful links.
15
73
2
354
Data engineers write the most complex piece of code to Upsert into tables. Here's THE command you need to know MERGE INTO/INSERT ON CONFLICT #data #dataengineering #SQL
2
3
49
2,747
Over the past 10 years I've seen multiple data engineers do this mistake when it comes to pulling data from an API! I'll show you how to avoid it in 2 minutes: #data #dataengineering #datapull
3
4
15
1,893
DeepSeek R1 is way better! No benchmarks, just based on personal exp.
2
1
15
1,241
If you’ve worked with business users, you know they love creating pivot charts in Excel—ever wonder how to do it in SQL? Transforming rows into columns for easy analysis doesn’t have to be limited to spreadsheets. Let me show you how: startdataengineering.com/pos… #data #SQL
1
1
9
1,050
Generate a range of number/date rows in SQL with GENERATE_SERIES Use cases: 1. Reporting on all dates: If your data does not have every date, but you want to report on them, use this. 2. Looping over series of values like numbers or dates #data #SQLtips #dataengineering
9
3
18
1,487
I've spent over 4,000 hours in stakeholder requirement-gathering meetings! Save hours of your life by asking these questions: 👇 #data #dataengineering #datastakeholder
3
2
24
2,107
Are you a solo Data Engineer feeling overwhelmed, not knowing where to start? I'll teach you how to excel in just 10 minutes: 👇 #data #dataengineering #solodev
3
11
40
3,208
I've been interviewing data candidates in system design for 10 years! Here are the key concepts you need to master – and I’ll break it down in just 5 minutes: 👇 #data #dataengineering
2
5
29
2,682