luminousmen (@luminousmen) header
luminousmen (@luminousmen) profile picture

luminousmen

@luminousmen

Senior Data Engineer @Spotify | Blogging at https://xitter.cf/t.co/ZrcSE7WJOu about Data Engineering, Python, ML | Author of Grokking Concurrency

luminousmen.com
Joined November 2017
  • Tweets761
  • Following92
  • Followers537
140Media
Photo by luminousmen (@luminousmen)Photo by luminousmen (@luminousmen)Photo by luminousmen (@luminousmen)Photo by luminousmen (@luminousmen)Photo by luminousmen (@luminousmen)Photo by luminousmen (@luminousmen)Photo by luminousmen (@luminousmen)Photo by luminousmen (@luminousmen)Photo by luminousmen (@luminousmen)Photo by luminousmen (@luminousmen)
Hello wonderful people! I’m thrilled to announce that my new book, “Grokking Concurrency,” has officially hit the shelves! You can find it here: manning.com/books/grokking-c… I could really use your support in spreading the word!
Image from luminousmen (@luminousmen)
6
28
107
15,033
Nine women can't make a baby in a month. A friend uses that line to explain agents.
1
1
38
She gives a one-day task to an agent, moves on to something else, and when the agent returns with a question, she has to get back into it. Now she juggles several tasks at once, but the main one still takes three days to a week.
1
9
Each agent means she has to reload the map in her head, and none of them actually shortens the task she cares about. Same for me. How many agents can you use before it stops being helpful?
8
1/ A clean code review used to mean something. Now I'm not sure it means anything at all.
1
167
more replies
4/ One reading is that the code got good. The other is that the alarm broke.
1
9
5/ You can't tell those apart from the output. A smoke detector that never goes off looks exactly like a house that isn't burning.
10
Manning emailed me that Grokking Concurrency is their Deal of the Day - half off today only. I'm bad at this promo stuff and honestly had to dig up the link, but it's a genuinely good deal so I'd feel dumb not telling you. hubs.la/Q03-d27Y0 - ends midnight ET.
Image from luminousmen (@luminousmen)
1
1
85
Your Spark job worked fine in dev. In prod it takes 4 hours. Welcome to the club.
1
1
95
more replies
The rest of it Joins that explode (broadcast, skew, bucket) JDBC reads that are silently single-threaded Spark 4.x defaults you probably haven't turned on yet What to actually look at in the UI before you guess
1
62
The answer is almost always in the UI. Every item came from a real production screwup. Full checklist: luminousmen.substack.com/p/t…
48
A data engineer on Reddit got drunk and wrote down everything he learned in 10 years. The post lives on. I agree with almost all of it. Preserved it here: luminousmen.substack.com/p/d…
1
2
207
"How many unique users did X AND Y?" - simple question, brutal at scale. Theta sketches solve it. But the gap between the algorithm and not getting burned in production is huge. Wrote the guide I wish existed: luminousmen.substack.com/p/m… #dataengineering
85
Your COUNT DISTINCT has been running for 3 hours. Your cluster is on fire. Nobody tells you early on: exact answers to simple questions can be impossibly expensive at scale. Data sketches fix this. Here's how 👇 #dataengineering
1
1
1
119
more replies
The real breakthrough: sketches are mergeable. Build them in parallel on hundreds of machines. Merge in milliseconds. Handle late-arriving data by merging it in. Pre-compute at ingestion and store a 2KB "unique users" column instead of 2GB.
1
40
The full sketch family: - Theta/HLL → COUNT DISTINCT - KLL/REQ → percentiles & distributions - Frequent Items → heavy hitters - Reservoir Sampling → representative samples Each has sharp trade-offs. I break them all down in the full post: luminousmen.substack.com/p/t…
40
Load more