A brief re-introduction for all my new followers. I’m Ergest (rhymes with “her guest”) I’ve spent the last decade and a half in the field of data working on everything from data engineering to data science. In 2022 I self published a book called Minimum Viable SQL Patterns (link below) In 2023 I started a newsletter called Data Patterns that combines technical aspects of data with business impact (link below) My current focus is on upgrading the methodologies used in the field of data.

I thought a really heavy truck passed outside!

There’s only two paths for dashboards: either they never get used or they end up becoming complex applications that people use to run SQL queries and export the results to Excel.

Today I (legitimately) used a RIGHT JOIN for the first time in nearly 20 years of writing SQL. AMA

The effort required to write a good LLM prompt is nearly equal to the effort required to design a good system. Because GIGO. Similarly the amount of effort and knowledge required to judge a good output is nearly the same so a great understanding of the fundamentals of software engineering is still essential. Therefore the only advantage of an LLM is the ability to generate code that matches your design faster than you can type it. Still I’ve been quite amazed by what even minimal prompting can do assuming I know exactly what I’m doing

The only way to succeed as an AI company long term is to collect and hoard proprietary, hard to get data.

The true purpose (and promise) of analytics has always been uncovering asymmetric opportunities hiding in plain sight. Things like underpriced assets, leverage points, etc. Let's look at some examples: In the movie Moneyball (based on the book by the same title) analytics is shown to uncover asymmetric returns from a simple baseball statistic called on-base percentage (OBP). Since what you want is to score more runs than your opponent and the best way to score runs is to get on base, it makes sense that players that get on base a lot increase your chance of scoring more runs. The reason why players with high OBP were undervalued was because the prevailing notion was that the only stats that mattered were batting average (AVG) and home runs (HR) Players with high OBP but low AVG and HR were frowned upon. Nowadays this is common sense and many teams already do it therefore the leverage point has to be found elsewhere and has become far more obscure. Don't like sports? Take quantitative trading. The whole point of factor analysis is to find a way to price assets such that you pay as little as possible for as high a return as possible. Factor investing for instance is the study of financial metrics that indicate good companies to invest in. In business, uncovering a single leverage point could be worth hundreds of millions of dollars. For example finding that 90% of revenue comes from 10% of clients allows you to focus your efforts on those clients, maximize their impact then use ML/DS to profile them and find more of them. THIS is the real power of analytics and the BEST value it can offer a business.

It’s giving RedSox circa 1986

There are two trends in the field of data I’m following very closely: 1. The rise of the “AI analyst” including agentic AI, text to SQL, etc. 2. The rise of commodified data warehouses. The first one is doomed to fail without a solid foundation of a well designed warehouse, clean data, well defined metrics. Imagine selling luxury sofas to home owners who haven’t even built the foundation yet. The second one I’m very excited about because it’s a huge opportunity to instantly level up a large number of companies and can also fuel the growth of the first trend.

Watching the comeback on Netflix about the 2004 Boston Redsox, which I saw live 20 years ago and I’m shocked at how many of the player names I remember!

I thought I’d never get here but my musical tastes have evolved to favor complex compositions. So for example: 1. I no longer listen to pop 2. Hard rock I used to favor in high school now sounds like pop 3. Power metal I got into later now sounds like hard rock 4. I’ve started to listen to more symphonic metal (specifically Epica, Kamelot, Nightwish) and some darkish metal stuff (Wintersun, Insomnium, Sleep Token) Any recommendations for similar bands?

Isn’t it interesting that where the Segway failed as personal transportation the electric scooter has succeeded?

The next evolution in AI will be faster algorithms for doing linear algebra

I’ve been trying to figure out what to focus my writing on in forever. I’ve gone back and forth between technology and business never quite choosing one or the other until today something clicked and I came up with this! It’s both and more. I’m calling it the data leadership triangle because it encapsulates three core themes of my writing and even though I haven’t spent much time talking about power, influence and organizational politics, it’s still an area of interest. I will expand on this in a future post but for now I want to get the idea out. This triangle represents the three core skills of a great data leader. 1. Understanding the business and building a metrics powered growth / operating model, which I believe is the highest value a data team can provide. 2. Understanding the data technology stack enough to make good choices for long term success. This means understanding the risks and costs of the data architecture 3. Understanding influence, power and politics which are crucial to success in any leadership role but which are never discussed openly.

Sure you can use AI to generate code or a design for you, but if you know nothing about software engineering how can you tell it’s doing the right thing?

In the age of AI slop books will again become important

The problem of “too many questions on the data team” is really a problem of separating the few questions that really matter from mere curiosities.

When you frame data as a technical discipline you doom it to become a cost center. Instead frame it as a business discipline and watch it blossom.

I wasn’t always comfortable writing online, in fact I was terrified. I published a few random articles that didn’t go anywhere. But I really wanted to write a book and writing online is a great way to start. I just couldn’t find a topic. Even when I took David Perrell’s course Write of Passage I tried random ideas that didn’t stick. I finally struck gold when I joined Daniel Vassallo’s course Small Bets Portfolio. He had written a book called “The best parts of AWS” which inspired me to write something like “the best parts of SQL”, a few key things you needed to know outside of the language syntax. I landed on the term “sql patterns” which after a brief research when I discovered no one was taking about it. Despite there being thousands of books on SQL almost all of them focus on how to query data whereas I was using SQL like a programming language to transform data. Since I was taking this course on “small bets” and Daniel himself had initially published his book as a Twitter thread, I decided to do the same thing. I published my “key SQL patterns” thread in January 2022. It took off! I added more than 1000 new followers overnight! I took that as a strong signal to write a book. Once I finalized the outline, the book just poured out of my head and 3 months later I published it. If you bought it, thank you for your support and please leave a review on the website. If you want to check it out, here’s the link. ergestx.gumroad.com/l/sqlpat…

SQL made its debut in a paper published in 1974 as SEQUEL. 50 years and many iterations later we’re still using it. The power of SQL isn’t so much about the language but rather about the query engines of modern databases. Up until very recently you couldn’t run a query outside of a database management system, many of which still require a knowledgeable DBA to run. Recent advancements have made it possible to separate the query engine from the storage system of the database, finally liberating SQL as we know it. Just look at DuckDB and the rapid development of its SQL implementation. Not only have they introduced “syntactic sugar” but with many extensions you can query data however you like. For instance you could invert the order of SELECT and FROM to be more natural: FROM tbl SELECT i, s; Or check out these beauties: SELECT * EXCLUDE (col) FROM tbl; SELECT * REPLACE (col / 1000 AS col) FROM tbl; More recently I learned about the duckdplyr extension which lets you query data in R using DuckDB as a backend. These language advancements would not be possible without the logical separation of data storage and query engines. You no longer need to focus on database specific optimizations, just store data in Iceberg, Delta or even CSV format and use any number of query engines to process.

I’ve mostly stopped posting here. You can find me on LinkedIn, I’m more active there