CTO of @InfluxDB (YC W13), founder of NYC Machine Learning, series editor for Addison Wesley's Data & Analytics, author of Service Oriented Design with Ruby.

New York City, NY
Joined February 2008
15
74
11
623
27,839
Boston’s @ApacheDataFusio meetup last week was a blast. Huge thanks to @andrewlamb1111 and to Pratibha Rana + @datadoghq for bringing the community together. A special thank-you to @pauldix for years of visionary support for DF (and for believing in @Bauplan_labs!): review in 🧵
2
2
5
325
Paul Dix retweeted
Replying to @MarcJSchmidt
I'm not commenting on Rust vs. Zig. I'm more commenting on the fact that AI wrote 1M LOC in 11 days and within a few months that code is running in production on millions of developer machines and it all "just works". Software development is changed forever. And it's magical.
2
4
1
40
2,083
Paul Dix retweeted
Time series workloads rarely stay simple. Fleets grow, schemas get wider and sparser, query patterns shift, and the database needs to stay fast under heavier load. #InfluxDB 3.11 is built for that reality, with a significant performance upgrade for demanding time series workloads: • Up to 4x faster for highly selective, single-series queries • Support for wide and ultra-sparse schemas • More predictable performance under heavy ingest and compaction • New Enterprise capabilities for backup and restore, bulk data import, and data management • The Explorer UI built directly into InfluxDB 3 Core and Enterprise The result: faster, more predictable performance as time series workloads grow in scale and complexity. See what’s new in InfluxDB 3.11: bit.ly/4x9gM2H
4
4
1,494
Paul Dix retweeted
Claude
@claudeaiJul 24
Introducing Claude Opus 5. It's a thoughtful and proactive model that comes close to the frontier intelligence of Fable 5 at half the price.
3,498
7,453
5,249
61,212
24,371,613
Tired of text-to-SQL and “agentic analytics” demos where the real bottleneck is data engineering? Me too. The next frontier is not data agents answering questions but changing the data estate. Introducing GitLake!
5
22
1
181
13,064
I was recently diagnosed with a rare autoimmune disorder called anti-NMDA receptor encephalitis. It's a "disease of chaos" that completely upturned my life for a couple months. I wrote a blog about it that goes into more detail and discusses prognosis. burntsushi.net/encephalitis/
35
92
9
437
55,721
Paul Dix retweeted
Vortex 🤝 Apache Iceberg Coming soon! @vortexdotdev @ApacheIceberg
2
15
4
72
8,368
Today’s WSJ reminds me that ammonia is bonkers! I’ll add my historical book review below, but some teasers: • Ammonia synthesis keeps half of humanity alive • Half of the nitrogen in your body is synthetic • BASF made “the biggest scientific effort in history” to find a catalyst — “the most important discovery ever made.” (bold claims from Alchemy of Air) Fast forward to today, and the industry is working toward a carbon-free ammonia future, and better catalysts are needed once again for the post-oil economy. We invested in Copernic to design those novel catalysts: coperniccatalysts.com/techno… WSJ today: “Ammonia doesn’t produce carbon emissions when burned as fuel, driving a new wave of activity from companies angling to ship it all over the world. Increasing domestic supply is a priority after Russia’s invasion of Ukraine in 2022 roiled the market and pushed up prices. Russia is one of the world’s biggest producers. ‘It’s important for us to have a diverse supply chain and make sure we’re not dependent on other countries for this really important chemical,’ Jigar Shah, head of the Energy Department’s Loan Programs Office, said in an interview.” — wsj.com/business/energy-oil/… And here’s my book summary, one of my all-time favorite works of non-fiction: Hager’s Alchemy of Air is the story of “the most important discovery ever made. See if you can think of another that ranks with it in terms of life-and-death importance for the largest number of people. Put simply, this discovery is keeping alive half the people on earth.” The Haber-Bosch process catalyzes the production of ammonia (NH3) from N2 and H2 gas. We need “fixed nitrogen”, available to our organic chemistries as atomic nitrogen. It is the limiting factor for the growth of all food. While nitrogen gas is about 80% of our atmosphere, not one atom of it is available for our use when tightly bound by the triple bond of N2 gas, the strongest chemical bond in nature. It is sequestered all around us. In nature, N2 is liberated to atomic nitrogen in small amounts by lightning strikes (it needs 1000°C) and slowly by nitrogen-fixing bacteria in the soil. Hager argues that if we reverted to relying on just those natural sources, three billion people would die of starvation in short order — our soils simply could not produce enough food for the mouths now on Earth. Historically, fixed nitrogen came from manure and compost, the first supply to be used up. Once local bat caves were depleted, the world looked for new sources. In 1850, the Chinchas Islands off the coast of Pisco, Peru became the most valuable real estate on Earth. They were covered by centuries of bird guano, excavated by slave labor in a hellish scene. Guano became 75% of the GDP of Peru by 1859. But the world wanted more. In 1856, the U.S. passed the Guano Islands Act, whereby anyone could annex a guano island they found anywhere in the world and make it a U.S. territory. Then the Guano War of 1863 broke out between Spain and Peru and Chile. Darwin had discovered peculiar nitrate deposits in high Atacama Desert of Chile. And by 1900, Chile was supplying 2/3 of all fertilizer on Earth. The Chilean harbor was the location of the first major sea battle of World War I, between France and Germany. The nitrate supply was essential to war-making. “They later called World War I the chemist’s war.” As we saw in the massive explosion recently in Lebanon, fixed nitrogen can also be used to make explosives or provide the “N” in TNT or nitroglycerine. As a latecomer to the nationhood, Germany did not have colonies to exploit for food or fuel, and their shipping lanes were vulnerable to foreclosure by the British navy. Germany’s chemical companies undertook a major effort to pull fixed nitrogen from the air, to support local food production and the munitions of war. “BASF’s nitrogen project grew into the biggest scientific effort in history, comparable in scale to the Manhattan Project in WWII.” The goal was to find a catalyst that could assist with the required chemistries by reducing the temperature and pressure required to something that could be economically feasible in an industrial plant. After 20,000 experiments, running through the periodic table, they discovered osmium could do the trick, and BASF cornered the market for this rare element, but even that would not be enough for the volumes needed. Then they found uranium, and finally, a more reasonable iron-aluminum-calcium combination. The factories required staggeringly huge pressure vessels, like had never seen before. We use the same catalysts today, in a codependency with the petrochemical economy of byproducts and waste heat. The Haber process consumes 4% of the world's natural-gas production and 1.5% of the world's energy supply. Half of the nitrogen in fertilizer is taken up by plants, much of the rest washes out. Fertilizer runoff from the Mississippi river has doubled nitrates in the the Gulf of Mexico, making it look like chocolate milk with an enormous Dead Zone. In Europe, the annual 1.5b tons of nitrogen fertilizer runoff from the Rhine has made the Baltic Sea one of the most polluted marine systems on Earth. Even our atmosphere has become a “huge fertilizer silo, with tons of growth-promoting fertilizers showering from the sky. The amount of fixed nitrogen filtering down to earth in some places has risen so high that it equals the amount American farmers typically apply to their spring wheat.” Nitrogen oxides also create acid rain. We have become dependent on fertilizer. To recap, “while the population nearly quadrupled during the twentieth century, food production — thanks first to Haber-Bosch, second to improved genetic strains of rice and wheat — increased more than sevenfold. That is the simple math behind today’s era of plenty.” And 50% of the nitrogen in our bodies came from the Haber–Bosch process. It’s in every protein and every strand of DNA. Ponder that — “half of the nitrogen in your blood, your skin and hair, your proteins and DNA, is synthetic.”
29
33
4
242
48,767
Performance Hints Over the years, my colleague Sanjay Ghemawat and I have done a fair bit of diving into performance tuning of various pieces of code. We wrote an internal Performance Hints document a couple of years ago as a way of identifying some general principles and we've recently published a version of it externally. We'd love any feedback you might have! Read the full doc at: abseil.io/fast/hints.html
106
1,071
142
7,615
2,119,965
On Async Mutexes, matklad writes matklad.github.io/2025/11/04…
11
84
8,332
Paul Dix retweeted
ODBC is getting tired. It can't keep up with the fast new kids in the data world these days. The next generation is ready to take the torch. Meet ADBC, a fast, modern data connectivity standard built on @ApacheArrow. Watch my talk from the @CMUDB seminar: youtu.be/TjlmNGNx77E
16
1
69
13,364
Lots of database action this week. Yes, I have a new start-up @sydhtai with my PhD students (@lmwnshn + @17zhangw) using LLMs/ML to optimize almost everything in @PostgreSQL. The improvements are stunning! @datadictum posted a new article on our approach: theregister.com/2025/10/22/c…
3
13
131
8,605
Prateek Gaur and co at @Snowflake reproduced the (great) results for the ALP encoding algorithm from @cwi_da / @afroozeh3 / @peterabcz. ALP achieves ZSTD levels of compression and much faster decode. We are discussing adding it to @ApacheParquet: lists.apache.org/thread/tjtl…
9
1
75
5,304
Paul Dix retweeted
A deep dive on wal3 - Chroma's open-source write-ahead log built on object storage. Featuring: - a 30-year-old lock-free algorithm - Amazon S3's newest conditional writes feature - a novel checksumming technique called setsum
7
26
8
192
32,226
"EDB claimed the new engine, which pushes queries to open source @ApacheDataFusio , returned queries 30x faster than standard Postgres while tiering offloads cold transactional data to storage is up 18x more cost-efficient." theregister.com/2025/06/20/e…
4
30
1,328
What if your database could have ALL the memory? When you’re running a database at scale, the most expensive part is memory. Databases primarily use memory for their buffer pool, a local cache of data objects stored on disk. Because the buffer pool is a cache, databases don’t need all their memory all the time–a database might need hundreds of gigabytes of memory one minute for a complex query, but only a few gigabytes the next minute when idle. So there’s interest in disaggregating database memory, creating a large pool of memory shared across many databases so each uses only the memory it needs. Right now, many distributed databases disaggregate memory using a technology called RDMA (remote direct memory access). RDMA lets servers read and write memory pages on other servers. That’s powerful, but it has a few downsides: - While RDMA is faster than reading from disk, it’s still much slower than reading from memory. So databases can’t use RDMA in place of memory–they instead use it as an intermediate cache between memory and disk. - You can only read and write RDMA at the granularity of whole pages. That introduces a lot of amplification, where if you want to write a single value you have to read and overwrite its entire page. - There’s no cache coherency, so it's difficult for multiple database nodes to share a pool of disaggregated memory. This paper is a glimpse into the future beyond RDMA. It proposes an architecture for database memory disaggregation based on a newer technology: CXL. CXL is a cutting-edge interconnect that enables high-speed, low-latency communication between CPUs and devices (including memory). Compared to RDMA, it has a few advantages: - 5-6x lower latency - Native support for loading and storing individual values, drastically reducing write amplification. - Native support for cache coherency. Because CXL is so much faster than RDMA, databases can treat remote CXL memory exactly like local memory. This enables the new architecture proposed by this paper: use local memory only for local variables and data structures and put the entire buffer pool in CXL memory. This is simpler and faster than using RDMA–the transaction engine doesn’t need to know whether a data pointer is in local or CXL memory, CXL is fast enough that it just works. There’s far more detail in the paper than I have space to cover–they also propose a faster new recovery protocol based on CXL and optimizations enabled by nodes of a distributed database sharing a CXL buffer pool–so I definitely recommend reading it to learn more! For me, the big takeaway from this paper is how much software architecture is influenced by hardware capabilities. Most of us are used to the idea that memory is physically assigned to a server. RDMA lets you use remote memory at cost, and now CXL is providing remote memory that’s much closer in performance to local memory. If these technologies become more available, what else can we rearchitect, and how fast can we make it?
3
17
2
98
8,079