Expert in tackling complex software challenges, with deep knowledge of TCP, MySQL and PostgreSQL kernels, and a passion for AI, history, math, and physics.
Beijing, China
Joined October 2012
- Tweets3.3K
- Following136
- Followers17.8K
- Likes1.4K
Pinned Tweet
After nine months of hard work, this Postgres course is almost complete. You don't have to buy it, but if you want to truly understand Postgres internals, this is one of the most detailed courses you'll find. wangbin579.gumroad.com/l/pos…
Once you understand how Rust avoids C's pitfalls, you'll see that Rust itself is actually quite elegant. However, the Rust code most people write is so ugly that it scares many people away.
Neon is another project where I can already see a lot of room for performance improvements.
You can't get a 100,000x speedup just by improving cache locality. Such big performance gains come from many things working together, but the most important thing is choosing the right data structures for large datasets.
Hard to say, honestly. It really depends. The biggest speedup I've ever seen was changing the data structure from a hash table to a well-designed array-based one. Overall, not just from cache locality, that made part of the code about 100,000 times faster.
I have spent the past ten days checking my new home, trying to find as many problems as possible, and asking the workers to fix them. btw, AI is still not very helpful in solving these problems.
I've traced through lsm myself, and I don't think it's the best data structure for neon. I choose more suitable data structures based on many factors, which I will share on discord (discord.gg/g8ksTXPPnY). Btw, good database design isn't something AI can do today.
I really like Postgres now. Once you understand it, almost anyone can build a database. It's not just for experts anymore.
Based on Neon's architecture, I abandoned the LSM tree and designed a new data structure tailored to the data's characteristics. It looks simple and efficient so far, though it hasn't been implemented yet. Anyone interested?
Using arrays well is not easy. It takes skill, and you need to really understand your data and how it's structured. That's why programming is truly an art. In the future, poorly made databases will be everywhere.
In performance optimization, the data structure I least want to use is a hash (only as a last resort), and the one I most want to use is an array. When a database is full of hashes, there's probably a lot of optimization opportunity hiding in there.
I think I've got the evidence now: in tokio's async model, Rust is far weaker than C in some critical situations. Full breakdown in the Discord group next week. Don't miss it.
I don't even like C++ and I don't want to use any C standard library, let alone Rust. Most of the time, good optimization is just about getting your algorithms and data structures right, which is something you have to figure out yourself, case by case.
Why C? Four reasons. 1) Postgres is written in C too, and I want to reuse its WAL replay code. 2) I don't want to change my taste. I just don't like Rust, which misses a 't' 3) I don't want slow builds or big binaries. 4) I'm not smart enough for Rust.
Our product will be written in pure C (sorry, I don't trust Rust). Even if nobody ever uses it, it'll still show you what the art of optimization really looks like.
The course will keep getting deeper, since I'm growing fast. After studying Neon carefully, I'll use TigerBeetle's ideas to build a faster version that separates storage and compute, making the course even deeper. Technically, I'm sure I can build a great database product.
If you've been at Postgres for over two years and still don't get it, you're using the wrong approach.
Two years ago, I was a Postgres newbie. I didn't even know how to install it. Now I feel like I really understand Postgres. Learning Postgres is only hard for DBAs. For developers, it's easy, and the ideas behind it are even easier, as long as you use the right approach.