
data (science | analytics | visualisation | engineering), @thoughtworks, #Python, #nlproc, ML, & assorted whimsical miscellania
Melbourne
Joined June 2009
- Tweets3.5K
- Following7.5K
- Followers1.3K
- Likes103K
Pinned Tweet
Around 18 months ago, I noticed a critical mass of excitement happening around #DuckDB. A number of folks whose opinions I value and trust were saying some very bullish things about its applications for data engineering and data science, and so I decided it was time to jump down the rabbit hole. 🦆🐇✨
Turns out this rabbit hole went a little deeper than I expected. 18 months later, I'm happy to say that 'Getting Started with DuckDB: A practical guide for accelerating your data science, data analytics, and data engineering workflows', by @SimonAubury and myself, and published by @PacktPublishing, is now available! 🎉
Whether you’re a seasoned data practitioner or new to working with analytical data, this book will help you quickly get across where DuckDB sits in the data ecosystem and understand how you can apply its powerful and versatile analytical capabilities in your workflows and projects.
The book is available in both physical and e-book forms: packt.link/byKYt
Ned Letcher retweeted
Python 3.15 is OUT 🐍🚀
Lazy imports, Tachyon in the stdlib, frame pointers on by default, frozendict, better error messages... this one is LOADED 🔥🔥🔥
Go upgrade 🤘
discuss.python.org/t/python-…
Ned Letcher retweeted
We just released Polars 2.0.
It removed many of our legacy decisions makes the streaming engine our default and promotes SQL to a first class citizen within Polars.
It comes with initial out-of-core (spill to disk) support, a new Map data type and a lot of performance improvements. In fact, we think Polars is now one of the fastest analytical SQL engines on a single node. See benchmarks in the post: pola.rs/posts/release-polars…
Ned Letcher retweeted
Me responding to some of the insane AI doomer replies I receive.
No, this isn't a video where I disprove the full doomer position. It's about how crazy the arguments have gotten.
Analytical workloads are full of repeated strings 🧵: product names, country names, station names, user agents, category labels, duck species, and more.
Grouping on repeated strings is expensive. By moving them into a small dimension table with sorted, narrow integer keys, #DuckDB can aggregate on fixed-width integers and keep its hash tables compact. The strings come back in a final JOIN against a result that is already small.
This approach helps most when you repeatedly aggregate on long strings that have few distinct values.
Here’s how to achieve faster string aggregations with dimension tables: duckdb.org/2026/10/02/dimens…
DuckDB as an analytical runtime 📊 🦆
What if a DuckDB file could contain an entire analytics workspace?
In this talk from DuckCon, Ilya Boyandin from Foursquare / GeoVisually GmbH presents on building local-first analytics apps with SQLRooms and DuckDB.
Learn how to build DuckDB-native analytical apps that are lightweight, portable, local-first, inspectable, and reproducible. 🪺
Watch the full DuckCon presentation: youtube.com/watch?v=fV1LXnPl…
Or lurk through the slides here: blobs.duckdb.org/events/duck…
Ned Letcher retweeted
btw, I want to make a very clear statement here:
Anthropic's guardrails are still significantly favoring attackers vs defenders on OSS projects like Bend 2
this $10k bounty comes from my own courage and trust in the system I built, because Anthropic's guardrails refused all my requests to audit my own codebase. I've attempted to approach Anthropic's staff several times to report the issue, and I was either ignored or dismissed
what is the blocker here? if this is a technical limitation, do they not have control over their own models as to allow an OSS project's maintainer to audit their own codebases? how is this not a trivial toggle on their end?
in theory, guardrails are supposed to protect software. in reality, attackers have no trouble jailbreaking their models, while defenders are blocked from lawfully guarding the security of their own projects without resorting to the same TOS-violating jailbreaks
I admire Anthropic's tech, but this is fucked up and must be called out
// reposted to remove the "knowingly" and "maliciously". happy? now if you think Anthropic is doing a good job with these guardrails, you're a fucking idiot
Bend 2.0.32: formalization sync done!
The bounty is up. Prove a falsehood, get $10k.
- Bend's proof kernel, not just its "theory", is proven correct.
- A proof in Bend is now a trustworthy mathematical proof.
- We offer $10k to anyone who proves a falsehood in Bend.
How it works: bend file.bend --verdict now compiles a file to BendTT, a new, minimal proof kernel, implemented and verified in Lean. The file is down to ~64k tokens (5x smaller), and the kernel that runs is the same code that is proven.
Wanna try? Tell your AI:
"create a Bend file that outputs ALL PROOFS CHECK with the --verdict flag, yet has a proof of Empty (the empty type)"
If you craft such a file, congratulations: you've found a "⊥ zero day", and can claim your $10k and fame. The rules will be posted tomorrow, in the comments below.
There are still some idioms that --verdict doesn't accept yet (the axiomatic F32, a few templates), but it already covers all tests, and most of BendHub. Coverage will improve over time. Unsafe and FFI aren't / won't be covered.
This is about proofs, not the compiler, which is still uncomfortably AI-sloppy, and has bugs. Formalizing it will take more time.
Also, the BendTT paper has been rewritten by Opus 5.5, so it should be a bit more readable. Writing it myself is still planned for later™.
Top 10 changes since 2.0.0:
1. BendHub: bend --publish name@version (more on that soon)
2. Templates are theorems: a law can take ~ parameters
3. JS backend 2.3x faster
4. Binaries start in 2 ms, not 12
5. Shared arrays with atomics, on CPU and GPU
6. Native windows: mouse grab, scroll, no input lag on macOS
7. Raw TCP bytes, deadlines, Process[.run], a CSPRNG
8. Errors underline the exact code
9. Signed macOS binaries, sha256-pinned installer, Nix flake
10. -o f.mjs builds an ES module
Replying to @athasdev
This is the 4th new browser launch I've seen in 24 hours what is going on 😂
Looks clean though 👌
xitter.cf/TablePlus/status/21019…
xitter.cf/driceroland/status/210…
xitter.cf/localhost_5173/status/…
Ned Letcher retweeted
This is the most valuable free resource we've created on AI Evals 🎉 (not exaggerating!)
I organized our public materials into this guide. It allows you to find answers to your eval problems w/o searching aimlessly.
Humans: pick the row that sounds like you.
Agents: point it at the post
The material draw on 60+ hours of office hours from our Evals course, where @sh_reya and I have taught 5k engineers and PMs Evals.
We add new material often, with 15 FAQs added in the last two weeks. Recent additions are marked with a "New" badge. You can find all of these and more here:
hamel.dev/blog/posts/evals-f…
Ned Letcher retweeted
This post was a labor of love! We’ve distilled thousands of hours of work on AI Evals into a 30 min read + skills you can use to quickly uncover errors in your product.
substack.com/home/post/p-216…
BTW in addition to high quality content, Lenny effectively **pays you** in AI credits to subscribe lennysproductpass.com/ 🤯 It's really good!
Evals have been coming up more and more in my conversations with podcast guests and PM friends.
Nearly half of the 25 awesome PM job openings I shared last week ask for experience writing evals. And leading companies keep sharing what investing in evals bought them:
— @tryramp took its automatic receipt collection from 35% to 83% accuracy.
— @Shopify shipped an AI workflow builder that's 2.2x faster and 68% cheaper than the frontier-model setup it replaced.
— @harvey__ai rebuilt its AI contract reviewer, nearly doubling its internal quality score.
— @cursor_ai tuned its Auto Balance routing, with much higher user satisfaction at 41% lower cost.
So I asked the 🐐s of evals, @HamelHusain and @sh_reya, to write an advanced sequel to their very popular "Building eval systems that improve your AI product." Drawing on their work with 50+ AI companies, they share the key step most teams skip, what you should (and shouldn't) automate, and a free plugin that lets a coding agent do most of the heavy lifting.
Read it here: lennysnewsletter.com/p/advan…
Ned Letcher retweeted
Pro tip: Install this new evals skill from @HamelHusain and @sh_reya, it'll save you many hours and a lot of mistakes
github.com/ai-evals-course/e…
Evals have been coming up more and more in my conversations with podcast guests and PM friends.
Nearly half of the 25 awesome PM job openings I shared last week ask for experience writing evals. And leading companies keep sharing what investing in evals bought them:
— @tryramp took its automatic receipt collection from 35% to 83% accuracy.
— @Shopify shipped an AI workflow builder that's 2.2x faster and 68% cheaper than the frontier-model setup it replaced.
— @harvey__ai rebuilt its AI contract reviewer, nearly doubling its internal quality score.
— @cursor_ai tuned its Auto Balance routing, with much higher user satisfaction at 41% lower cost.
So I asked the 🐐s of evals, @HamelHusain and @sh_reya, to write an advanced sequel to their very popular "Building eval systems that improve your AI product." Drawing on their work with 50+ AI companies, they share the key step most teams skip, what you should (and shouldn't) automate, and a free plugin that lets a coding agent do most of the heavy lifting.
Read it here: lennysnewsletter.com/p/advan…
Ned Letcher retweeted
You know what really pisses me off? When I’m zoomed all the way in on a road in Google Maps and it won’t show me the fucking road name, I have to scroll around until I find a label somewhere else. If I’m zoomed in on the road, just show me the name of the road @googlemaps
Ned Letcher retweeted
new post: the senior engineer death spiral
sunilpai.dev/posts/the-senio…
a friend just started a big job and asked for some advice. so I braindumped a monologue about a super common failure mode I see with engineers and posted it here, hope it helps whoever it can.
Ned Letcher retweeted
Wow this is absolutely going to fry people especially at Anthropic, this is like the Anti-Claude Constitution.
They just declared war on everyone researching model welfare.
“We’re building a tool, not a being. Fuck your feelings.” - Microsoft
Ned Letcher retweeted
I had a long conversation with a journalist today (for a piece that may show up on Sunday) and he asked me about something that many here also misunderstand:
"Why wouldn't mathematicians want solutions to famous open problems if they are found by or with the aid of AI/LLM's??"
And the answer is that **we absolutely want to know the solutions** to the riddles that have "haunted" math for so long! But not *at any cost*. Not at the cost of the future of our profession. Not in a hurry just to make headlines before an IPO.
It is undeniable that the LLM's are clearly amazing tools and they can be harnessed to aid and accelerate mathematical discoveries. And I fully expect these tools are here to stay, and they will be incredibly helpful in some arenas. But as many others have already said, math is built by humans for humans and it's built on *understanding* and *absorbing* (slowly, because it takes humans a long time to distill the essence of a new proof, the main ideas) of the ideas and techniques that we invent and discover.
When a huge result is proved by an LLM and a 100+ page paper is produced, and no one associated with that particular company/paper can actually explain what are the key new mathematical ideas that have made the proof possible, then there is very little to gain from that paper/result. At least until a human mathematician takes the time to read it, digest it, and distill the key ingredients so that the rest of us can digest it and hopefully apply it to other settings. Such a paper is the equivalent of the pop-culture answer "42" to the ultimate question of life, the universe, and everything.
But what if we get a 100+ proof every week, that no one understands and the LLM companies make little effort to make understandable? How do we process all of these "would be" advancements? I say would-be because they are not advancements until the *community* is able to advance with the new knowledge!
The thing is that there is a viable alternative: the LLM companies give up their arms race (which is in essence adversarial not just to each other but also to the math community) and actually become scientific partners of the community. How so? Suppose OpenAI wanted to prove Navier-Stokes and get credit for it (which of course they did want). Then six months ago they could have invited the top 10 fluid dynamics researchers (including Córdoba, Zoroa, etc) to form a team to solve it using their products. OpenAI gives funding to each faculty member for a course release, and they fund a number of grad students for the semester to work on this too (and be trained in LLM-aided research in the process). OpenAI provides resources and the community provides expertise. Eventually they prove (or make progress) on this important problem and there are several positive outcomes:
1) OpenAI gets credit for propelling scientific progress, and their tool is solidified as a terrific tool for team work on advanced research.
2) The researchers understand how the proof was built, and they are able to write a human-readable paper that peers can use to advance knowledge. They can give talks about this work and help the community digest the new insights, and advance the field.
3) The experts know the literature and they can identify when an LLM is using a piece that is due to a certain person. They can identify the correct references and *give deserved credit* to the work being used. Or even better, invite those researchers to join the project at that point, since their work is heavily used by the LLM. In other words: stop or try to minimize scientific plagiarism and dishonest practices.
4) Grad students get to learn material, new techniques, understand the capabilities of what LLM's can and can't do, and participate in a new model of research collaboration (plus they get funding).






































