Former entrepreneur, current professor, starting author. I mostly write about AI/LLMs in science/psychology, with some random thoughts. Principle: make history.
Seoul; Portland
Joined July 2014
- Tweets153
- Following0
- Followers141
- Likes1
Pinned Tweet
One superpower of Gemini is its integration with so many products offered by Google, such as Gmail. This is how it helps manage my travel:
An essay on common misconceptions about language models, now published in PNAS Nexus (I wrote it because I needed a reference for teaching introductory courses on AI last semester, but could not find one!):
Abstract:
Large language models (LLMs) are now embedded in scientific, educational, and governance workflows, with debates centering on their capabilities, mechanisms, and impacts. Yet these debates remain structured by persistent folk theories—intuitive, informal explanatory models that guide attitudes and actions. Deflationary slogans (“just autocomplete,” “stochastic parrots,” and “average of the internet”) and anthropomorphic framings (“emergent agents” and “proto-minds”) each capture genuine features of current systems but mistake those features for the whole. This Perspective proposes a minimal working model of LLM-based systems centered on four distinctions: between pretraining and deployed systems; between the learned distribution and particular samples; among parametric, contextual, and external memory; and between task competence and agency. The model is used to diagnose six misconceptions about LLMs: next-token prediction, regression to the mean, training-data regurgitation, model memory, alignment, and understanding. For each, the analysis identifies what the misconception gets right, which distinctions it conflates, and what follows for capability evaluation, system design, and governance. Applied to publisher AI policies as governance case studies, the framework shows both how policy language can conflate these distinctions and how such errors can be corrected. The model thereby avoids the parrot–mind binary by treating LLMs as simulators of discourse and task performance, offering a diagnostic toolkit for locating and correcting the errors these folk theories perpetuate.
academic.oup.com/pnasnexus/a…
2026 Issue 7 of @PNASNexus is now available! Explore new research on topics including six misconceptions about large language models, the confident judgments of (mis)information veracity, and more. Explore now: ow.ly/VMLA50ZxGhT
ALT PNAS Nexus – 2026 Issue 7 is now available. Pnasnexus.org
It's really unfortunate that we easily fall into dichotomous thinking. Of course, writing and thinking are not the same, but at the same time, they also reinforce each other, quite greatly, I would say. So writing remains essential. (Writing this post, I use dictation, but oftentimes I also need to hand write, hand type to get my thought better sorted out.)
The head of the philosophy department at Williams College downplays the pedagogical value of writing:
“What if we are mistaken in treating writing as the highest cultivation of thought? The evidence that supports this belief is thin at best. Writing can be useful cognitive scaffolding for memory and self-regulated learning, but the notion that it is the best process for developing thinking skills is a fairly recent idea, traceable to the 1970s writing-across-the-curriculum movement.”
Our recent comment is now out in Nature about the importance of incentive structures to shape a future where AI can expand instead of contracting our research horizon:
Funders, journals and universities must reward the creation of new scientific terrain, not just the efficient mining of old ground
go.nature.com/3VlURaT
Image from Gemini's new Nano Banana Pro. Very impressive for scientific illustrations (teaching or just sketching out ideas).
You can also ask Gemini to create XML code (left) to be imported into draw.os (right), which is great for vector graph and for manual editing:
Prompt:
Image of two-panel conceptual scatter plot visualizing Human-AI complementarity. The figure title is "Human–AI complementarity as correlation vs augmentation".
Panel A on the left is titled "Emulation: human-like cognition" with the subtitle "High correlation of errors". The X-axis is labeled "Human performance (across decision situations)" from "Low" to "High". The Y-axis is labeled "AI performance (across same situations)" from "Low" to "High". A dense cloud of points is clustered tightly along the diagonal from the bottom-left to the top-right corner, indicating a strong positive correlation. A red-tinted rectangle in the bottom-left corner covers the area where both human and AI performance are low, and it is labeled "Shared failures (correlated errors)". The points show that when human performance is low, AI performance is also low, and when human performance is high, AI performance is high.
Panel B on the right is titled "Augmentation: targeted complementarity" with the subtitle "AI accurate where humans are weak". The X-axis is labeled "Human performance (across decision situations)" from "Low" to "High". The Y-axis is labeled "AI performance (across same situations)" from "Low" to "High". The points are now concentrated significantly in the top-left corner, showing high AI performance where human performance is low. There are still some points along the diagonal and in the top-right, but the bottom-left corner is relatively empty compared to Panel A. A green-tinted rectangle in the top-left corner covers the area where human performance is low and AI performance is high, and it is labeled "Targeted gains (AI strong where humans are weak)". The points demonstrate low correlation, with AI performance being particularly high in situations where human performance is low.
Analyzing 764 retracted Al-related publications indexed in PubMed, this paper highlights that China is the leading contributor to these retractions (n=551), followed by India and Bangladesh. The primary drivers for retraction were found to be peer review issues and data concerns, pointing to systemic vulnerabilities such as paper mills and compromised editorial processes. Additionally, a significant portion of retractions involved the unethical use of Al, including unreported Al-generated content and data fabrication.
jkms.org/pdf/10.3346/jkms.20…
From Seeing to Doing by Fei-Fei Li at NeurIPS, posted Nov 7, 2025
youtube.com/watch?v=UxDMA8kS…
Some interesting points with highlights:
- In the early 2000s, computer vision was in its infancy, struggling to define its goals until it settled on object recognition as a "north star problem."
- The Biederman number: a conjecture by psychologist Irv Biederman that humans can recognize "tens and thousands of object categories" by age six.
- Progress in visual understanding has been driven by better representation learning, moving from ResNet to masked autoencoders (MAE) which learn representations in a "self-supervised way." Recent advancements have extended this to video through Siamese MAE, which encodes "temporally ordered pair-wise frames" to achieve better generalization across tasks like reconstruction and semantic segmentation.
- Utilizing scene graphs, models can now understand complex interactions, such as distinguishing a person sitting on a chair versus a "person sitting on fire hydrant," a capability bolstered by the Visual Genome dataset containing millions of annotated relationships and attributes.
-The journey toward generation began with visual style transfer and early experiments with Generative Adversarial Networks (GANs) and Variational Autoencoders. Initial attempts to generate scenes from text required translating sentences into scene graphs and layouts, resulting in "blurry" and low-resolution output that was nevertheless a significant historical milestone just six years ago.
-The field rapidly advanced with transformer-based diffusion models, exemplified by work like W.A.L.T, a video generation model that encodes both images and video.
- New approaches combine dense reconstruction with generation, such as ZeroNVS and Zero-1-to-3, which allow for "360 degree view synthesis work from a single image." These techniques can also reconstruct "dynamic human movements" even under heavy occlusion, outperforming previous methods like HumanNeRF in both quantitative and qualitative measures.
-To address the brittleness of current robots, the BEHAVIOR project provides a benchmark for "everyday household activities in virtual, interactive and ecological environments." With over 1,000 tasks, 50 scenes, and 10,000 object assets (possessing properties like "articulation, deformability"), it allows for the instantiation of an "infinite number of synthetic environments" to train robust robotic policies.
-Data collection extends beyond vision to include sound and touch, as seen in the ObjectFolder dataset of "multi-sensory objects." Furthermore, to solve the sim-to-real gap, the concept of the Digital Cousin was introduced: unlike a precise digital twin, a digital cousin "strives to preserve similar geometric and semantic affordances," making it "less costly" and "more scalable" for generalizing robotic learning.
Nice work on AI math performance:
- AlphaProof combines the game-playing logic of AI systems like AlphaZero with the rigorous, verifiable environment of the Lean proof assistant.
- The system's core is an "AlphaZero-inspired" agent trained using reinforcement learning (RL) on a massive, custom-generated curriculum of 80 million formal math problems.
- For exceptionally hard problems, it uses a novel technique called Test-Time RL (TTRL), where it rapidly learns from "millions of related problem variants" generated specifically for the problem at hand.
Excited to announce our recent @GoogleDeepMind paper, AlphaProof, out in @Nature today!
It has been over a year since AlphaProof achieved silver-medal standard solving International Mathematical Olympiad (IMO) problems, by teaching itself mathematics in LEAN (@leanprover). This milestone was special to me because it was often hard to believe we would reach it.
You’re crazy until you're successful
Blog: tomzahavy.com/post/how-we-ac…
Paper: nature.com/articles/s41586-0…
Cool work, finding:
- Deep neural networks often fail to generalize as robustly as humans because their underlying representations differ. Human conceptual knowledge is "hierarchically organized from fine- to coarse-scale distinctions," whereas model representations "do not accurately capture all these levels of abstraction"
- The AligNet solution: To address this, a "teacher model" is first trained to imitate human judgments (the THINGS dataset). This teacher is then used to generate AligNet, a large-scale dataset of "human-aligned pseudolabels" (triplets sampled from ImageNet). Essentially, the triplets task = given three images, which two are most similar, and which one is the "odd one out"? For example, a triplet might be: Image of a Poodle, Image of a Golden Retriever, Image of a Llama). A "pseudolabel" is a simulated or generated label; instead of having a human provide the answer for every single triplet (which would be impossibly time-consuming for millions of images), the researchers use the "teacher" AI model to generate the answer. The pseudolabel is a "soft distribution" of probabilities (e.g., the teacher model might say: "I'm 95% sure the Llama is the odd one, 3% sure the Poodle is, and 2% sure the Golden Retriever is"). State-of-the-art vision models ("students") are then fine-tuned on this dataset.
- The resulting "human-aligned models" (AligNet models) "more accurately approximate human behaviour and uncertainty" across various similarity tasks. They also "perform better on a diverse set of machine learning tasks," demonstrating increased "generalization and out-of-distribution robustness."
- Local vs. global structure: Models are often good at encoding "local human-perceived similarity structure" (e.g., different dog breeds) but fail to model the "global relationships between concepts" (e.g., dogs and fish, which are both animate).
What aspects of human knowledge do vision models like CLIP fail to capture, and how can we improve them? We suggest models miss key global organization; aligning them makes them more robust. Check out @lukas_mut's work, finally out (in @Nature!?) + our new blogpost! 1/4
ALT Representation space: messy vs. organized after our alignment method
Interesting talk on AI search optimization:
- New AI-driven search trends like GEO (Generative Engine Optimization) or AEO (Answer Engine Optimization) are not a replacement for traditional SEO but its "newest evolution".
- The fundamental tactics remain the same; concepts like content chunking, multimodal content, and query fanout are all rooted in existing SEO best practices like good formatting, digital PR, and long-tail keyword research.
- AI models rely on Retrieval-Augmented Generation (RAG), meaning they retrieve information from search engine APIs (Google and Bing) to "ground the truth". Therefore, "SEO has never been more important".
- AI search does introduce new areas of focus; these "new" tactics are primarily evolutions of E-E-A-T (Experience, Expertise, Authoritativeness, and Trustworthiness), focusing on new metrics (like branded impressions), proactively answering branded questions, managing reputation on platforms like Reddit and YouTube, and adapting to personalization and agentic commerce.
I am excited to share a virtual recording of my talk from #MozCon last week titled:
GEO, AEO, LLMO: Separating Fact from Fiction & How to Win in AI Search
Check it out!
youtube.com/watch?v=2nJkT8zO…
The diagnosis misses the point. We're conflating two entirely different things--learning and education--and expecting one tool to serve both.
Learning versus Education
Learning is spontaneous, self-directed cognitive work.
Education is institutional, top-down knowledge transmission.
AI can be catastrophic for education (think essay-writing shortcuts, "vibe coding") while being transformative for learning. The failure mode only happens when students use AI to avoid the institutional demands of education, not when learners use AI to augment their genuine cognitive work.
The aristocratic tutoring example proves this. Ada Lovelace didn't have Mary Somerville writing her essays for her. She had Somerville creating "useful friction" that forces genuine cognitive exertion. The student using ChatGPT to complete an assignment is simulating compliance while bypassing learning.
The Metacognitive Trap
The real danger is the illusion of understanding. It's most obvious in math--reading a proof feels like comprehension until you try to reconstruct it and find you can't.
AI accelerates this trap because it delivers outputs (polished prose, working code, coherent explanations) that look like the products of understanding, all without requiring the cognitive work that produces actual understanding.
But again, this critique only applies when AI replaces the work. The solution isn't to avoid AI, but to redesign how we use it. We need to transform passive consumption into active learning.
-- Personalization: "Rewrite this section on neural networks using analogies from basketball strategy." (This forces you to verify if you actually understand the analogy).
-- Socratic dialogue: "Convert this dense paragraph into a dialogue between a skeptical student and a patient professor." (This externalizes the internal questioning that learning requires).
-- Self-assessment: "Generate multiple-choice questions testing deep understanding of this concept, not fact recall. Explain why the wrong answers are plausible but incorrect." Use AI to generate tests that break the illusion of understanding.
The Testing Problem
The point about testing connects right here. The recent UCSD finding (that math readiness declined after making tests optional) shows something fundamental: objective assessment isn't optional. It's the mechanism that converts the illusion of understanding into an awareness of ignorance.
AI's danger (metacognitive laziness) must be countered by AI's capability (frictionless test generation). A student who uses AI to write an essay learns nothing. A student who uses AI to generate progressively harder problem sets--and honestly assesses their performance--is replicating the aristocratic tutoring model.
The "one-size-fits-all" textbook problem shows AI's real potential. A standard textbook can't adapt to whether the reader is a basketball player learning neural networks or a musician learning stats. AI can. It scaffolds the cognitive work you must do.
The difference is architectural. Current AI tools (like ChatGPT's "Study Mode") fail because they're designed for frictionless assistance, not "useful friction." They're optimized for user satisfaction, rather than cognitive development.
The Structural Challenge
There's a deeper issue, too. Genuine learning requires interdisciplinary synthesis, but higher education forces specialization that real-world problems ignore (e.g., mental health isn't just psychology--it's biology, sociology, environmental science). AI's ability to make hard things accessible, especially outside your main domain, addresses a huge limitation of conventional education.
The student who wants to learn to code doesn't just need motivation--they need a system that converts effort into capability. That system requires:
- Personalized explanations that connect new concepts to existing knowledge.
- Socratic prompts that force active reasoning, not passive absorption.
- Rigorous self-testing that exposes the illusion of understanding.
- Immediate feedback on misconceptions.
AI can provide all of this, but only if we stop using it to avoid cognitive work and start using it to structure that work.
The prediction that fewer than 1% will do this is probably accurate. But that's not an indictment of the technology--it's an indictment of how we're deploying it.
*the next installment in steve's spicy morning takes*
AI is terrible for education
This came as a shock to me. I've long-anticipated AI solving all problems of education in one fell swoop. We in education circles talk about Bloom's Two Sigma Problem, which says that if we can scale 1:1 human tutoring to every student, then every student gets As. Now we have ChatGPT – a personal tutor for every student! Education is solved!!!
This is why the world is so sane right now, and everyone you meet is so freaking smart. Everyone has become a super-genius by talking to their personal AI tutors.
I wish. Instead it seems like, if anything, we've all gotten dumber. How is this possible? How can this not make us smarter?
If you look at recent history, we've gone from a society of illiteracy to full literacy to public libraries to the internet, wikipedia, youtube, google scholar, audible, khan academy... The tools to become smarter abound and yet smartness does not. What is going wrong? Why don't more educational tools help?
Being excited about AI's impact on education in 2025 is much like being excited about MOOCs impact on education in 2012. Less than 1% of people will use these to actually become smarter.
With MOOCs, people just ignored them, but in the case of AI it's actually worse than neutral.
We all know that in order to learn a physical skill you have to get up and do that skill physically. You can't learn surfing without getting wet. You can't learn pottery without putting your hands in the clay. You learn tennis on the court.
Cognitive skills are no different. You learn them through active mental exertion. You learn them by actually trying things - venturing - making mistakes, and learning from those mistakes. We call this 'fuck around find out' method "play" and it central to learning any new skill.
You absolutely cannot learn any skill by ingesting material on it passively.
YOU ABSOLUTELY CANNOT LEARN ANY SKILL BY INGESTING MATERIAL ON IT PASSIVELY.
You can learn history that way. You can be entertained by stories that way. But you cannot learn HOW TO DO SOMETHING that way. We all know this for physical skills - imagine someone learning to dance by sitting on the couch watching people dance - but we seem to forget it for mental ones. They are no different.
So why does AI make this story worse?
Because it gives you the illusion of learning while at the same time isolating you from its source.
Vibe coding is the prime example of this. Vibe coding is the equivalent of playing Guitar Hero, thinking that you're learning to play the guitar.
In order to learn to code, you have to learn some incredibly difficult mental skills. Just imagine learning the basic skills of a sport. I'm picturing basketball. The motions are weird and counter-intuitive and you've gotta practice them a lot before they are second nature. Learning to code is just the same.
Ok so now enter vibe coding. This would be the equivalent of playing a basketball video game. Sure, it looks like you're playing basketball, but you're on your couch, miles from a hoop, a controller in your hand, when you need a ball in it.
Ok now to address some counter-arguments.
1. "But I use AI and it makes me smarter!"
Sure whatever, you're also the sort of person who reads books and gets smarter too. You're what we like to call "resourceful", and AI is just another resource to you.
I'm talking about the people who resist resources. If you wanted to learn to code, there were are a million resources to teach you before AI, but you didn't learn then. Will AI help you learn now? I don't think so. You have to do uncomfortable things with your brain to learn. AI helps you avoid those things, which means you avoid the learning.
If you want to learn to code you have to get off the mental couch and get on the mental basketball court.
If you're looking for the basketball court for learning to code, you can find one at the corner of val street and town way
2. "Now that AI exists I don't have to learn to code. I just have to wait for the AI to get smart enough."
Yeah, that's what my brother thought about about learning to drive. Now he's 26 without a license and has to take ubers everywhere. The progress in AI intelligence is literally the hardest thing in the world to predict, and people have been getting it wrong for 100 years. You may be in for a long wait.
And more importantly, why do you want to avoid mental gymnastics? You are the humans in WALL-E who don't move their bodies. But you go to the gym, right? Just the same way you push your physical body, you should push your brain! Stop working so much on your biceps and start working on your cerebral cortex. Smart is the new sexy, haven't you heard?
3. "But wait, didn't you say that Bloom's Two Sigma Problem showed that 1:1 tutoring made humans smarter? Why can't AI replicate that?"
Finally, a good question! Took you long enough.
Yes, this is where hope lies. The dream is no less than what @erikphoel calls 'Aristocratic Tutoring' for all. Hoel's argument is that all the geniuses of the past had incredible personal tutors. From Alexander the Great to Einstein. But we lost that when we went to mass schooling.
But we can get it back with AI!
But clearly that hasn't happened yet. Why not?
Imagine a young Ada Lovelace, living on the estate of her dad Lord Byron, surrounded by the intellectuals of her day, tutored by Mary Somerville, the women for who the word "scientist" was coined to be a gender-neutral alternative to "man of science". Imagine young Ada being called to her daily intellectual lessons, walking through idyllic pastures, and getting to test her wits, new theories, and writings amongst her tutor and parents' friends, the intellectuals of the day
Now contrast this with a modern school child surrounded by entertainments, one of which is ChatGPT, which of course they use just to write their essays for them ("vibe writing", if you will), so they can go back to watching TikTok
In his new book Sal Khan talks about designing his AI to be Socratic, and like a human tutor, spend most of its time trying to uncover what the student doesn't know, so it can best address that, without simply giving away answers. It's not impossible, but that it's incredibly hard, and ChatGPT slapping a stupid system prompt on, and launching "Study Mode" is disrespectful to the scope of the problem.
The job of the human tutor is the same as that of a human coach of a sport. To get the kid on the court, practicing. This is why AI is currently so bad for education. It isn't trained to get kids to practice mental skills. It's trained to display mental skills.
Let's also not forget that the most important job of a human tutor/coach is the social/emotional scaffolding to actually get you to do the hard practice day in and day out. AI is woefully set up to be able to help with that.
If you are reading this essay – and I'm pretty sure you are – and you're non-technical, I hope this was a wake up call for you to get off your butt and start working your brain out.
Here's a secret about me: I am an absolute sucker for a good student. If you want to learn to code and you put in the work, I can not resist teaching/mentoring you. Literally nobody takes me up on this offer. Maybe you will.
