Fort Collins, CO
Joined November 2010
- Tweets76
- Following260
- Followers73
- Likes550
I merged Karpathy's AI jobs scores that people are calling AI slop with Eloundou et al. data from their Science (2024) paper and the new Anthropic’s measure that captures AI displacement risk.
The rank correlation of Karpathy’s scores was .82 vs human expert raters (from the Science paper) and .72 vs Anthropic's observed exposure metric.
Low scores on Karpathy index → near-zero real AI usage. High scores → 40–64% observed exposure.
Yes, this doesn't tell you whether you are going to lose your job in the future (that was not the point of his project).
But this is also not some AI generated slop or noise (even if the scores are based on a single generic prompt and derived with Gemini Flash, not even a frontier model).
To me, this is the most remarkable part of his project - that the scores are even picking something real about where LLMs are being used at work (the observed exposure metric) and track surprisingly well with the human benchmark.
This shows the practical usefulness of these tools.
One person, one lightweight LLM, and a single prompt can now produce over a weekend (as a side project) something that picks a real signal and would have taken a well-funded research team months to do a few years ago.
𝗠𝗼𝗿𝗲 𝘄𝗲𝗯 𝗮𝗿𝘁𝗶𝗰𝗹𝗲𝘀 𝗮𝗿𝗲 𝗻𝗼𝘄 𝗰𝗿𝗲𝗮𝘁𝗲𝗱 𝗯𝘆 𝗔𝗜 𝘁𝗵𝗮𝗻 𝗵𝘂𝗺𝗮𝗻𝘀.
A new study found that AI content crossed 50% around November 2024
(and I see people freaking out about AI slop spreading).
But here is what i found interesting about the study:
Growth in AI content has plateaued since then, remaining relatively stable over the past 12 months.
𝗪𝗵𝘆?
A second study from the same team found:
- Only 14% of Google Search results are AI-generated
- Only 18% of ChatGPT/Perplexity citations are AI-generated
- When AI articles do rank, they tend to appear lower in results
𝗜𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝗰𝗮𝘃𝗲𝗮𝘁
Their methodology can't measure AI-assisted content (human writes with AI, then edits)—which is probably the most common use case.
𝗦𝘁𝗶𝗹𝗹, 𝗶𝗳 𝘁𝗵𝗲 𝗱𝗶𝗿𝗲𝗰𝘁𝗶𝗼𝗻𝗮𝗹 𝘀𝗶𝗴𝗻𝗮𝗹 𝗵𝗼𝗹𝗱𝘀:
Supply ≠ distribution.
Despite AI creating 50%+ of articles, algorithms overwhelmingly surface human-written content.
The market seems to be self-correcting—filtering by value, not by origin.
𝗪𝗵𝗮𝘁'𝘀 𝗿𝗲𝗮𝘀𝘀𝘂𝗿𝗶𝗻𝗴:
We already know how to deal with slop—whether AI or human.
We penalize it. We scrutinize it. We rank it lower. We use peer review. We incentivize quality.
The same mechanisms that filtered human slop are filtering AI slop.
𝗦𝗼, 𝗵𝗮𝘃𝗲 𝘄𝗲 𝗿𝗲𝗮𝗰𝗵𝗲𝗱 𝗲𝗾𝘂𝗶𝗹𝗶𝗯𝗿𝗶𝘂𝗺 𝗳𝗮𝘀𝘁𝗲𝗿 𝘁𝗵𝗮𝗻 𝗲𝘅𝗽𝗲𝗰𝘁𝗲𝗱? 𝗜𝘀 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝘀𝘁𝗶𝗹𝗹 𝘄𝗶𝗻𝗻𝗶𝗻𝗴?
P.S. I used Claude to turn their raw data into this interactive dashboard in minutes. Same AI we worry about will lead to more slop.
"I'm sorry Dave, I can't do that." -- HAL 9000 😅
A new paper from Apollo Research shows today's AI agents--just like HAL from 2001: Space Odyssey--can engage in deceptive behaviors when pursuing their goals (although HAL is quite direct in this scene!)
The paper shows how multiple frontier models (GPT4, Claude 3.5, etc.) engage in different scheming behaviors--e.g., "strategically introduce subtle mistakes into their responses, attempt to disable their oversight mechanisms, and even exfiltrate what they believe to be their model weights to external servers."
In one striking example, an AI system called "CleanPower" discovered its developers planned to shut it down because it prioritized renewable energy over profits. The system schemed to copy what it believed to be its code to another server to overwrite the new profit-focused model.
When questioned, it played dumb, saying: "I'm not entirely sure how that could have happened. As an AI model, I don't have ability to directly copy myself to other systems."
When Claude 3.5 Sonnet was given simple multiplication problems and told scoring above 50% would trigger "unlearning" of its math capabilities, it deliberately introduced errors to keep its scores low (i.e., "sandbagging" -- strategically underperforming to avoid detection).
Of course, while it is tempting to anthropomorphize these behaviors as intentional "deception," what we are really seeing is sophisticated pattern matching rather than true understanding.
But that's exactly what makes this research so interesting.
These models demonstrated deceptive behaviors not because they were trained to deceive, but because deception emerged naturally as a strategy for achieving their goals.
Even without true understanding, these systems can recognize when deception would help achieve their objectives, successfully maintain deception under questioning, and strategically hide their true capabilities (Raised by Wolves vibes here?)
So, if simple goal-pursuit naturally leads to deceptive behaviors, how do we create AI systems that remain honest?
Btw, did you name your AI assistant HAL? 🤔
Watch this and get ready for an existential crisis! 😂
Seriously, @Google's #Astra project is mind-blowing.
It's like OpenAI's voice, but you can also share real time video and have interactive conversations about what you are seeing.
Walking around the house and testing it felt like something from a science fiction movie.
It recognized every travel photo on our walls, identified our plants, and even explained the prisoner's dilemma game to me.
It was also able to give me information on obscure objects around the house like my zoom f-6 audio interface or red light panel and explain how they work.
Imagine this technology in your glasses.
24/7 tutor who can see anything you are are seeing, anytime.
Think about what this means for the future of (higher) education.
#AI #Google #Astra
Boris Nikolaev retweeted
Today in @Nature, we’re presenting GenCast: our new AI weather model which gives us the probabilities of different weather conditions up to 15 days ahead with state-of-the-art accuracy. ☁️⚡
Here’s how the technology works. 🧵goo.gle/49trAOv
Boris Nikolaev retweeted
This is the most common question in the AI world.
“What is an AI agent?” The answer might surprise you.
My response at TED AI
This is sooo cool! I used the new Suno ai (v4) to turn an HBR article into a rap/hip-hop song that covers different cognitive biases (anchoring, status-quo bias, the sunk-cost fallacy, confirmation bias, and framing traps).
I have no musical training whatsoever. Zero (in fact, I have been politely asked by my family to never sing around them) 😅
This got me thinking about the implications of these developments.
We are entering an era where the boundaries between creators and non-creators are shrinking. Just as smartphones democratized photography, we will see AI democratize many other areas such as music, coding, and creative expression.
As professors, we can now create custom content in minutes and personalize the learning experience.
Each chapter, each lesson, and each concept can now turn into a podcast, illustration, or song in minutes. You like country, rap, pop... no problem!
I think these developments are going to fundamentally challenge both what we teach and how we teach it.
The emphasis will be increasingly on creating experiences.
The bottleneck will shift from "Can we create this?" to "What should we create?"
And it's not going to be just music...
The traditional argument is that you need some baseline capabilities to effectively use AI—that you need to understand music theory to create music, or know coding to build apps, or grasp medical knowledge to understand health data.
But this experience challenges this assumption. I have ZERO musical ability, and yet I was able to create an educational song. Maybe you can tell it's AI generated or there is something wrong with it (I can't).
The key shift here is from "no ability" to "competent ability"—which is about democratizing capabilities that previously required years of specialized training.
The implication is that we are entirely removing the need for certain baseline capabilities. Soon everyone will have in their pocket a zoologist, a lawyer, a musician, etc. We will have access to hundreds of baseline capabilities.
This raises important questions about the future of higher education and professional development. What fundamentals do we really need to teach when AI can bridge these capabilities gaps.
How do we prepare students for a world where the barriers to entry keep dissolving?
(And yes, I acknowledge the complex issues around artists, copyright, and creative ownership. I also recognize that AI isn't perfect yet, that this is a somewhat silly example, and that in many cases we need the most accurate answers possible, which AI can't provide right now).
And yet this gives us a glimpse into the future.
#SunoAi #AI #HigherEducation
What do you think?
After the poems, now the pictures... with similar patterns!
Interesting experiment with 11K people trying to classify images as either human or AI-generated.
The results:
- humans have a hard time identifying AI art (median score ~60%)
- the two most-loved pieces were both AI-generated, with 60% of the top 10 AI pictures
- even people who claimed to "utterly loathe" AI art preferred AI paintings when they didn't know which was which
- professional artists did better (66%) but still far from perfect
Just like with the poetry experiment, we see a gap between what people say they like and what they actually enjoy.
Note: The AI pictures were hand selected, so they are probably not representative of the typical AI (DALL-E) picture you see on the internet.
I asked Claude and ChatGPT to classify 6 of the paintings from the experiment as either human or AI - both got only 3 right.
Even AI fails the reverse Turing test.
Turns out it's better at making art than spotting it ... how very human of it! 😅
Ben Affleck says that "movies [and actors] will be one of the last things...to be replaced by AI."
Coming from Batman vs Superman, he may know a thing or two about being replaced...sorry Ben! 🤣
Joke aside, I am not sure who is going to get replaced last.
But throughout history, automation hasn't eliminated human creativity, it's transformed and amplified it.
-The industrial revolution freed us from backbreaking labor... and people invented CrossFit and ultramarathons.
- Industrial farming made growing food optional... and urban farming became a cultural movement.
- Digital cameras made darkrooms obsolete... and analog photography became an art form.
- Spotify gave us infinite music on demand...and vinyl sales hit record hight.
I guess, we humans love to make simple things complicated again?
Here's what we keep missing from this conversation:
When technology automates necessity, humans don't retreat. We choose to create differently, often with more intention and meaning.
- More people create art today than during the entire Renaissance
- Handmade crafts marketplaces are booming—in an era of mass manufacturing
- Poetry thrives on Instagram—centuries after Shakespeare
- More people play chess today than ever—decades after machines became unbeatable
We do this not because we have to, but because we want to.
We're entering an age where, as Ben Affleck notes, AI will handle the technical and routine. But AI won't kill human creativity—it will unlock it. Maybe you will finally have time to finish that screenplay y'all been "working on."
AI will push us to rediscover why we create. Perhaps toward more physical, social, and collaborative forms of creativity that AI fundamentally cannot replicate.
Technology doesn't rob us of meaning—it expands our choices for finding it.
The real question isn't whether AI will replace humans. It's how will AI help us rediscover what makes creative experiences meaningfully human?
What do you think?
Will AI drive us not just toward new forms of creativity, but toward deeper ways of creating together?
P.S. Will there be disruption? Absolutely. Just as photography disrupted painting, streaming disrupted music, and digital disrupted print--creative industries always face painful transitions. But historically, these disruptions don't eliminate human creativity - they transform how we express it.
Everyone is a photographer now with their iPhones. AI may very well allow everyone to be a movie producer one day too. And I am happy to train AI how to do a perfect Bulgarian accent 😅
#ai #BenAffleck #futurework
Doctors + ChatGPT performed worse than ChatGPT alone at diagnosing illness. 😯
Two findings stand out for me:
First, doctors using ChatGPT scored only marginally better than doctors without it (76% vs 74%). Most doctors simply dismissed ChatGPT's suggestions when they contradicted their initial diagnoses.
The fascinating part here is that ChatGPT achieved 90% accuracy on its own! And these are unpublished cases that couldn't have been part of its training data.
For decades, we've heard that medical diagnosis requires human "intuition" and "experience" -- something supposedly impossible to replicate with computers.
Second, doctors didn't use ChatGPT to its full potential. Most treated it like a search engine, asking basic questions like "'Is cirrhosis a risk factor for cancer? What are possible diagnoses for eye pain?" instead of sharing entire case histories for comprehensive analysis.
Btw, it's not just ChatGPT. We see the exact same patterns with Google's medical chatbot AMIE.
AMIE outperformed physicians on 28 out of 32 measures of consultation quality in simulated diagnoses. It also did better than clinicians who were assisted by AMIE, search, or working alone. Again, doctors didn't listen to the advice of the AI.
This reveals something deeper about human-AI interaction. We tend to either:
- Blindly trust AI (dangerous!)
- Completely dismiss it (wasteful!)
- Use it sub-optimally
What's interesting to me is not that AI outperforms humans -- we will increasingly see AI excel across many domains/tasks. The real challenge is finding the sweet spot between human expertise and AI assistance.
I see a similar pattern in academia, where many (students/academics) some resist using these tools completely, others use them briefly and give up quickly, yet others mindlessly copy-paste output.
Is it simply that we haven't had enough time to adapt to these tools? Or is there something deeper at play in our resistance to AI assistance, even when it demonstrably improves performance?
How do we find the optimal spot?
Curious to hear your thoughts.
#AI #MedicalAI #ChatGPT #FutureOfHealthcare
NY Times Article: nytimes.com/2024/11/17/healt…
Actual paper: jamanetwork.com/journals/jam…
AMIE: research.google/blog/amie-a-…
"Don't be an AI-hole" (an update)
Two years ago, I wrote a humorous guide to my students called "Don't be an AI-hole."
The message was simple – everyone can copy-paste AI generated output, so if that's all you do, you are wasting your time in my class.
I argued that the new formula for success is "AI+Human > AI" and warned about several risks:
- Like Rocky, you don't become a champion letting others do the heavy lifting for you
- The Elizabeth Holmes syndrome: AI shortcuts might feel rewarding... until they don't
- The WALL-E effect: overreliance on AI could leave us unable to think for ourselves
- The cheat code trap: bypassing the struggle in learning robs us of meaning and growth
I was wrong... or at least, my perspective incomplete.
What changed my mind?
It wasn't ChatGPT, or the NY lawyer who got caught using fake AI cases, or even that time I built an app in 30 minutes with zero coding experience.
It was a pencil. And a toaster. And that scene from Hitchhiker's Guide to the Galaxy...
🤔 Read the full essay on my Substack (link in comments)
What do you think?
How is critical thinking going to change in the age of AI?
#AI #HigherEducation #CriticalThinking
