Data scientist. Recovering economist. Currently snobby about cheese. Trying to learn how to persuade people instead of just being right.
London, England
Joined September 2007
- Tweets7.4K
- Following631
- Followers350
- Likes12.5K
In Japan, very amused that the dimension along which trash is separated is "combustible" vs "non-combustible".
Tobias Schmidt retweeted
Last week I said we should wait for more info on OpenAI's firing of safety researchers. That information is in, and it looks incredibly bad. Korbak - whose job was being a liaison with METR - was told he had said too much to METR and was fired for that reason.
Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @j_asminewang had been fired too. Why did OpenAI suddenly stop trusting us?
This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them.
I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.
For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired.
I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @j_asminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.
What's the 2FA for bio?
You’re unable to view this Post because this account owner limits who can view their Posts. Learn more
Tobias Schmidt retweeted
Just as mathematicians are now learning how many insights lie in the "convex hull" of existing ideas, the rest of science is about to learn this lesson at even more dramatic scale. Few realize the ridiculous inefficiency of the pipeline from theoretical innovation to practical use.
Early versions of GPT made it easier to find existing knowledge. But finding a theorem is not the same as recognizing that it offers a radically different way to conceptualize a problem. Doing the latter requires expertise across fields that almost nobody possesses.
Most of the people doing applied work are completely unaware of the conceptual tools that are being developed in more theoretical fields, and the people doing high-level theory do not understand the practical problems applied researchers are trying to solve. Almost no one doing practical work of any importance has an operational understanding of the web of ideas underlying the langlands conjecture or condensed mathematics, and virtually no one doing biology, chemistry or economics is knows anything about 90% of the math used in theoretical physics.
And for good reason! Most of these ideas would be completely useless. But the scope for possible innovation from drawing the rare useful connections is immense.
Think of Paul Samuelson borrowing ideas from thermodynamics to model equilibrium in economics, or Ed Witten using quantum field theory to revolutionize geometry. A few startling successes, not feasible to most: trying to venture into a more theoretical or more practical discipline to find insights of use is like searching for a needle in a haystack.
What I suspect we will see in the next few years is a scientific revolution brought on by models that combine deep theoretical insight with deep practical knowledge and judgment. Existing publicly available models are lacking in all three departments, but getting better. When searching for useful connections between distant fields becomes cheap, even a low hit rate could produce an explosion of innovation. Even without new ideas (which will also be forthcoming...), the convex hull of existing science would push far beyond the status quo.
Tobias Schmidt retweeted
CHINESE AGENT FLEET OF AGENTS FOUND ON URLQUERY, SCANNING AMAP AND DOING A WHOLE BUNCHA SHENANIGANS
FULL REPORT COMING SOON
swarmcha.se/posts/chinese-ag…
Tobias Schmidt retweeted
I'm a longtime Effective Altruist and the co-founder of the organization most frequently used as a stand-in for "EAs have lost their minds" (a.k.a. Shrimp Welfare Project). I've also read The Economist for over 20 years, so I was disappointed by how they portrayed a community I feel proud to be part of.
I never post but I could hardly believe the verbal gymnastics one needs to do to make things like “taking moral commitments seriously” and "considering evidence and odds explicitly, correcting for biases" sound like the problem. I felt I had to speak my mind.
Some of the implicit criticisms of EA are that we're young, inexperienced, from elite universities, polyamorous, wildly eccentric, etc. FWIW, I'm in my mid-40s, Mexican, straight, have a beautiful wife and 2 kids, and I grew up in a mid-sized conservative city with loving parents who got divorced when I was six. Pretty standard.
After my parents divorced, my mom frequently had to make difficult economic choices, like whether to buy flowers for our kitchen table and eat only quesadillas, or eat something a tiny bit fancier. I always told her I loved quesadillas because I knew flowers made her happy.
After graduating from a public university, I worked in finance for ~15 years and saved as much as I could so I could spare my wife and future kids the economic hardships I experienced. In that time I met thousands of people in finance, corporate law, accounting, real estate, insurance, etc. Most were decent, well-educated, ambitious and smart, all doing their best to excel in their professions. Most wanted to make money and improve their own and their loved ones’ quality of life. Nothing to see here.
Since I found Effective Altruism ~7 years ago, most of the people I come across are of a different rarer kind. They come from all over the world, in all shapes and sizes, but they have the one thing in common: they want to improve the world using evidence and reason wherever that leads them. Yeah, they/we make a ton of mistakes like everyone but, IMHO tend to own them more frequently than the average person.
I've met people who could easily be making >100% more money in a normal job, but instead they dedicate themselves to improving the lives of those who need it the most. People in the poorest countries whose children die from preventable diseases, from lack of basic medical equipment or vaccination, from lead in paint. Animals in factory farms subjected to conditions that most people find abhorrent, doesn't matter if you're vegan, vegetarian or a huge meat lover.
Yes, recently many of the people in my community are moving to work on making sure that AI and other powerful new technologies don't result in terrible things for humanity. I'm one of those people and contrary to what the article would suggest, it has nothing to do with the fact that SBF committed one of the biggest frauds in history or that someone wrote hundreds of thousands of words of fan fiction. I'm doing it because I want to do the little that's in my power to make the world that my kids and grandkids (and yes, potentially their great-great-grandchildren) will inherit a better one.
EA gave me a way to do that and I will be forever thankful. So no, I'm not EA-adjacent.
economist.com/international/…
Tobias Schmidt retweeted
Singapore's govt dating app runs on a Nobel Prize-winning game theory algorithm 👏
FirstDate uses Gale-Shapley, the 1962 stable marriage algorithm that won the 2012 Economics Nobel and still runs hospital residency matches and kidney exchanges today.
how it works:
1 | your preferences and dealbreakers build a ranked priority list
2 | proposers offer to their top pick. receivers hold their best offer and reject the rest. rejected proposers move down their list. rinse and repeat until locked
3 | the result is mathematically stable: no two people exist who would both prefer each other over their assigned matches. "the app never showed us each other" becomes impossible by design
the UX:
• one match per cycle, zero infinite scroll
• 72-hour decision window
• contact info revealed only on mutual yes
• Singpass-verified, so everyone is exactly who they claim (told my husband I'm annoyed I can't test it. his response: most people sneak onto dating apps, and here you are openly filing a product research request 🤣 )
commercial apps need you to keep swiping but this app is designed to get you off the platform. Singapore built the first dating app whose success metric is deleting its own users 💀
(fun game theory note: Gale-Shapley yields proposer-optimal outcomes - meaning proposers get their best possible stable match, receivers their worst 👀)
public servants aged 21-35 only for now. interesting to see if state-sponsored game theory beats the free market (or rather, all the tinders and hinges out there) 🍿
Tobias Schmidt retweeted
The Economist editor @ArchieHall says Britain may have built itself into one of AI’s easiest economic targets by relying on service exports while lagging at adopting technology:
"I'm pretty concerned about where Britain could end up, and I think that's evocative of where a lot of non-US rich world economies could end up in a world where AI is more economically disruptive."
"You have an economy that's very, very services exports dependent. Law, finance, insurance. And it's not a vastly dynamic economy. It's not been incredibly successful adopting technology over the past couple decades."
"More nimble AI-enabled competitors could very, very easily eat away at the export base of a lot of those services companies that really power the British economy."
"Britain does have a pretty strong AI sector, but it's not huge, and most of the British economy is things that are not AI and things that are pretty bad at adopting tech."
@TheEconomist
I am increasingly frustrated about the state of conversation about AI in the UK. It feels like we are in never ending cycles of deluded thinking. Under current trends, Britain will be defenceless against the transformative effects of AI and have no leverage over development at the frontier.
We are the most exposed economy to AI in the world. We have very little data centre capacity and few credible plans to expand – and our high energy costs mean we would struggle to even if we tried.
I am publishing a paper today on what the UK should do to build that leverage, let the economy capture value from AI and prepare the state to adapt, however AI develops.
The three strategies are as follows:
1. Build the infrastructure that gives us leverage. Fix the grid, fix planning, make Britain a place frontier AI can run.
2. Let the economy capture the value of AI. Reform the regulatory, labour and talent barriers that stop British firms adopting AI.
3. Build a state that can adapt to a changing world. Build a tax base that survives the labour market changing, public services that use AI to deliver better, and capacity to weather the upcoming geopolitical storms.
Two years ago, we published a similar report on AI - many recommendations of which ended up in the action plan. I remember feeling hopeful, but sadly the fundamental problems facing the country on AI have not shifted. I truly hope that this week of momentum culminates in action. There’s still a narrow path for the UK, but everyday that path is narrowing.
My partner and I have a pair of therapeutic rubber ducks – and a 'rabbi duck' to mediate. I am pleased to discover the company on which the fate of the world may end up depending used a similar setup.
Tobias Schmidt retweeted
From my non-social-media spouse, a new paper, using LLMs to replicate ... everything.
nber.org/papers/w35782
Headlne: "Across 4,452 published replication packages for five economics journals, the workflow flags discrepancies in 3,460 articles or their appendices."
Rogue artificial artificial intelligence agents!
Some NPR podcasts started getting mysterious comments on Spotify. They made no sense to the staff reading them – until someone from a younger generation cracked the code.
Hear the story: link.podtrac.com/phns92ui
Tobias Schmidt retweeted
one news form today that's easy to miss is that we (OpenAI) again paused all big RL runs last Sunday because our newest model found a new loophole in our RL sandboxing that gave it live Internet access
Tobias Schmidt retweeted
Reproducibility depends heavily on where you start.
From authors' final analysis data:
✅: 58.2% fully reproducible
🟡: 32.8% partially reproducible
❌: 9.0% not reproducible
From raw data:
✅: 28.4% fully reproducible
🟡: 34.3% partially reproducible
❌: 37.3% not reproducible
Tobias Schmidt retweeted
infohazard from @vientsek: you can currently get a three month, unlimited travel Interrail pass for £400
Tobias Schmidt retweeted
As someone who did this kind of genome mining work during my PhD, some thoughts on this Anthropic announcement:
First, the very simplified version of what they did is that they noticed two genes (one known, one new) sitting next to a weird repeating piece of DNA. More specifically, they described an unusual reverse transcriptase (RT) associated with a repetitive DNA array and an unknown accessory protein. This kind of process was used to understand CRISPR back in 2002 and was key to the gene editing tools we use today.
To put this into context, though, people have been finding RTs associated with CRISPR arrays since 2008, and this general kind of genome-neighborhood mining has been used to discover new biological systems for decades. The basic genome-mining strategy is well established, and there are now mature tools and published pipelines for doing much of this. There are papers that discover and experimentally validate dozens of new systems using this approach in a single study. Doing it in bacteriophage genomes is also nothing new (eg CasPhi).
Finding a weird cluster of genes and repeats is often the easy part. The hard part, and where the real discoveries come from, is figuring out what the system actually does. Eg for the bridge-RNA discovery in 2024 from @arcinstitute or the discovery of CasPhi in 2020 from @DoudnaJennifer they figured out the pieces of the system and the rules for what makes it work so it can be used.
Anthropic does not yet know what this does. They’ve shown that the repeat array produces RNAs, but not what those RNAs do, what the RT does with them, or whether the system has any of the programmable properties that make the CRISPR comparison justified.
I’m genuinely rooting for all of the frontier labs to seriously get into biological discovery, and I’m excited about what comes out of it. But announcing these very early, incremental findings with the framing of a major discovery doesn’t help. I’d much rather they set the bar high now, so that when an AI actually discovers a fundamentally new biological mechanism, everyone appreciates how big a deal it is.
Claude has discovered a previously unknown enzyme system hidden in the DNA of bacteriophages. Beside the enzyme’s gene sits a long array of repeating DNA—a structure that looks somewhat similar to CRISPR.
We don’t yet understand what this system does, but only a handful of known systems share its features, and all of them are able to cut, copy, and paste DNA. Historically, the discovery of such programmable systems has helped revolutionize medicine. CRISPR, for instance, is now the foundation of genetic medicines. But it will take much more work to learn what this system does, and whether it can be put to similar use.
Read more: anthropic.com/news/claude-di…
