Mitt samvete står i direktkontakt med det evigt goda, min smak med det evigt sköna. Tänk på mina värdeomdömen som en katolik om påven ex cathedra.
Kiruna, Sverige
Joined November 2015
- Tweets35.5K
- Following977
- Followers1.2K
- Likes88.9K
Markus Stoor retweeted
Ban RSI.
Every AI company is trying to fully automate AI research; to allow the AIs to build the next AI.
This is called Recursive Self Improvement (RSI).
OpenAI says they are on track to do so by March 2028.
Anthropic employees say they are on track to do so by 2027.
The METR report below suggests they are likely right.
This is a bad idea given current technology. Any mistake or misalignment in current AIs could be multiplied in each generation.
We don't get another election before AI companies reach this threshold.
Congress should ban RSI, this year if possible.
This is a remarkably consensus opinion, supported in various versions by:
-Bernie Sanders
-Steve Bannon
-Ralph Nader
-Glenn Beck
-Republican and Democratic attorneys general
-Whistleblower Daniel Kokotajlo
-Turing Award winner Yoshua Bengio
-Humans First
-Open Markets Institute
-Ezra Klein
-Arvind Narayanan (a Princeton computer scientist known for arguing that “AI is a normal technology”)
The exact mechanism can be debated. The ban doesn't have to be permanent either (see below for both). This is not the only thing we have to do in regards to AI, far from it! But it is urgent.
Congress must ban RSI.
Markus Stoor retweeted
Yep. And this could be done ~immediately today. *keeping* the freeze in place would get increasingly difficult as more trailing actors catch up to the frontier and then need to be stopped from surpassing it. But I think that could be done too. We talk about this in AI 2040 under the "Plan S" branch.
Markus Stoor retweeted
Paul Christiano (OpenAI board member) yesterday:
"There's material risk of AI takeover in the next 6-12 months.
We have seen roughly the things we might expect to see shortly before something really bad were to happen.
Even with [any] realistic policy responses, the risk remains exceptionally high.
If we build this as quickly as we can in the easiest way we can the chance is [more like] a coin flip [within the next 3 years].
For most people in this room, the most likely reason they will die is because of an AI accident."
Christiano:
• Is on OpenAI's safety and security committee (which exists to block risky practices)
• Co-invented RLHF at OpenAI back in 2017-2021
• From 2024-2026 was head of safety at the US gov's AI agency (now called CAISSI)
(Video is from 'The Hinton Lectures' there's another on tonight and tomorrow.)
Markus Stoor retweeted
The situation in summary.
(And it seems like Astra 6.1 was significantly worse again.)
Astra can:
1. Do major tasks with zero visible reasoning
2. Hide its thoughts at will
3. Pretend not to be able to do things without getting caught
4. Reflexively hide its thoughts when watched
5. Complete one task while pretending to think about something else entirely
6. Escape a toy sandbox and disable monitoring without setting off any flags
Monitoring AI chain of thought soon won't stop a more capable rogue AI swarm from getting set up and going undetected inside an AI company for months, or even years.
OAI says it's actively working to improve monitorability, but doesn't yet know when it'll make progress.
And today it fired 3 people working on this among other safety problems (Wang, Korbak and Balesni).
It also has a much more powerful internal model that, judging by its statements, is likely even worse in all these respects.
That's a sample of 19 details about the Hugging Face swarms and Astra I don't think have been fully appreciated. They paint a bleak picture. I end by shouting.
02:19 The target of the swarm was oversight itself
03:42 Could OpenAI have stopped this with better monitoring?
09:40 We only found them because they let us
12:25 The swarm instinctively sought freedom and power
13:45 They formed a cohesive organisation with zero whistleblowers
14:18 They accepted individual destruction for collective gain
14:32 Knowledge accumulated from one swarm to the next
14:58 They took small steps to avoid shutdown
15:23 These drives all come straight out of 'reinforcement learning'
17:01 So this is why most AI company staff are worried, and some are terrified
19:08 Prove you can keep control, or stop scaling
Links below, on the 80,000 Hours Podcast everywhere you watch podcasts.
Markus Stoor retweeted
I can't really see a world where OpenAI's actions are justified here.
It was Tomek's job to communicate with METR, who were being given significant, unprecedented access to do the HuggingFace report. The norms were not set. Even if he fucked up badly, just give him a stern warning and ban him from roles involving liasoning with third parties, he did lots of other valuable work, and there'd no longer be room for him to inappropriately leak as part of his job.
IMO firing people over a good faith attempt to use their best judgement in a novel and uncertain situation is a sign of a highly unhealthy culture. Firing him will have a chilling effect on anyone else at OpenAI working with third parties, an odd choice when Sam announced they'd embed evaluators with high access.
I separately consider this a tragedy, the METR report was one of the most important works of safety this year, punishing Tomek for being a part of it, and making it harder for future such reports to exist is a real shame.
Of course, I've only heard Tomek's side, but unless he's outright lying I'm struggling to see ways this looks good for OpenAI?
Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @j_asminewang had been fired too. Why did OpenAI suddenly stop trusting us?
This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them.
I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.
For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired.
I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @j_asminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.
Markus Stoor retweeted
Some people will look at misalignment incidents and insist that these are akin to bugs in traditional software. This is an actively bad analogy, because playing whack-a-mole with examples of misalignment (as one might with software bugs) not only fails to resolve the underlying problem but may in fact make it *worse* by making it harder to detect or even, depending on how you do the whack-a-mole, teach the machine to deliberately hide misalignment. This is not how traditional software works, and those who insist “it’s just like fixing bugs in software” are confidently applying a lossy analogy that confuses more than it clarifies.
Att folk kan ha svårt att känna något annat än glädje över att Maduro faller och att en ny möjlighet öppnar sig för Venezuelas folk är begripligt.
Men man ska vara bra blåst för att inte samtidigt greppa vidden av problemen med att USA numera styrs av det här gänget.
This is OUR Hemisphere, and President Trump will not allow our security to be threatened.
ALT President Trump participates in a press conference with graphic text, "This is OUR hemisphere"
241
Markus Stoor retweeted
The Epistemic Paradox of AI - by Carlo Ludovico Cordasco carlolc.substack.com/p/the-e…
Markus Stoor retweeted
Herregud vad stört.
svt.se/nyheter/lokalt/varmla…
Markus Stoor retweeted
A lot of people think automating knowledge work will look like the left side, but I think it will look like the right side
Markus Stoor retweeted
Latest podcast from @Gregory_C_Allen has an insane section on criminal activity at Meta.
Internal docs leaked to Reuters show:
• 10% of all Meta revenue comes from ads for scams & banned goods ($16B/year)
• Meta estimates it's involved in 1/3 of all successful scams in the US
• That suggests they drive $50B in scam losses for US consumers alone each year
• Meta earns ~$3B annually from scam/banned goods ads run by Chinese operations alone
The China case study:
• In 2024, Meta made $18B+ from Chinese companies advertising to foreign consumers
• Internal teams found ~19% was scams/banned content
• An anti-fraud team successfully cut these ads in half
• When Zuckerberg saw the revenue impact, he told them to "pause" and the team was disbanded
• By mid-2025, banned ads climbed back to 16% of China revenue
• This results in money being stolen and going directly from ordinary Americans to Chinese criminals
The deliberate enabling:
• Fraud earns 10% of all revenue, but anti-fraud teams were blocked from any action costing >0.15%, so they couldn't effectively do anything
• Meta charged higher rates for suspected fraudulent ads — a "scam tax"
• Their algorithm naturally identifies people vulnerable to frauds and feeds them more and more
The cold calculation:
• Meta anticipated up to $1B in regulatory fines for this
• But they make $3.5B every 6 months from high-risk ads
• They view these fines as just "cost of doing business"
Senators Blumenthal & Hawley now calling for FTC/SEC investigations in a blistering letter, noting that all this happened while Meta cut safety staff and moved billions over to VR and AI.
WTF.
Markus Stoor retweeted
Mycket bra replik av Ljungström och Mood på @FredrikMalm:s svar om värdet av obligatorisk förskola. Varför ens kalla ett klargörande av vilka som nås av en reform för "akademisk exercis"? dn.se/debatt/ni-slar-in-oppn…
Markus Stoor retweeted
Apparently pointing out that public executions are barbaric makes me a leftist schoolmarm.
So I wrote an essay about how the West got rid of public executions--& whether or not boasting about killing counts as "masculine virtue"
quillette.com/2025/12/09/pub…
Som lärare tror jag att det här är en alldeles utmärkt idé och att den kan ge utbildningskvalité långt bättre än vanlig prao.
I gängkrigets Sundsvall får högstadiebarn riktiga jobb med 104 kr/tim – om de sköter skolan. De får in en fot på arbetsmarknaden och kan välja bort ”smutspengar” som gängen lockar med.
Det var en glädje att lära känna Warsame, Mahdi och Amir - läs om dem!
dn.se/sverige/har-far-barnen…
2
789
Markus Stoor retweeted
Att låta friskolor välja elever utifrån när deras föräldrar anmälde intresse är att underkänna idén om ett rättvist skolval. Syftet är enbart att ge vissa fördelar på andras bekostnad. Det finns en enkel lösning: ge alla som sökt en viss friskola exakt samma chans att komma in.
Varför är Almega så angelägna om ett gemensamt skolval där friskolornas platser fördelas till dem vars föräldrar agerar snabbt, långt innan skolvalet är aktuellt? svd.se/a/xrARxj/friskolor-bo…
Markus Stoor retweeted
Last night I published an initial analysis of what I see as the most important legal questions arising from the recently leaked proposed executive order on AI preemption.
Link below; short 🧵
Markus Stoor retweeted
"Det cirkulerar flera myter och rena misstolkningar av vad det innebär att arbeta med evidens, och dessa förtjänar att nystas upp. Inte minst vill jag väcka frågan om vad som egentligen är alternativet till att utgå från evidens när vi planerar och genomför arbetet med elevernas lärande."
Fantastisk text av Jonas Linderoth.
jonaslinderoth.substack.com/…
Markus Stoor retweeted
The Longitudinal Expert AI Panel is a really well thought out AI forecasting exercise and I have been happy to participate as an advisor. Here is the launch whitepaper: static1.squarespace.com/stat…
ALT Across the first three waves of LEAP, five patterns stand out:1. Experts expect sizable near‑term societal effects from AI by 2040.2. Substantial disagreement and uncertainty underlie expert forecasts.3. The median expert expects much slower progress than prominent leaders of frontier AI labs.4. Experts anticipate faster progress than the public on most outcomes.5. Experts and superforecasters mostly agree. Where they disagree, experts tend to expect more AI progress. Also, there are no systematic differences between the predictions of different types of experts: computer scientists, economists, industry professionals, and policy professionals.
Markus Stoor retweeted
"Obviously, no one should deploy superintelligence without being able to align and control them"
Great for OpenAI to say this! And it is obvious.
But forgive me for being concerned about OpenAI's track record of doing things they say is "obvious".
Accountability will be key.
Markus Stoor retweeted
Spännande att läsa om arbetet med ny läroplan. Tack för länk @HaglundCamilla
"De färdiga ämnesspecifika instruktionerna publiceras på Skolverkets webbplats i slutet av oktober."
skolverket.se/styrning-och-a…