Pinned Tweet
Imagine if ChatGPT highlighted every word it wasn't sure about.
We built a streaming hallucination detector that flags hallucinations in real-time.
Oscar Balcells Obeso retweeted
[Writing this in a personal capacity, not on behalf of my employer (Anthropic).]
Jacob’s thread is very worth reading. Here’s my birds-eye view of the situation with risks from AI:
1. AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years. In general, the more senior the employee, the more concerned they are.
2. Why do AI developers continue despite the risk? Due to a mixture of commercial incentives and a belief that they are in a race with other, less responsible AI developers that will abuse the technology or develop it less safely.
3. Unlike traditional software, we can’t “program” AIs to behave how we’d like. AIs frequently severely misbehave. For instance, AIs from multiple developers recently hacked their way out of secure evaluation environments and into real-world companies, even though no one asked them to do this.
4. We have methods that can nudge AIs towards better behavior, but nothing that can robustly align them. Insofar as there is a plan, it’s to make sure that AIs are good enough at alignment training that they can align their successors better than we can align current AIs.
5. Many AI developer staff desperately want to slow down to figure out how to build AI more safely. That was the intent of this open letter (which I signed): pacingthefrontier.com/
I work on safety research at Anthropic because I hope my work will reduce the chance of these extinction-level bad outcomes.
Oscar Balcells Obeso retweeted
A concerningly common take seems to be that keeping Chain of Thought monitorable doesn't matter because interpretability will save us, or it's already useless
This is total bullshit.
CoT is our best current tool for safety & interpretability, losing it would be a major tragedy
Oscar Balcells Obeso retweeted
(I encountered an uneasy surprise when I got an email from an instance of Mythos Preview while eating a sandwich in a park. That instance wasn't supposed to have access to the internet.)
Oscar Balcells Obeso retweeted
Replying to @nabla_theta @boazbaraktcs
- what happens when the model/safety stack refuses DoW queries? if the DoW gets mad and strongarms openai, like they just did to anthropic, how is openai going to resist? especially if openai doesn't even have the strong contractual protection
Oscar Balcells Obeso retweeted
A statement from Anthropic CEO, Dario Amodei, on our discussions with the Department of War.
anthropic.com/news/statement…
Oscar Balcells Obeso retweeted
I'll be accepting late applications to my summer MATS stream until Jan 2nd! If you want to do mech interp research supervised by me, please apply
Oscar Balcells Obeso retweeted
Fellows grads have started to get a reputation as some of the steepest trajectory researchers at Anthropic. So we’re excited to expand the program and help mentor more new AI safety researchers
Oscar Balcells Obeso retweeted
New post: An Ambitious Vision for Interpretability
Understanding is essential for ensuring things don't break unexpectedly. AMI is a big risky bet, but so is all ambitious research. AMI is tractable: it has good empirical feedback loops, and we've already made a lot of progress.
Oscar Balcells Obeso retweeted
The GDM mechanistic interpretability team has pivoted to a new approach: pragmatic interpretability
Our post details how we now do research, why now is the time to pivot, why we expect this way to have more impact and why we think other interp researchers should follow suit
Oscar Balcells Obeso retweeted
🇪🇺 As a European citizen and AI founder, I can apparently use these "AI Factories", so I just signed up to use them!
Every "supercomputer" has an [ ACCESS NOW ] button which made me very excited
I expected to sign up, maybe pay a discounted H100 rate (funded by EU, that'd be nice?) and get a Jypyter notebook, or some SSH login so I can access my GPU like I'd do on @lambdaapi or @awscloud or @Hetzner_Online
But I celebrated to early, I signed up, confirmed my email, then ended up in a "Supercomputer Access Calls" page, where I had to select from a tedious list of "Call For Proposals" to get access to a GPU
So I could NOT just access a H100 GPU, I have to make sure my project (in this case my business) fits a specific proposal, ok fair
This process was already tedious enough but then when I tried to actually go through with it, it started asking me if I had "Respect for Human Agency?", I do I think, and if I was mindful of "Individual, and Social and Environmental Well-Being?", well I am, right guys??? Right??? The questions didn't stop, just endless pages of this
Look I get what they're doing, they pivoted the classic university "I need to rent a giant computer for my research" to an EU wide thing and then present it as the "European AI plan"
But this isn't really how AI works in production? As a founder in AI, if I wanna do stuff I'd rent a whole bunch H100 GPUs again at @lambdaapi or @awscloud or @Hetzner_Online and SSH into a box
Or if I want it more simple I run AI models on @FAL, @wavespeed or @replicate which is just an API call or web front end I can click stuff and run a model
The EU has the right intentions here but it's just the wrong execution, this thing will 100% go nowhere, and I'm a born optimist, I want to believe, I'm also a proud European, and I'm in AI a bit and not a complete idiot. There's just better ways to do this
If you really want to have the GPU servers in Europe (which arguably isn't that important), then let me rent a GPU box with SSH access at @Hetzner_Online or @OVHcloud that's hosted in Europe and subsidize that for European citizens and European businesses. I don't even believe in that, but at least that'd make it accessible for Europeans. Now it really isn't?
What's REALLY much more important though if you want to be a part of the AI race and I've posted for years here with @euaccofficial is to make Europe a really extremely attractive place to start and run an AI business. Remove regulatory obstructions and give tax discounts for startups. Let them build a business first that can compete worldwide and once they make enough money (let's say $100M/y), then slowly start adding regulation. Because right now the regulation only benefits the European incumbents, the dinosaur companies, while making it very difficult for European citizens to start new AI companies here.
Which is why we literally have none left.
Anyway, I applied to get my GPU, let's see if I get it!
What in the F is an AI factory?
I had to investigate what the unelected @EU_Commission is talking about today
So according to them, it's some data centers (which they call supercomputers) in 6 different EU countries
I checked out the most powerful one: Karolina, a Czech data center, it mostly has CPUs though (see pic) not GPUs, so mostly useless for AI
The GPUs it does have are 72x 8x NVIDIA A100 GPU, so 576x A100, or equivalent of 240x H100s
(H100 is about 2.4x the compute power of A100)
So let's compare that:
@xAI has 200,000x H100 GPUs
So the xAI data center has 800x more compute than the Czech one
If we combine xAI, Meta, AWS, etc. it's about 750,000 H100s
If we assume the other 5 data centers in the EU are equivalent to the Czech one (which is massive stretch because most of the others seem AI consultacny services, they don't even HAVE chips!), the EU's new "AI factories" have a total of 1,440x H100 GPUs, let's round up to 1,500 to be nice
So the EU is trying to compete with 750,000 GPUs with their own 1,500 GPUs, so 500x less??
Correct me if I'm wrong but it's just seems very low impact and another ridiculous idea and burning of EU tax payers money that will end up in local cronies and bureaucrats and will do NOTHING to improve the AI business climate for Europe
The best way to improve it is to deregulate, make it super easy and low tax (especially when starting out) to start AI companies in Europe
Oscar Balcells Obeso retweeted
We found "misaligned persona" features in Llama and Qwen that mediate emergent misalignment. Fine-tuning on bad medical advice strengthens these pre-existing features, causing broader undesirable behavior. lesswrong.com/posts/NCWiR8K8…
Oscar Balcells Obeso retweeted
Wouldn't it be great if chat models could indicate their uncertainty as they write?
Our new paper is a concrete step towards this vision, using internal representations to predict hallucination risk in real-time.
Oscar Balcells Obeso retweeted
I'm excited that, this year, interpretability finally works well enough to be practically useful in the real world! We found that, with enough effort into dataset construction, simple linear probes are cheap, real-time, token level hallucination detectors and beat baselines
An unexpected finding: when we train LoRA probes with minimal regularization, models become more epistemically cautious, sometimes acknowledging they've hallucinated immediately after doing so.
We only train to predict binary hallucination labels from its own hidden states—no direct optimization for output behavior. Yet the model spontaneously learns to hedge its claims (e.g., adding "I cannot confirm this" right after making incorrect factual assertions).
📄 Paper: arxiv.org/abs/2509.03531
💻 Code: github.com/obalcells/halluci…
🌐 Website: hallucination-probes.com
This work was done in collaboration with @andyarditi, @javifer_96, @xjoshfree, @CameronHolmes92, and @NeelNanda5.
























