When an AI system gives you the wrong output in production, where do you most often find the actual failure?
17%Context/Retrieval
50%Model/Reasoning
17%Handoffs/Retries
17%Evals/Checks
12 votes • 7 hoursAI products are increasingly becoming systems, not single model calls.
A model retrieves context. Another transforms it. Another checks it. Another acts on it.
At that point, “did the model get the answer right?” stops being a complete reliability question.
The answer can be right at one step and wrong by the time it reaches the user.
We’ve spent the last few months building a research desk & studying what accuracy looks like at the system level.
Tomorrow, we’re publishing the first piece.
If AI could generate high-accuracy documents, which use-cases would you finally trust it with?
21%Legal Documents
64%Financial Documents
7%Healthcare Documents
7%I'll add to replies
14 votes • Final resultsMost AI debugging still starts with:
“Can we use a better model?”
But once an AI product has multiple steps, the model is only one place things can break.
A bad result could mean:
→ the right document wasn't retrieved
→ the right document was retrieved, but the wrong section was used
→ the answer was right, but a number was copied incorrectly
→ one step hallucinated and every step after it trusted the hallucination
The question that's being asked now: “Which model failed?”
The question that needs to get asked: “Where did truth leave the pipeline?”
A model can be 99% accurate and still power an unreliable product.
Because the model is only one step in the system.
The wrong context gets retrieved.
A source is stale.
A field gets dropped between steps.
A correct output gets transformed incorrectly downstream.
Production accuracy isn't a model score.
It's the probability that the entire chain produces the right result.
The next phase of AI won’t be defined by which model we trust most. It’ll be defined by how little trust the system needs to place in any single model.
As agents get access to money, data and real-world actions, model quality alone stops being enough.
The controls around the model start to matter just as much: what it can access, what it’s allowed to do, what gets logged, how its outputs are checked, and whether the system can stop it before a bad decision turns into an action.
Verification, permissions, auditability and containment have to live outside the intelligence being governed.
The goal isn’t a perfectly trustworthy model. It’s a system that stays trustworthy even when the model isn’t.
That’s the premise behind Mira: outputs are treated as something to be independently checked, not something to be trusted because a capable model produced them.
Capability can sit inside the model. Trust has to be designed around it.
Drop the most suspiciously specific thing you've read on the internet this week for us to Probe!
The internet is full of claims that look convincing at first glance.
Probe lets you check them where you find them — right in the browser.
Don’t trust us. Probe it.
Install the extension here: useprobe.xyz/
The internet is full of claims that look convincing at first glance.
Probe lets you check them where you find them — right in the browser.
Don’t trust us. Probe it.
Install the extension here: useprobe.xyz/
Is this true? 👀
Replying to @hilbertspaess
The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.
95% accurate sounds reassuring until your agent is about to act on the 5%.
Benchmarks tell you how a model performed across a test set.
They don't tell you whether the output sitting in front of you right now is correct.
How often do you get misled by exaggerated claims in a week?
35%Everyday
13%Every hour
32%That's just the internet
19%Never
31 votes • Final resultsIn Saturn’s test of 18 AI models, financial answers were wrong 57% of the time on average. For complex questions, that rose to 88%.
Financial tasks expose a difficult property of AI: an answer can be internally coherent while resting on a false premise.
The arithmetic can be correct while the tax rule is outdated. The source can be real while applying to the wrong jurisdiction. Every subsequent step can look reasonable because it inherits the same initial error.
Give that output access to tools, accounts and payment rails, and the error can propagate into an action.
This is why verification needs its own place in the architecture.
Before execution, a financial agent should have to pass several distinct checks:
• Evidence: do the sources actually support the claims?
• Context: are those sources current and applicable to this user?
• Computation: can the numbers be reproduced with deterministic tools?
• Authorization: is the proposed action within the user’s permissions and limits?
Asking the same model to reconsider its answer provides limited independence. Agreement between multiple models is also insufficient when they share the same blind spots.
Verification needs external evidence, explicit constraints and a way to stop execution when checks fail.
The architectural question for agentic finance is becoming unavoidable:
What has to be independently checked before an output is allowed to move money?
A model can encounter the same wrong number on a thousand websites.
It may look like a thousand supporting examples.
In reality, they might all trace back to one bad source.
Self-review helps catch some errors.
It can also repeat the same assumption in a more convincing paragraph.
The writer and reviewer are working from the same learned patterns.





