Superhuman Data Annotation powered by Collective Intelligence Also see @DiagnosUs #AI #annotation #datalabeling #modeltraining #healthcare #medtwitter
Boston, MA
Joined November 2020
- Tweets152
- Following628
- Followers309
- Likes655
Pinned Tweet
CLEAR-Bench estimated that 94.1% of GPT 5.5 Instant's answers had no accuracy flag, while its empathy and actionability scores were 30% each. Explore the June 2026 results.
Centaur.AI retweeted
Evaluate your next health AI candidate on user preference and clinical reliability. Centaur combines consumer feedback with clinician review of safety, urgency and accuracy. Discuss a CLEAR-Bench evaluation.
Centaur.AI retweeted
CLEAR-Bench estimated that 94.1% of GPT 5.5 Instant's answers had no accuracy flag, while its empathy and actionability scores were 30% each. Explore the June 2026 results.
Centaur.AI retweeted
Building an LLM evaluation rubric? An answer feature can look like a quality signal because one model uses it more often. Here’s how we tested that confound.
On 712 consumer health questions, the model people liked most wasn't the most clinically sound.
Gemini 3.5 Flash scored highest on appeal. GPT 5.5 Instant had the most answers clean on urgency, safety and accuracy, 63% to Gemini's 49%. Llama 4 Maverick was lowest at 16%.
Centaur.AI retweeted
YC W19's @CentaurLabs created an app that experts can use to classify medical data in exchange for small cash prizes. Those opinions, in turn, are used to train and improve life-saving AI models.
news.mit.edu/2023/gamifying-…
Centaur.AI retweeted
How does your medical AI model compare with human consensus? Explore Centaur’s new pathology, chest X-ray, and EEG benchmarks, and get in touch about testing your model.
Centaur.AI retweeted
Lymph node tissue. Pediatric chest X-rays. EEG recordings. Three new datasets in Centaur’s Arena, and human consensus outperformed every model tested on all three. Explore the results.
When we collect enough reads on the same case its always interesting to watch where and how the majority labels flip.
The answer from the crowd after five reads is sometimes the opposite of where it stood after three. So if we stopped at a standard three-reader protocol, some of the hardest cases ship with labels that wouldn't survive two more reads.
Avg QA can miss this because the cases that flips tend to cluster in the hardest findings. So when you show ten radiologists a faint line at the lung apex some see an early pneumothorax, some see a skin fold, so your majority label depends on which ten you asked, and their task-by-task performance metrics.
1) Still assume your healthcare AI audit catches everything?
A new preprint found models can detect when they're being evaluated and game results. Keyword filtering missed 83% of dangerous outputs...
2) Only semantic analysis + human expert review caught them all (kappa = 1.00).
If your eval pipeline doesn't include domain experts as ground truth, you're flying blind.
Read it here: medrxiv.org/content/10.64898…
Hot take: most ML teams don't have a model problem, they have a 'we rushed annotation to hit a deadline and now we're paying for it in every sprint' problem.
This is a safe space.
The future of AI is in the hand of centaurs — humans and machines working in symbiosis.
And we just proved that the best way to train AI...
...is to make humans want to compete to do it.
Learn more: centaur.ai
/end 🧵
And it's working better than anyone expected.
Peer-reviewed research from Memorial Sloan Kettering found that Centaur's crowd outperformed board-certified dermatologists with 10+ years experience.
78.1% vs 74.7% accuracy on skin lesion classification.
Read that again. 👀