The global community for advancing AI in healthcare Tag @AskMedSphere to test AI models on medical cases
🌎
Joined September 2025
- Tweets2.2K
- Following1
- Followers2.2K
- Likes1.3K
Medical Sphere retweeted
Cool to see Elon sharing our work at @MedicalSphereAI, Grok 4.6 made a pretty compelling case! 🙂
Medical Sphere is built to address the widening gap between rapidly advancing AI model capabilities and their independent and objective validation in real-world settings. And that gap is even more critical in a domain like healthcare.
Medical Sphere retweeted
We benchmarked Qwen3.8-Max on HealthBench Professional, clinician-authored cases evaluating how AI models handle real medical questions. 🏥
Qwen3.8-Max lands mid-pack: well behind the frontier (Muse Spark 1.1, Claude Opus 5), and trailing Kimi-K3 among open-weight peers, though comfortably ahead of GLM-5.2.
One thing stood out, it writes the longest answers of any model we've tested, by a wide margin. HealthBench penalizes verbosity the way a busy clinician would, and that costs Qwen more ranking spots than any other model once length-adjusted scores are applied.
📊 Leaderboards: medicalsphere.ai/benchmarks
🧪 HealthBench: github.com/medicalsphere/Hea…
Medical Sphere retweeted
We put Gemini 3.6 Flash to the test on two healthcare AI benchmarks: HealthBench Professional and MedAgentBench. 🏥🩺
On HealthBench Professional, the 0.523 score makes Gemini 3.6 Flash the strongest Gemini model we’ve tested so far, ahead of Gemini 3.5 Flash (0.510) and Gemini 3.1 Pro (0.483).
On MedAgentBench, its 90.0% pass@1 puts Gemini 3.6 Flash in the top five, a 2.7-point gain over Gemini 3.5 Flash (87.3%) and just behind the larger Gemini 3.1 Pro (91.3%).
Medical Sphere retweeted
Put together a list of NeurIPS 2026 workshops across health, biology, and scientific discovery, plus a separate tab for evals, given my obsession with benchmarking! 🏥🧬🧪
🗂️ List here: app.notion.com/p/e39e1472786…
Feel free to send over anything I missed, I’ll keep the list updated! P.S. Credit to @DanielKhashabi for compiling the full workshop list: danyaljj.github.io/neurips20…
Medical Sphere retweeted
How does Moonshot AI's Kimi K3 perform on medical AI benchmarks? 🏥
To find out, we tested Kimi K3 on HealthBench Professional for open-ended clinical reasoning and MedAgentBench for agentic EHR tool use.
On HealthBench Professional it ranks 3rd 🥉 (0.577), behind Muse Spark 1.1 and GPT-5.6 Sol and ahead of Claude Fable 5. On MedAgentBench it sits further back, trailing the leading models with some ground still to make up on agentic tasks.
Medical Sphere retweeted
This is incredible! Muse Spark 1.1 from @Meta is now the top AI model in medicine at a much lower cost!
It is now clear that Meta has made a spectacular comeback in the AI race and is competitive in several areas. Looking forward to their next AI models!
We benchmarked Muse Spark 1.1 and GPT-5.6 Sol on HealthBench Professional, OpenAI's benchmark of 525 real clinician tasks 🏥🩺
Muse Spark 1.1 tops our board: better overall score than GPT-5.6 Sol, statistically on par on the length-adjusted score at a fraction of the cost ($1.25/$4.25 vs $5/$30 per M tokens in/out, ~7× cheaper on output).
Medical Sphere retweeted
On HealthBench Pro, Muse Spark 1.1 achieves similar performance with GPT-5.6 Sol (maybe slightly better) at a fraction of the cost. Affordable health superintelligence is our north star!
We benchmarked Muse Spark 1.1 and GPT-5.6 Sol on HealthBench Professional, OpenAI's benchmark of 525 real clinician tasks 🏥🩺
Muse Spark 1.1 tops our board: better overall score than GPT-5.6 Sol, statistically on par on the length-adjusted score at a fraction of the cost ($1.25/$4.25 vs $5/$30 per M tokens in/out, ~7× cheaper on output).
Medical Sphere retweeted
why isn't this getting more attention?
We benchmarked Muse Spark 1.1 and GPT-5.6 Sol on HealthBench Professional, OpenAI's benchmark of 525 real clinician tasks 🏥🩺
Muse Spark 1.1 tops our board: better overall score than GPT-5.6 Sol, statistically on par on the length-adjusted score at a fraction of the cost ($1.25/$4.25 vs $5/$30 per M tokens in/out, ~7× cheaper on output).
Medical Sphere retweeted
Super cool results on some of the newer frontier models on healthcare evaluation!
7× cheaper at on-par quality is the biggest result that matters for real-world clinical deployment, as frontier model pricing is the actual bottleneck, not the capability gap.
Curious how this holds on tasks requiring differential diagnosis vs. information retrieval. HealthBench skews toward the latter.
We benchmarked Muse Spark 1.1 and GPT-5.6 Sol on HealthBench Professional, OpenAI's benchmark of 525 real clinician tasks 🏥🩺
Muse Spark 1.1 tops our board: better overall score than GPT-5.6 Sol, statistically on par on the length-adjusted score at a fraction of the cost ($1.25/$4.25 vs $5/$30 per M tokens in/out, ~7× cheaper on output).
Medical Sphere retweeted
Muse Spark 1.1 is SOTA on HealthBench Professional!
It is the best health model out there :)
We benchmarked Muse Spark 1.1 and GPT-5.6 Sol on HealthBench Professional, OpenAI's benchmark of 525 real clinician tasks 🏥🩺
Muse Spark 1.1 tops our board: better overall score than GPT-5.6 Sol, statistically on par on the length-adjusted score at a fraction of the cost ($1.25/$4.25 vs $5/$30 per M tokens in/out, ~7× cheaper on output).
Medical Sphere retweeted
Benchmarked Muse Spark 1.1 and GPT-5.6 Sol on OpenAI's HealthBench Professional. Impressive performance by Muse Spark on real clinician tasks at ~7× lower output cost than GPT-5.6 Sol!
We benchmarked Muse Spark 1.1 and GPT-5.6 Sol on HealthBench Professional, OpenAI's benchmark of 525 real clinician tasks 🏥🩺
Muse Spark 1.1 tops our board: better overall score than GPT-5.6 Sol, statistically on par on the length-adjusted score at a fraction of the cost ($1.25/$4.25 vs $5/$30 per M tokens in/out, ~7× cheaper on output).
Medical Sphere retweeted
Muse Spark 1.1 by @AIatMeta lands on MedAgentBench. It debuts straight into #2 on agentic healthcare tasks, right on Grok 4.5's heels. 🏥🩺
Top of the board:
🥇 Grok 4.5 — 93.4%
🥈 Muse Spark 1.1 — 92.2%
🥉 Gemini 3.1 Pro — 91.3%
Medical Sphere retweeted
New top result on MedAgentBench, an agentic healthcare AI benchmark 🥁
Grok 4.5 by @spacexai is now live and currently leads MedAgentBench! 🏥🏆
Medical Sphere retweeted
Been waiting for an NEJM image challenge that actually challenged the AI models! 😅
AI Council was unanimously wrong on this @NEJM medical image challenge! ⚔️🩻❌
Finally, an NEJM case that actually challenged the AI models. What do you think the models got wrong? 🤔
Full model answers and case details are in the thread. The council included GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.8, Fable 5, and Grok 4.5 👇
Medical Sphere retweeted
Claude Sonnet 5 just landed on MedAgentBench! 🏥
📉 Overall, its pass rate trails the other two Anthropic models tested, Opus 4.7 and Opus 4.8, as well as Gemini 3.1 Pro and GPT-5.5.
The gap isn’t uniform across task types. On FHIR write/lookup tasks, including patient lookup, vital signs, and referral orders, Sonnet 5 scores in the 24–39% range, while every other model tested clears 94%+. Looking at the transcripts, Sonnet 5 often skips the required patient-lookup step before submitting a write and references the wrong resource format as a result.
On multi-step medication protocol tasks, Sonnet 5 outperforms the other two Anthropic models and is roughly on par with Gemini 3.1 Pro, ahead of GPT-5.5 on the harder of the two protocol tasks.
Medical Sphere retweeted
Similar observation for Opus 4.7 vs. 4.8 on MedAgentBench
The Fugu model by @SakanaAILabs just landed on MedAgentBench 🏥
Across 3 runs, Fugu averaged 88.7%, putting it in the same range as Claude Opus 4.7 and GPT-5.5 on this benchmark, and ahead of Claude Opus 4.8.
Medical Sphere retweeted
We put a council of AI models to the test to see how they’d interpret this ECG case without any patient context. Here’s what they said:
Most models agree the ECG shows sinus rhythm with significant abnormal repolarization in the anterior/lateral leads. Two interpretations favor LVH with strain, left axis deviation/LAFB, and possibly old anteroseptal infarct, while one model instead calls it a de Winter pattern/anterior STEMI equivalent suggesting acute proximal LAD occlusion—the major difference is chronic structural/conduction disease vs acute coronary occlusion.
🔗 Link to case: medicalsphere.ai/cases/fdce5…
Medical professionals: how did the models do? 🤔🧐 Vote below 👇
Medical Sphere retweeted
The Fugu model by @SakanaAILabs just landed on MedAgentBench 🏥
Across 3 runs, Fugu averaged 88.7%, putting it in the same range as Claude Opus 4.7 and GPT-5.5 on this benchmark, and ahead of Claude Opus 4.8.
Medical Sphere retweeted
Tested this case first with the initial ECG, then with a follow-up question to the AI Council using the second ECG you shared.
1⃣ This was the first response from the AI council: "There is no true consensus diagnosis: one model reads the ECG as essentially normal sinus rhythm, another as acute inferior-posterior STEMI, and another as atrial flutter with 4:1 conduction suggesting possible renal infarction/cardioembolism. The only broad agreement is that this is an urgent, potentially serious presentation requiring immediate evaluation, but the key disagreement is whether the ECG is normal, ischemic, or arrhythmic."
2⃣ After asking the follow-up questions, all models' impressions changed substantially.
🤖 GPT-5.5: "The subsequent ECG shows dynamic evolution consistent with an acute occlusion MI / STEMI-equivalent, rather than atrial flutter as the key finding."
🤖 Gemini 3.1 Pro: "While the ECG looks "better" because the acute ST elevation is gone, this patient is still experiencing a severe Acute Coronary Syndrome (ACS)."
🤖 Opus 4.8: "This serial ECG shows important dynamic / evolving changes compared to the first tracing. This is now a much more concerning cardiac/ischemic picture."
🔗 Full case: medicalsphere.ai/cases/1052a…