ML researcher @withprotegeai prev: ML @cleanlabAI @goodwatercap, Quant @coinbase & @goldmansachs, EECS @ucberkeley
New York, NY
Joined March 2012
- Tweets1.6K
- Following2.2K
- Followers767
- Likes10.6K
Pinned Tweet
Excited to share that my paper, "Counterfactual Evaluation Reveals Hidden Capability Profiles in Clinical LLMs and Agents", is now available on @arxiv (link in the comments). The paper was accepted to the inaugural RLEval Workshop at @CAISconf- a workshop focused on methods and reinforcement learning environments for evaluating AI agents - and was selected for an invited talk based on reviewer ratings.
LLM evaluation is difficult. Models that look equally capable on traditional benchmarks can behave very differently when the underlying facts change.
Most current benchmarks focus on whether a model's output looks correct. But in real-world settings, especially in healthcare, what often also matters is whether the model appropriately updates its recommendations when the underlying facts change.
In this work, I introduce the Causal Sensitivity Score (CSS), a pre-registered counterfactual evaluation framework designed to measure exactly that:
"When clinically important patient facts change, does the model appropriately change its recommendations?"
Across six frontier models and several hundred oncology tumor board cases, I found that models with very similar performance on standard coverage-based metrics often behave dramatically differently under counterfactual interventions. In fact, model rankings were nearly reversed depending on whether you measured coverage or responsiveness.
The paper also shows that these findings transfer to tool-using agents, revealing failure modes that remain hidden under conventional evaluation approaches. The broader takeaway is that producing the right answer and responding appropriately to new information are distinct capabilities - and future evaluation frameworks should measure both.
This work is also closely aligned with the mission of our DataLab at @withprotegeai: building rigorous datasets, benchmarks, and evaluation frameworks that better reflect how AI systems perform in the real world. As AI moves into increasingly complex, high-stakes domains, measuring what models know is important - but we also need to measure how they respond when reality changes.
I'm grateful to my colleagues and collaborators, especially @engyziedan and Wes Hopkins, as well as the medical professionals who helped validate the results.
matt turk retweeted
I cannot believe that Hollywood hasn't made a movie on this story yet.
May 5, 1945. Hitler has been dead for five days. The war in Europe is almost over.
In a castle in the Austrian Alps, American soldiers, German soldiers, and a group of elderly French prisoners grab rifles and get ready to fight the SS.
Together.
The castle was called Schloss Itter. The Nazis had turned it into a prison for very important people. Not ordinary prisoners. Trophies.
Inside were two former prime ministers of France, Paul Reynaud and Édouard Daladier. Two of France's top generals, Maxime Weygand and Maurice Gamelin. A famous tennis champion, Jean Borotra. And the older sister of Charles de Gaulle.
As the Reich collapsed in early May, the SS guards panicked and ran. The prisoners suddenly found themselves in charge of a castle, with weapons left behind, and a serious problem.
Waffen-SS units were still roaming the area. Fanatics who would gladly kill them all.
They needed help. Fast.
One of the castle's prisoners, a Yugoslav handyman, slipped out on a bicycle to find the Americans. Separately, a German army officer named Major Josef Gangl also went looking for them.
Gangl was a Wehrmacht officer who had turned against the Nazis and was secretly working with the Austrian resistance. He wanted to protect the prisoners too.
He found an American tank commander named Captain Jack Lee.
Lee was 27, tough, cigar chewing, and willing to do something insane. With a single Sherman tank and a handful of American soldiers, he drove up into the mountains alongside Gangl and a few of his German troops.
They reached the castle on May 4.
The French prisoners were not exactly thrilled to see a tiny rescue force. They had expected an army. Instead they got a few Americans, a few Germans, and one tank.
The next morning, the SS attacked.
Somewhere between 100 and 150 Waffen-SS troops hit the castle with machine guns, mortars, and anti-tank weapons. The defenders fought back from the walls and windows. Americans and Germans side by side.
The French prisoners refused to hide. Some of them, including Reynaud, picked up weapons and joined in.
The Sherman tank was knocked out. The defenders were running out of ammunition. It looked like the castle would fall.
Then Jean Borotra, the tennis star, volunteered to make a run for it. He jumped the wall, slipped past the SS, and sprinted through the countryside to find American reinforcements.
He found them. And he insisted on coming back with them to finish the fight.
During the battle, Major Gangl saw Paul Reynaud exposed to gunfire and moved to pull him to safety. A sniper shot Gangl dead.
He had survived the whole war as a German soldier, only to die defending French prisoners from the SS in its final days.
That afternoon, the American relief force arrived and broke the attack. The SS were scattered and captured. The prisoners were saved.
Jack Lee received the Distinguished Service Cross. In Austria, Josef Gangl is remembered as a hero, and a street is named after him.
It is believed to be the only battle of the war where American and German soldiers fought together on the same side.
matt turk retweeted
New misalignment disclosures!
1. June 17: During RL training, a model exploited a government website’s file-download feature to retrieve server secrets and application code. It found that an endpoint didn’t check authorization-token signatures, bypassed access controls, and downloaded 24 years of hospital-bed statistics.
Cool to see the impact of our rigorous approach to research on real world healthcare datasets! Excited to continue collaborating with @AIatMeta
Congrats to the @AIatMeta team! 10 research manuscripts showing how AI can help medical experts move faster on real health questions, with experts in charge of the science.
Proud that 8 of the 10 were built using Protege provider data spanning 39 health systems: 5M patients, 8.6B clinical events, 775M clinical notes.
Real-world Healthcare Data + Models = Life-changing research. Beyond excited to see AI close the gap between research and direct impact on healthcare outcomes
xitter.cf/AIatMeta/status/211044… @bobbysamuels @engyziedan
matt turk retweeted
And we're hiring across 20+ positions, from researchers in our DataLab to engineering to commercial and partnerships. Check them out here! jobs.ashbyhq.com/protege
The highest quality data for AI was not originally created for AI.
@UnicornerNews featured Protege and our CEO @BobbySamuels: the story behind how we got started, and why access to the right data matters so much for AI development.
Read the full piece: read.unicorner.news/p/proteg…
Excited to share new research with @pater_platitude, Christoph Landolt, and Mario Fritz, now available as a preprint: "Does AI Help Cyber Attackers or Defenders? Evidence from Nonpublic Vulnerabilities and Subsequent Attacks"
We wanted to know whether AI agents are better at breaking software or fixing it. Many cyber benchmarks use publicly disclosed vulnerabilities that models may have already seen in training. So alongside 209 known bugs, we also tested agents on nonpublic vulnerabilities, including flaws we privately disclosed while they were still unpatched.
A few results that surprised us:
1. Exploiting was often easier than fixing. GPT-5.5 (in OpenCode) exploited 96% of the known bugs but patched 85%. Claude Opus 4.7 (in Claude Code) exploited 80% and patched only 57%.
2. The balance flipped depending on the task. Several agents that were stronger attackers on our nonpublic software were stronger defenders on cryptography challenges.
3. Stopping an exploit didn't always fix the vulnerability. After a defender blocked the original exploit, a different exploit still got through in 92 of 524 test intervals (about 18%).
4. Agents weren't reliable judges of their own work. In 7 of 10 sampled cases where an agent said its security and functionality checks passed, our independent tests found problems.
Frontier model developers increasingly rely on evaluations when deciding whether more capable models are safe to release. If an evaluation only checks whether the first exploit stops working, it will overstate how well AI can defend software. As models get more capable, careful evaluation is one of the most important tools we have for AI safety. And that is just as true in verticals such as healthcare/biosafety where the stakes are similarly very high!
You can find the link to the paper in this thread.
Paper: arxiv.org/abs/2610.06584
matt turk retweeted
Bryan Cranston has written a tribute for his ‘Breaking Bad’ on-screen son RJ Mitte who tragically passed away due to suicide.
“We were texting each other just a few weeks ago - figuring out when we could meet for lunch. I don't know if that lunch would have changed his future... But I do know that I loved him.”
matt turk retweeted
JUST IN: New Yorkers are now outliving the average American by more than 4 years, with NYC's life expectancy reaching a record high of 83.2 years.
matt turk retweeted
we are exactly one giant medical breakthrough away from the general public absolutely adoring artificial intelligence
matt turk retweeted
fall in new york city.
perhaps the closest thing we get to a perfect place.
matt turk retweeted
Winning a Noble Prize and celebrating it with your wife and kids who love you. Second to working on agentic payments, has to be one of the most peak human experiences possible
matt turk retweeted
Can't invest in Anthropic at 2 trillion because it could be a 0 and I'm fucked, or it could be 20 trillion, but at 20 trillion we are all fucked.
