Mehdi Fatemi (@mefatemi) header
Mehdi Fatemi (@mefatemi) profile picture

Researcher and Team Lead | RL & Reasoning | ex-Microsoft | views are personal

Canada
Joined June 2009
Photo by Mehdi Fatemi (@mefatemi)Photo by Mehdi Fatemi (@mefatemi)Photo by Mehdi Fatemi (@mefatemi)Photo by Mehdi Fatemi (@mefatemi)Photo by Mehdi Fatemi (@mefatemi)Photo by Mehdi Fatemi (@mefatemi)Photo by Mehdi Fatemi (@mefatemi)Photo by Mehdi Fatemi (@mefatemi)Photo by Mehdi Fatemi (@mefatemi)Photo by Mehdi Fatemi (@mefatemi)
📈 We trained Qwen3.8-27B on 1,105 @SnorkelAI expert-built GDPval+ tasks. Pass@1 on OpenAI’s GDPval gold set rose from 31.9% to 40.7%. Standard GRPO on the same data barely moved. The difference was learning from failure. 🧵 (1/N) Blog: snorkel.ai/blog/gdpval-agent…
8
9
2
85
6,216
Snorkel published a new post from @mefatemi on training agents with binary-gated GRPO on GDPval tasks. Qwen3.8-27B was trained on 1,105 expert-built GDPval+ tasks, with about 102K rollouts in total. GRPO learned from groups of eight attempts per task. A group where every attempt failed gave it no signal, and 79% of generated groups had no passing rollout. The failed attempts were far from identical, and the median best attempt in the discarded groups scored 78% on the rubric. Binary-gated GRPO used pass/fail when a group had a pass and fell back to the rubric score only when all eight attempts failed. Pass@1 rose from 31.9% to 40.7%. The trained model beat the base model on rubric pass in all nine sectors. Standard GRPO on the same data barely moved. Full write-up: snorkel.ai/blog/gdpval-agent…
📈 We trained Qwen3.8-27B on 1,105 @SnorkelAI expert-built GDPval+ tasks. Pass@1 on OpenAI’s GDPval gold set rose from 31.9% to 40.7%. Standard GRPO on the same data barely moved. The difference was learning from failure. 🧵 (1/N) Blog: snorkel.ai/blog/gdpval-agent…
1
6
285
New from our research team: Qwen3.8-27B, trained on 1,105 expert-built GDPval+ tasks, goes from 31.9% to 40.7% pass@1 on OpenAI’s GDPval gold set. Standard GRPO barely moved. Binary-gated GRPO learns from failed attempts. Write-up by @mefatemi: snorkel.ai/blog/gdpval-agent…
📈 We trained Qwen3.8-27B on 1,105 @SnorkelAI expert-built GDPval+ tasks. Pass@1 on OpenAI’s GDPval gold set rose from 31.9% to 40.7%. Standard GRPO on the same data barely moved. The difference was learning from failure. 🧵 (1/N) Blog: snorkel.ai/blog/gdpval-agent…
2
23
912
(5/5) When a group contains a pass, pass/fail keeps strict priority. When all eight attempts fail, a training-score fallback takes over, with a 40% floor. Groups with signal: 21% → 95% Budget exhaustion: 17.2% → 7.0%
7
214
(4/N) In the median discarded group, the best rollout reached a 78% training score. In 66% of discarded groups, at least one rollout scored 70% or more. Binary-gated GRPO uses this variation only when every attempt in the group fails.
6
208
(3/N) With a binary return, 79% of generated groups had no passing rollout and zero advantage. Dynamic sampling could skip these groups, but it could not create a learning signal. Yet the failed attempts were far from identical.
7
320
Replying to @mefatemi @SnorkelAI
(2/N) Its deliverables were rated above the human expert’s in 44.0% of LLM-judged comparisons, up from 36.4%.
8
371
✈️ We’ll be at @NeurIPSConf Australia! Accepted papers, happy hours, and more: snorkel.ai/neurips-2026
1
5
33
1,540
Mehdi Fatemi retweeted
Using a theoretically grounded and Snorkel-developed lens (RIFT) we investigate 147 rubric-graded tasks, 35% had redundant criteria. The rate ranged from 3% of software engineering tasks to 73% of reasoning tasks. Redundancy was nearly absent where tasks have specific and checkable behaviors. It was most common in open-ended tasks, where several criteria often score the same idea. Full results: snorkel.ai/blog/checking-the…
1
6
19
815
Mehdi Fatemi retweeted
An important and good point! Ideally we should be judging based on output criteria: - Quality (incl realism) according to detailed and properly factorized spec - Complexity/difficulty - Alignment with target usage/gap distribution - Robustness to reward hacking / general rubric quality - Etc If the spec is sufficiently well defined, the input should purely be part of the how, not the what
"real envs > synthetic envs" feels like mostly 3 things: - a complaint about sloppy synthetic data pipelines - a way to make envs pricier without making them better - synthetic task authors guessing the task distribution because they've never worked with real production traffic at scale real envs ship very hackable tasks too. whereas a carefully built synthetic env that's realistic enough has all the same benefits as real envs, + you get to aim at one failure mode on purpose and isolate it from the random noise
2
7
1
31
2,931
amazing progress from the AA team and proud to support some of the new and upgraded benchmarks via Open Benchmarks Grants!
Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers. Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale. New to the Intelligence Index: ➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change ➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment Upgraded: ➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions ➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised ➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0 Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1 Key results: ➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50 ➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36) ➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04
5
6
41
1,532
Mehdi Fatemi retweeted
The @SnorkelAI team did an audit of these findings using our internal QC platform. Tl/dr: We agree with many of the findings, but ~55% of the fixes from @preferencemodel had their own issues with reward hacking or over-prescriptiveness. Additionally: ~40% of the issues found were already tracked in the upstream repo - including 21 tasks with outstanding fix PRs! Building datasets like this - and maintaining ambitious open benchmarks around them - is hard. We believe it requires open, collaborative review with community maintainers and relevant domain experts to get right. @terminalbench was one of the first community efforts to build an open, continuous benchmark, and drive a vision of how OSS contributes to the rapidly advancing frontier of AI. We hope that the community will invest in working collaboratively with benchmark maintainers and task authors to keep this vision alive. --- Some takeaways + details of our preliminary audit: (1) We confirm 100% of the reward hacking / verification gap findings using our QC system. However, we find that 17/44 (39%) of the corrected tasks are just as or more reward hackable. Additionally, we don’t yet see evidence that these verification gaps meaningfully affected benchmark rollouts within CIs, though our analysis here is still ongoing. (2) Our preliminary review raises questions about the underspecification findings for 22/41 (54%) of the flagged tasks, including 7 explicitly marked by task authors as intentional design choices. These cases warrant expert peer review and adjudication: balancing over/under-specification is a critical and tricky design decision - much of human intelligence concerns dealing correctly with reasonable and realistic underspecification - and we believe these corrections made to the tasks are likely too over-specified, reducing realism and difficulty relative to spec. (3) Around 40% of the issues found were already tracked in the upstream repo (including 21 tasks with outstanding fix PRs)! OSS projects require community efforts to correct identified open issues, and ideally audits like these are done collaboratively with task authors and maintainers directly, in the traditional community spirit that has kept OSS alive for decades.
We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated. Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
25
61
12
266
42,808
Join us Wednesday, November 4 at 4:00 p.m. PT (+🧋) for our next Reading Group on Intelligence per Watt. RSVP: luma.com/41dbkzu4 @JonSaadFalcon and @Avanika15 (@HazyResearch, @Stanford) will present Intelligence per Watt: Measuring Intelligence Efficiency of Local AI, accepted to @NeurIPSConf 2026. Across 20+ models, 8 accelerators, and 1M real-world queries, IPW measures how much capability local AI delivers per watt, and which queries belong on-device instead of in the cloud.
1
5
2
25
2,708
Mehdi Fatemi retweeted
As the online hype around AI data brokering hits a fever pitch, a lot are missing the basic distinction between - (A) Unrefined data for pre-training: I.e. raw company records (potentially de-identified) - (B) Refined data for post-training/eval: I.e. well-formed, high-quality tasks, envs, rubrics based on private data or human traces (A) is low margin brokerage - and will commoditize rapidly, esp as pre-training data curation techniques mature and increase discriminative capabilities for buyers. (B) is an incremental (but exciting) evolution of the sophisticated frontier data/envs that AI data labs/vendors already build. Margin/differentiation here will only increase as the frontier continues to advance, raising the quality and complexity bar.
19
13
10
258
30,093
Mehdi Fatemi retweeted
Which model would you rather have? One always makes partial progress but never finishes (pass@k = 0). The other finishes one time in five. Yet partial credit prefers the first! Our new post shows when and why, and how "binary-gated GRPO" addresses it: snorkel.ai/blog/when-partial…
2
11
3
27
3,624
not all failures are equal! @mefatemi + @SnorkelAI research shows how partial credit can lead to models that prefer 'better' failures over task completion
Which model would you rather have? One always makes partial progress but never finishes (pass@k = 0). The other finishes one time in five. Yet partial credit prefers the first! Our new post shows when and why, and how "binary-gated GRPO" addresses it: snorkel.ai/blog/when-partial…
4
16
752
Mehdi Fatemi retweeted
Very cool result from the @SnorkelAI research team on how to adjust GRPO to effectively improve recall on RL-tuned models for long horizon tasks (esp impactful in domains with available verifiers!)
Which model would you rather have? One always makes partial progress but never finishes (pass@k = 0). The other finishes one time in five. Yet partial credit prefers the first! Our new post shows when and why, and how "binary-gated GRPO" addresses it: snorkel.ai/blog/when-partial…
3
21
1,012
New research from @mefatemi: partial credit can rank a policy that never finishes above one that does. Here's when it happens, and how binary-gated GRPO fixes it.
Which model would you rather have? One always makes partial progress but never finishes (pass@k = 0). The other finishes one time in five. Yet partial credit prefers the first! Our new post shows when and why, and how "binary-gated GRPO" addresses it: snorkel.ai/blog/when-partial…
2
3
21
1,078
Which model would you rather have? One always makes partial progress but never finishes (pass@k = 0). The other finishes one time in five. Yet partial credit prefers the first! Our new post shows when and why, and how "binary-gated GRPO" addresses it: snorkel.ai/blog/when-partial…
2
11
3
27
3,624
Today, we’re expanding @SnorkelAI’s Open Benchmarks Grants by 10x to a $30M commitment: - Funding a more diverse, robust, and continuously-updated ecosystem of open benchmarks - Launching the Open Benchmarks Red Team to continually test and strengthen them - Introducing the Snorkel Research Fellowship to support independent contributors developing new evaluation methods Since our launch earlier this year, we’ve been humbled to partner with the teams behind Terminal-Bench, TB-Science, ARC-AGI-3, OSWorld 2.0, Agents’ Last Exam, Continual Learning Bench, Senior SWE-Bench, SlopCodeBench, and more. Alongside these teams, we’ve helped design evaluation methodologies, develop task construction pipelines, and scale quality control. OBG-funded benchmarks have appeared on the latest model cards from every major frontier lab; have helped to index and measure frontier progress and alignment; and have contributed to guiding the frontier of AI development. Benchmarks have always been guideposts for—and drivers of— AI progress: setting a metric and then “hillclimbing” against it is the core of how AI works! But when benchmarks fall behind frontier capabilities, our ability to evaluate and align these systems does as well. They become too simple, static, or correlated, and the risk of “benchmaxxing” increases. We need more diverse, robust, and continuously-updated benchmarks, developed by a broader ecosystem of researchers and domain leaders. We’re incredibly excited to accelerate frontier evaluation with Open Benchmarks Grants— raising the ambition and investment in robust, open, and independent measurement. Read our full announcement and apply here: benchmarks.snorkel.ai/fronti…
49
89
25
307
54,406