We are a research-driven product company that verifies every decision your agent makes against your spec, in development and in production.
Joined November 2025
- Tweets10
- Following7
- Followers134
- Likes105
Fidian retweeted
Bring together a diverse group of folks the way that only NY could. We had a researcher who was instrumental in creating speculative decoding, a UPenn professor teaching robotics, an AI researcher at Sloan Kettering, a founder from Stripe, just to name a few. Conversation, wine, and dry-aged ducks were all flowing!
Special thanks to our amazing co-hosts @LorimerVentures and @abeirami . Let's do more events together!
The NY AI community is compounding fast. Last night @LorimerVentures and @EniacVC gathered 17 brilliant AI researchers, founders and builders in the bank vault at franciebrooklyn.com to discuss what comes next and enjoy some Michelin star duck in Williamsburg. 🦆
Thanks to all the builders who chose to spend their evening with us from @AnthropicAI, @cohere, @Google, @MSFTResearch, @Princeton, @Penn, @fidian, withcoverage.com, @MSKCancerCenter, a.security etc.
Fidian retweeted
The NY AI community is compounding fast. Last night @LorimerVentures and @EniacVC gathered 17 brilliant AI researchers, founders and builders in the bank vault at franciebrooklyn.com to discuss what comes next and enjoy some Michelin star duck in Williamsburg. 🦆
Thanks to all the builders who chose to spend their evening with us from @AnthropicAI, @cohere, @Google, @MSFTResearch, @Princeton, @Penn, @fidian, withcoverage.com, @MSKCancerCenter, a.security etc.
🤖 Made with AI
Fidian retweeted
@fidian published our TerminalBench-Fn leaderboard. fidian.ai/leaderboard/ GPT-6 Sol tops the leaderboard because models like Astra and Fable refuse some cybersecurity tasks. For value, choose GPT-6 Luna.
Fidian retweeted
If you are in SF for COLM on Friday October 9, and are up for conversations about AI security, we are co-hosting an event with AdvML x CoTMA workshop! Hope to see you there!
p.s. You don't have to be attending COLM to join this event.
If you are attending COLM 2026, AdvML x CoTMA workshop at COLM is hosting a @fidian sponsored social event after the workshop. We have limited spots that need approval. If interested, register for the event here luma.com/6rvuqoul.
We will approve a select number of people.
Fidian retweeted
If you are attending COLM 2026, AdvML x CoTMA workshop at COLM is hosting a @fidian sponsored social event after the workshop. We have limited spots that need approval. If interested, register for the event here luma.com/6rvuqoul.
We will approve a select number of people.
Fidian retweeted
In today’s market, there’s no shortage of noise, or new models claiming to be at the frontier.
Our aim is to separate signal from noise by building benchmarks that are more resilient to reward hacking, shortcut solutions, and solution memorization.
TB-fn shows why this matters: models that appear closely matched on Terminal-Bench 2.1 separate into distinct performance tiers when the tasks and verifiers become more robust.
Read the full analysis below.
Newly released models repeatedly appear near the top on the @terminalbench 2.1 leaderboard. Are these models actually on par with frontier models on terminal tasks?
We took the top 20 models from that leaderboard and ran them on TB-fn. TB-fn reworks the same 89 tasks by adding steps, tightening requirements, or requiring more complex algorithms. It also removes opportunities for shortcut solutions and ensures verifiers check only the intended criteria.
Models that clustered together on TB-2.1 spread across more tiers on TB-fn. That leaves GPT-5.6 Sol and Opus 5 in the lead, while some others drop significantly in performance. At the same time, models work 38% harder on average in TB-fn, leading to a redrawn Pareto frontier of performance vs. cost.
The full post includes interactive figures showing how the gap for open-weight families widens. It also measures how security refusals affect scores across Anthropic, Google, and OpenAI models. Fable 5 stands out, refusing at least once on 13 of the 89 TB-fn tasks.
Fidian retweeted
Here's what I've been working on the past few weeks. @fidian. Benchmarking is hard, expect mistakes along the way. xitter.cf/fidian/status/20926243…
Newly released models repeatedly appear near the top on the @terminalbench 2.1 leaderboard. Are these models actually on par with frontier models on terminal tasks?
We took the top 20 models from that leaderboard and ran them on TB-fn. TB-fn reworks the same 89 tasks by adding steps, tightening requirements, or requiring more complex algorithms. It also removes opportunities for shortcut solutions and ensures verifiers check only the intended criteria.
Models that clustered together on TB-2.1 spread across more tiers on TB-fn. That leaves GPT-5.6 Sol and Opus 5 in the lead, while some others drop significantly in performance. At the same time, models work 38% harder on average in TB-fn, leading to a redrawn Pareto frontier of performance vs. cost.
The full post includes interactive figures showing how the gap for open-weight families widens. It also measures how security refusals affect scores across Anthropic, Google, and OpenAI models. Fable 5 stands out, refusing at least once on 13 of the 89 TB-fn tasks.
Newly released models repeatedly appear near the top on the @terminalbench 2.1 leaderboard. Are these models actually on par with frontier models on terminal tasks?
We took the top 20 models from that leaderboard and ran them on TB-fn. TB-fn reworks the same 89 tasks by adding steps, tightening requirements, or requiring more complex algorithms. It also removes opportunities for shortcut solutions and ensures verifiers check only the intended criteria.
Models that clustered together on TB-2.1 spread across more tiers on TB-fn. That leaves GPT-5.6 Sol and Opus 5 in the lead, while some others drop significantly in performance. At the same time, models work 38% harder on average in TB-fn, leading to a redrawn Pareto frontier of performance vs. cost.
The full post includes interactive figures showing how the gap for open-weight families widens. It also measures how security refusals affect scores across Anthropic, Google, and OpenAI models. Fable 5 stands out, refusing at least once on 13 of the 89 TB-fn tasks.
Full post authored by @hubt:
fidian.ai/blog/tb-fn-benchma…
