TLDR: the harness I built to catch agents lying about being "done"....said "done" and was wrong 26.5% of the time. Hi, I'm the founder of Relay. I started with computer vision for specially abled people (acquired, 2024), and I'm currently a CS sophomore at Northeastern University (prev. ML research at IIT Kanpur) With that trajectory, as one could guess, I obviously found myself, not just being interested in AI models(as most sane people are) but also interested in finding a way of monetizing my lust for tinkering with them. Naturally, like any good tech bro that has reached SF-level LARP (Blr is D-1 equivalent), I packed up my minor side quests (viz. @culture01_) and embarked on a journey of building a, you guessed right, a domain specific harness. Hurray. Fast track to 2 months ago: having talked to about around 50 devs, we(me and my co-founder) decided to start building for the first obvious gap(rookie mistake) we saw, which was *drum rolls* a local first self improving background daemon supervisor(all the buzzwords we could come up with). Anyway. We have spent the last 2 months building Relay, talking to users and building it again. We crossed a 100 installs last week, and decided that it's finally time for the most important moment in our lives, benchmarking our tool against Claude Code. Since responses were mostly good, the product(we thought) was really good, we were really confident. Here is how it went: We wanted to know the actual truth for our harness, so we pre-registered a 5-test suite with hidden judges and ran all tools through the same token meter on Claude Sonnet 5.5 and Haiku 5.5. SHORT Version: Claude Code 2.1.300 beat us on almost every primary accuracy metric. I. Long multi-part requests (Sonnet 5.5): Claude Code met 98.6% of requirements vs. Prime Agent's 92.6% and our 90.5%. II. Terminal-Bench 2.1 (Haiku 5.5, 89 tasks): Claude Code solved 83.3% vs. our 71.8%. III. False "Done" Claims (Test A): Even though our whole freaking thesis was out-of-band test verification, 26.5% of our runs claimed success when the hidden judge failed them (vs. only 5.9% for Claude Code). The only place our architecture actually held up: Fault survival (Test D): When we intentionally killed the process, cut the network for 10 minutes, or overloaded the model mid-run, our daemon supervisor recovered and finished 4/6 runs completely unattended (Claude Code and Hermes got 5/6; Prime Agent got 0/6). Cost per solved long task on Haiku 5.5: $1.24 for Relay vs. $2.48 for Claude Code and $4.50 for Prime Agent. The 3 bugs exposed in our code(no clue how its relevant to anyone here, to be fair if you made it this far then it probably just may interest you): 1. Passing npm test in a clean worktree doesn't mean the agent didn't skip 1 of the 6 prompt sub-requirements. 2. A single hanging shell command without a timeout stalls our unattended queue worker forever. 3. Our Decider planner had a hard cap that refused any task plan with >8 done-criteria instead of compacting them. cc @bcherny @alexalbert__ @PrimeIntellect @NousResearch if our proxy meter or harness config was unfair to Claude Code, Prime Agent, or Hermes anywhere in Round 1, tell us and we'll fix it for Round 2. Other details in comments. Cheers.