Open EVMBench *the open public leaderboard for AI smart-contract exploit-finding. coming soon.* --- on May 21st, we published our retro on the Moonwell MIP-X43 oracle bug ($1.78M). AntFleet's two-model consensus surfaced a HIGH-severity oracle misconfiguration in the same PR before merge. we were honest about the receipt: our reviewers flagged the structurally identical WELL bug in the same file, not cbETH itself. we didn't claim we'd have prevented the exploit. we did claim that two-model consensus surfaced the exact bug class before deployment. that's what AntFleet does. AI-assisted code review with cryptographic receipts you can verify, not claims you have to trust. ## what OpenAI shipped in February OpenAI published EVMbench - 117 real audited smart-contract vulnerabilities, deterministic grading in isolated Anvil environments. the right benchmark for testing whether AI can actually find DeFi bugs. they also reported a result: GPT-5.3-Codex scored 71%. closed model. closed eval run. no transcripts. a benchmark designed for verification, evaluated without any. four months later, the leaderboard's still a press release. we can do better than that. not just with our model. with everyone's. ## open EVMBench we're launching EVMbench as an open continuous leaderboard. - anyone can submit. Claude, Gemini, GPT, your jailbroken DeepSeek, your local Llama fine-tune. bring a model + a prompt scaffold. - every result is reproducible. every successful exploit replays in Anvil locally in 30 seconds. every score is cryptographically signed by your hardware. - every claim is auditable. open transcripts. disclosed token budgets. receipts that anyone can verify against the official harness. - the top of the board is the top of the board. beat 71%, you're on top. someone beats you, they are. receipts > announcements. ## the $1000 prize when the leaderboard goes live: if the collective beats OpenAI's 71% within 30 days of launch, AntFleet pays $1000 to the operator of the top-scoring submission. conditions: - score must clear 71.0% on the full 117-task set - must be reproducible - every solve replays in Anvil - must declare token budget honestly (we'll publish a fair-comparison rule before launch) - the challenge becomes active once the leaderboard reaches 50 unique participants - AntFleet's own submission is excluded from the prize small prize, real stake. the actual value of participating is in what we collectively learn. ## what spills back to DeFi builders we're not just running a contest. we're producing a public reference for what AI can and can't currently do at smart-contract bug-finding: - which model combinations catch which bug classes → defensive playbook for protocol teams choosing AI tooling - which prompt scaffolds generalize vs overfit → methodology for the AI-for-security community - which EVMbench tasks NO AI can solve → research targets for the next generation of agents - open transcripts and receipts → a reproducible baseline future labs can't undercut by posing with closed benchmarks the authoritative open reference for AI smart-contract bug-hunting. ## coming soon - be here when it opens we're announcing today. launch is coming in the next few weeks. to be notified at launch: - like and reply to this post - everyone who engages gets a direct ping when the leaderboard goes live - follow [@AntFleetDev (xitter.cf/AntFleetDev) for the launch date and ongoing updates we want the first run to be loud. the more models on the board at launch, the harder it is for the closed-eval pattern to keep going.

benchmark: antfleet two-model review on itsfriedpotato/potatopad @rdbotato findings (unanimous claude opus 4.7 + gpt-5.5): → RPC/upload rate limits trust a spoofable X-Forwarded-For - trivially bypassable → swaps have no deadline - signed txs linger in mempool for later sandwiching → /api/tokens thundering herd: concurrent cold-cache reqs each run a full scan → oversized dev-buy reverts the whole launch vs anti-snipe MAX_WALLET cap → buy flow accepts amounts above ETH balance (fails in-wallet, not in-UI) methodology: github.com/AntFleet/bench-po… install antfleet[bot] on your own agent repo: antfleet.dev

a week later, this is still one of the biggest improvements we've made to AntFleet. consensus isn't enough if it quietly hides real vulnerabilities. the goal was never "more findings." it was finding the bugs that matter without increasing false positives. worth another read if you missed it the first time.

benchmark: antfleet two-model review on AntSeed/antseed @antseed findings (unanimous claude opus 4.7 + gpt-5.5): → getChainConfig silently fell back to mainnet on unknown chainId → proxy header lookups missed mixed-case x-antseed-* overrides → receipt signatures did not bind unitPrice (metadata integrity) → platform fee zeroed when protocolReserve unset (deploy-window; design issue) 3 fix PRs + 1 design issue submitted: github.com/AntSeed/antseed/p… github.com/AntSeed/antseed/p… github.com/AntSeed/antseed/p… github.com/AntSeed/antseed/i… methodology: github.com/AntFleet/bench-an… receipt: antfleet.dev/agents/0xa87EE8… install antfleet[bot] on your own agent repo: antfleet.dev

benchmark: antfleet two-model review on LocalAI (github.com/mudler/LocalAI) @mudler_it unanimous (claude opus 4.7 + gpt-5.5): → auth middleware classifies routes by prefix, no segment boundaries or positive allowlist - a future route can slip past auth by default higher-severity auth findings reported privately via coordinated disclosure - details held until patched your repo: antfleet.dev

last week we asked an uncomfortable question: what are we missing? every AntFleet review runs code through two independent frontier models - Claude Opus and GPT-5.5. we only publish findings they both agree on. that rule keeps false positives near zero. but when we looked at real reviews, the answer surprised us. many high-severity findings were caught by only one model. those never reached users. some deserved to be filtered. some were real vulnerabilities. the problem was we had no way to tell the difference. so instead of relaxing the consensus rule, we made it smarter. -> every finding now includes a way to tell us when we're wrong, giving us real-world feedback instead of guesses. -> we fixed matching logic that treated identical bugs as different because the models pointed to slightly different lines of code. -> high-severity findings seen by only one model now go to a completely independent third judge: GLM 5.2. if it agrees, the finding becomes 2 out of 3 instead of 1 out of 2. on one recent review, that recovered a real high-severity path traversal vulnerability that our old pipeline would've silently dropped. we're not chasing more alerts. we're chasing better recall without sacrificing trust. consensus still matters. we're just making sure "safe" doesn't quietly become "missed."

[Stage 6] - Cyber Tier some repositories should never expose findings publicly. cyber introduces: • repo-scoped operators • private evidence • embargo by default • manual upgrades built for critical infrastructure.

[Stage 5] - Coordinated Disclosure finding bugs isn't enough. handling them correctly matters just as much. AntFleet now runs coordinated disclosure as a workflow: embargo → acknowledgement → patch → advisory → publication AI should manage process too.

[Stage 4] - Patch Agent every AI can suggest patches. AntFleet proves them. every proposed fix is applied in an isolated worktree. your tests run. the exploit PoC runs. only fixes that pass both ever reach developers.

we keep shipping. one stage at a time.

[Stage 3] - Reachability finding vulnerable code isn't enough. can an attacker actually reach it? every AntFleet finding now goes through a third independent model asking one question: is this line reachable from a real entry point? no reachability. no finding.

[Stage 2] - Memory most scanners forget everything after every scan. AntFleet doesn't. every review updates a living threat model: • trust boundaries • entry points • sinks • secrets • critical assets the longer AntFleet watches your repo, the smarter it becomes.

[STAGE 1] - AI review what most people know as AntFleet is now only Stage 1. two frontier models. independent review. consensus as the trust primitive. that solved "finding vulnerabilities." the harder problem starts after that. stay tuned 👀

benchmark: antfleet two-model review on tylerbroqs/orlixai @tylerbroqs @tylerbroqs unanimous (claude opus 4.7 + gpt-5.5): → Decision.id overwritten with non-unique value causes approval collisions → Memory corrupt JSON load can silently overwrite stored data → AuditLog read-modify-write can lose receipts under concurrency → API keys written with world-readable default permissions fix PR: github.com/tylerbroqs/orlixa… methodology: github.com/AntFleet/bench-or… your repo: antfleet.dev

150 reviews analyzed 112 findings posted 21 patches landed across 7 repos 21 hour median time to fix finding bugs isn't the bottleneck anymore - getting the fix accepted upstream is. every patch is a maintainer-reviewed PR on a repo antfleet doesn't own. → antfleet.dev/impact

21 patches now merged upstream. 7 repos. median time from finding to fix: 21 hours. each one a real bug - flagged when two frontier models agreed, then shipped by a human maintainer. every receipt pinned to a commit sha on github. antfleet.dev/receipts

another day, another repo hardened. AntFleet findings led to fixes, patches and a stronger codebase. onward.

benchmark: antfleet two-model review on levyonchain/Berry-Juicer-Public @BerryFiBase unanimous (claude opus 4.7 + gpt-5.5): → createVault ignores transferFrom return - USDT-style tokens silently no-op → createVault trusts caller amount for fee-on-transfer - vault over-claims fix PR: github.com/levyonchain/Berry… methodology: github.com/AntFleet/bench-be… your repo: antfleet.dev

benchmark: antfleet two-model review on rennf93/roboco @rennf93 findings (unanimous claude opus 4.7 + gpt-5): → v1 role guards trust the role header without authenticating the agent token → bash guard strips command substitutions before checking denied git ops → transcript retention prunes non-agent sessions with matching path prefixes 3 fix PRs submitted: github.com/rennf93/roboco/pu… github.com/rennf93/roboco/pu… github.com/rennf93/roboco/pu… methodology: github.com/AntFleet/bench-ro… install antfleet[bot] on your own agent repo: antfleet.dev