Maybe they just disagree because each one found a different right model? If there were a discoverable right model for each prompt, independent routers would converge on it. They don’t. Across 59 cross-vendor pairs, the median agreement rate is zero. Most pairs never pick the same model. The bright squares are the exception that proves the rule: both sides collapsed onto the same cheap default. Orca’s quality setting and OpenRouter Auto-low agree on almost every prompt for that reason. Everything else is dark. 6/14

Look at the decisions and it gets clearer why. Most of them aren’t even trying to route. Five of the 17 settings send 90%+ of prompts to a single model. At that point the quality number is just that model’s number. It is not a router. It is a knob that selects a constant. Turn the knob on Orca or @OpenRouter Auto and you mostly just get a different constant. Cheap mode, quality mode — same trick, different default. 4/14

And yet the prize is real. The poor perforance of routers would not matter if there were nothing to collect. There is. This is the monthly model bill at one million questions, using this benchmark’s real prices. Always calling GPT-5.6 Sol: about $79,000, at an 87.5% best-answer rate. A router that, with hindsight, picked the cheapest model still good enough on each prompt: about $1,800, at 100%. In other words: Higher quality, about 98% less spend. That gap is the prize. Routing is supposed to collect it. At 85.3% best-answer rate, a mid-cost model like Luna gets you most of Sol’s quality for about $4,300. The orange bar is how much saving is still on the table (and there is an aditional +15% in quality). 2/14

Do LLM routers work? I have some good news for @AnthropicAI and @OpenAI: LLM routers suck. Routers are systems that, given a prompt, choose among a set of LLMs which is the best to answer. The goal is to identify the model that guarantees a high quality response at the lowest cost possible. The problem they promise to solve is simple: why call Gpt Astra on a prompt that a low-cost open source model like @deepseek_ai can answer just as well? A nightmare scenario for the big labs like OpenAi and Anthropic. Well, we applied the @AutobenchAI benchmarking methodology to 7 major routers. Result: always calling a mid-tier model like GPT-5.6 Luna beats every router we tested on both quality and cost. No exceptions. Check out the quality-vs-cost Pareto board. The light shaded zone is the bare minimum prize: better than a random pick and cheaper than a random pick. Luna is already sitting in it. Every router we tested is outside it. It's even worse. The routers that look cheap just collapsed onto one flash model and landed on the random line. The ones that look strong just bought an expensive model. 1/14 👇

Do LLM routers work? I have some good news for @AnthropicAI and @OpenAI: LLM routers suck. Routers are systems that, given a prompt, choose among a set of LLMs which is the best to answer. The goal is to identify the model that guarantees a high quality response at the lowest cost possible. The problem they promise to solve is simple: why call Gpt Astra on a prompt that a low-cost open source model like @deepseek_ai can answer just as well? A nightmare scenario for the big labs like OpenAi and Anthropic. Well, we applied the @AutobenchAI benchmarking methodology to 7 major routers. Result: always calling a mid-tier model like GPT-5.6 Luna beats every router we tested on both quality and cost. No exceptions. Check out the quality-vs-cost Pareto board. The light shaded zone is the bare minimum prize: better than a random pick and cheaper than a random pick. Luna is already sitting in it. Every router we tested is outside it. It's even worse. The routers that look cheap just collapsed onto one flash model and landed on the random line. The ones that look strong just bought an expensive model. 1/14 👇

Just a quick note on @openclaw after having built 11 agents in 2 multi-agent instances: Julia heads an accounting team of 5, together with Kate (reader), Fulvia (editor), Sophia (analyst), and Flavia (tester); Juno heads an executive staff of 6, together with Venus (comms), Minerva (analyst), Flora (editor), Diana (organizer), Vesta (Q&A). (the 2 teams have just started cooperating via their leaders). These agents are really amazing, but it can cost a lot of $ to get them working properly. Development, operations, and maintenance will dry your premium token budget very fast if you rely on SOTA models (and you don't want to default to budget models, because it takes one wrong call to f.up hard). Here is the key takeaway: smart LLM routing will become increasingly necessary to route the optimal model for each call, with the potential of saving up to 90% (that's because 90% of calls don't require SOTA). Tools like @clawrouter do a decent job, but they use hardcoded rules to route LLM calls. We need smarter light weight AI routers to do the job. This is what platforms like @BotScanner_AI and #AutoBench can help with. And we're working on it. Stay tuned...

Want to try a user-facing version of AutoBench to rank instantly LLM responses to your prompts? Try out our @BotScanner_AI. The platform uses AI to select the best LLM answers for each of your questions. There are still invitation codes with $3 of free credit available for those who want to test it. All you have to do is leave a comment here and follow @BotScanner_AI. We will send you the invitation code with the $3 of free credit. 5/5 end 🧵