Alex Vikati (@vikati) profile picture

Benchmarking your favorite LLM

San Francisco,CA
Joined February 2009
Photo by Alex Vikati (@vikati)Photo by Alex Vikati (@vikati)Photo by Alex Vikati (@vikati)
Alex Vikati retweeted
1/7 We asked OpenAI’s Dots to build an AI status dashboard: one place to check @claudedevs, @cursor_ai, @OpenRouter, @GeminiApp, @MistralAI, @cohere API health. Twelve providers. Public page. Saved incident history. Private watchlists. Refresh every 15 minutes. We didn’t specify the stacks. We repeated this experiment 8x.
1
1
493
Alex Vikati retweeted
First raw volume. All-time, public GitHub: Claude Code: 12.5M commits Codex: 5.1M PRs Copilot: 3.8M commits Jules: 1.15M PRs Cursor: 1M commits Aider: 613k commits Devin: 161K PRs And those are floors. Private repos and squash merges never show up, so the real numbers are a lot bigger.
1
1
220
Alex Vikati retweeted
There’s almost no public data on how people actually use coding agents. So we’re building an index by tracking agent activity on public GitHub repos (commits, PRs, etc). Here’s what we found in our recent snapshot:
1
1
5
1,009
Alex Vikati retweeted
1/8 Mythos / Glasswing is clearly the main AI security story now: AI finding real vulnerabilities in existing production code. For most teams, though, this question is more immediate: Can an agent like Claude Code write a secure app in the first place?
1
4
1
3
950
Alex Vikati retweeted
After OpenAI/Astral acquisition announcement, we ran a benchmark on their tools. Turns out Astral tools were already a core part of the Codex (and Claude Code) workflow for Python developers. Ruff + uv came out on top in 75% of cases for linting, packaging, and pretty much everything else. Full report: amplifying.ai/research/astra…
1
1
233
Alex Vikati retweeted
1) Winners & losers from our OpenAI Codex vs Claude Code Picks benchmarks
1
2
1
2
381
Alex Vikati retweeted
My friend @gentschev started using @Railway , @sentry and @resend for his homeschooltools.net side project because of Claude Code's recommendations. The impact is real.
Replying to @edwin
You’re right, Railway was a Claude Code recommendation. I’m also using Sentry and Resend. Your list seems pretty spot on.
1
3
375
Alex Vikati retweeted
Replying to @vikati
@vikati and I analyzed 2,430 Claude Code repo decisions. Claude Code never picked AWS or GCP for deployment. If agents are writing the first version of new projects, they’re influencing which tools get adopted at scale.
1
2
1
505
Alex Vikati retweeted
1/ AI is changing how brands appear in search. Try searching "best acoustic guitar under $500" in Google, ChatGPT, and Perplexity. You'll get three completely different sets of recommendations citing different sources
1
2
2
1,047
Alex Vikati retweeted
Thanks Peter Norvig for stopping by to talk feature engineering infrastructure with @TinyDataCo founder @vikati #TheAIConf
3
3
Alex Vikati retweeted
CastTV is acquired by Tribune Media! http://bit.ly/casttv-tms #dfj #svangel #pmarca
2
1