Andrew Tate retweeted
Introducing QA-Bench, the world's first benchmark for how well AI agents catch bugs in real apps.
Today we're open-sourcing it so you can check our numbers yourself.
We ran Codex, Claude Code, and Devin through it alongside Mo, our AI QA engineer.
Mo found 3.1x as many bugs as Codex, 4.6x as many as Claude Code, and 10x as many as Devin.
Teams keep asking if they can just have Claude Code or Codex test their app, so we built QA-Bench to find out.
We took 4 popular open-source apps (Gitea, Metabase, Twenty, and Saleor) and had humans confirm 1,396 bugs across them. Every agent got the same apps, the same prompt, and one hour on each of 20 product areas.
# of bugs found, on average:
→ Mo: 269
→ Codex: 86
→ Claude Code: 59
→ Devin: 27
Mo also kept finding bugs as the hour went on. Doubling the time limit from 30 to 60 minutes got it 52% more bugs. Codex found only 16% more.
And it recorded a video of every bug it caught.
We built and graded QA-Bench ourselves, so don't take our word for it.
Every bug, run, and score is public. Run it on your own agent and send us your numbers👇
Results: momentic.ai/qa-bench
Full write-up: momentic.ai/blog/qa-bench
Dataset: huggingface.co/datasets/mome…
Harness and labels: github.com/momentic-ai/qa-be…
Andrew Tate retweeted
AI writes all your code. Does it still work?
That's the gap: coding agents are scaling output, but the same with bugs and incidents. The biggest bottleneck is QA: how can you make sure the code your agents just wrote doesn’t break everything?
The best QA are the ones who know everything about your product. Today, we give that knowledge to Momentic’s most capable agent.
Every Linear ticket, every PRD in Notion, every PR - Momentic knows all.
In just the past few weeks, our agents analyzed 70k+ test failures and created 600 tests. Achieving a 73% PR merge rate, and it’s growing fast.
We’re grateful for the trust of our customers, including Notion, Xero, Webflow, Retool, Runway, Bilt, and more.
Andrew Tate retweeted
If you are using Postgres, you already have a vector database. We built a RAG app that peps us up with inspirational quotes using LLama 3.1, Neon, and @OctoAICloud 👇
neon.tech/blog/building-a-ra…
Andrew Tate retweeted
Big @neondatabase news: We’re expanding to Azure, first with Azure regions in Neon console, deeper integration after that. This is backed by a strategic investment from M12 (MSFT VC arm)
Full story in thread:
Andrew Tate retweeted
We are @premierleague.
Andrew Tate retweeted
we teamed up with @CausalHQ to dashboard the generative ai market, breaking it down to show the fastest growing segments and companies.
the modeling and dataviz in causal helped bring the data to life. check it out below.
The folks at @sacrainc have the juiciest proprietary data on private tech companies, and we've worked with them to share some of it with you!
Here's a rundown of the Gen AI market, ft. @OpenAI, @AnthropicAI, @huggingface, and more 👇
go.causal.app/ai
Andrew Tate retweeted
The folks at @sacrainc have the juiciest proprietary data on private tech companies, and we've worked with them to share some of it with you!
Here's a rundown of the Gen AI market, ft. @OpenAI, @AnthropicAI, @huggingface, and more 👇
go.causal.app/ai
Andrew Tate retweeted
EFL GOAL OF THE SEASON, scored by Wes Burns of Ipswich. 🚜
Thirteen passes. Eight players. One human truth of a shot. 😱
Andrew Tate retweeted
i'm a big fan of negative visualisation: pick whatever terrifies you, imagine the worst possible scenario, and try to develop a plan
let's run the process with generative AI:
---
worst case scenario:
~ all content becomes (to some extent) AI generated
~ willingness-to-pay for content plummets
~ vast reduction in content roles/salaries
~ massive saturation of online content
~ massively diminished returns from SEO
~ [AGI develops, humanity squashed like bugs]
plan:
~ ride the wave. it's still early: there is leverage to be found through early adoption. become a vocal proponent of the tech; help others understand and use it; explore novel use cases; help companies get a headstart in the AI race
~ move up the value chain. content writing is a small portion of a great content marketer's total skill set. focus on the higher-leverage stuff: data analysis, distribution, research, technical SEO, editorial. think like a marketer, not "just" a writer
~ get strong where AI is weak. think second-order effects: if AI-generated content is everywhere, how could you differentiate? my bet: get good at thought leadership, industry analysis, data content
~ diversify. the future is very uncertain: be as flexible as possible and willing to explore new skills, interests, marketing channels. strive to be less wedded to your current situation - it's probably going to change
~ [buy a plot of land in rural scotland. learn to homestead. develop a taste for watery porridge and life without internet access]
Andrew Tate retweeted
What a turnaround for @Docker. Left for dead a few years back now reports they're well over $100M in ARR 🤯 @sacrainc has a great write up on the comeback. Never quit!
sacra.com/research/scott-joh…
Andrew Tate retweeted
jan is basically the best in the biz these days
lots of folks are surprised to learn that sacra is only 3 people with @SeekingN0rth
it's being long term in business, relationships etc that's enabled us to build this company
I can pick a winner
we hired the first ~10 team members at animalz exclusively off of craigslist
that's how i met @janerikasplund and now we've worked together over the last 7 years at @animalzco and now as co-founders at @sacrainc
Andrew Tate retweeted
Have you embraced the edge?
deno.com/blog/the-future-of-…
Andrew Tate retweeted
4 of 7 execs & vps at @dbt_labs worked together before at analytics pioneer rjmetrics
@jthandy ceo
@connormcarthur cto
@janessalantz marketing vp
erin vaughan cust success vp
a powerful advantage and rare in the increasingly professionalized & mercenary world of tech leadership
Andrew Tate retweeted
superpath gets shouted out by the @WSJ today
wsj.com/articles/how-to-netw…
Andrew Tate retweeted
Things you can do in 3️⃣0️⃣ days:
🏊 learn to swim
🦮 train your dog
...build a content strategy!? 🤯
But in 30 days? 🤔
Join us tomorrow on LinkedIn @ 10 AM EST (4 PM CEST) for this hot topic hosted by our own @TWadeinTO & @AnimalzCo 's @mrog91588.
linkedin.com/video/event/urn…
Andrew Tate retweeted
here’s how the Animalz crew helped vertical saas company @simple_legal increase their traffic by 515%
the secret: a rock-solid content refreshing strategy





















