Andrew Tate (@ajtatey) header
Andrew Tate (@ajtatey) profile picture

co-founder & ceo @ https://xitter.cf/t.co/Scv8zarhQI

Frederick, MD
Joined April 2008
Photo by Andrew Tate (@ajtatey)Photo by Andrew Tate (@ajtatey)Photo by Andrew Tate (@ajtatey)Photo by Andrew Tate (@ajtatey)Photo by Andrew Tate (@ajtatey)Photo by Andrew Tate (@ajtatey)Photo by Andrew Tate (@ajtatey)Photo by Andrew Tate (@ajtatey)Photo by Andrew Tate (@ajtatey)Photo by Andrew Tate (@ajtatey)
Andrew Tate retweeted
Introducing QA-Bench, the world's first benchmark for how well AI agents catch bugs in real apps. Today we're open-sourcing it so you can check our numbers yourself. We ran Codex, Claude Code, and Devin through it alongside Mo, our AI QA engineer. Mo found 3.1x as many bugs as Codex, 4.6x as many as Claude Code, and 10x as many as Devin. Teams keep asking if they can just have Claude Code or Codex test their app, so we built QA-Bench to find out. We took 4 popular open-source apps (Gitea, Metabase, Twenty, and Saleor) and had humans confirm 1,396 bugs across them. Every agent got the same apps, the same prompt, and one hour on each of 20 product areas. # of bugs found, on average: → Mo: 269 → Codex: 86 → Claude Code: 59 → Devin: 27 Mo also kept finding bugs as the hour went on. Doubling the time limit from 30 to 60 minutes got it 52% more bugs. Codex found only 16% more. And it recorded a video of every bug it caught. We built and graded QA-Bench ourselves, so don't take our word for it. Every bug, run, and score is public. Run it on your own agent and send us your numbers👇 Results: momentic.ai/qa-bench Full write-up: momentic.ai/blog/qa-bench Dataset: huggingface.co/datasets/mome… Harness and labels: github.com/momentic-ai/qa-be…
32
7
2
48
4,051
Andrew Tate retweeted
AI writes all your code. Does it still work? That's the gap: coding agents are scaling output, but the same with bugs and incidents. The biggest bottleneck is QA: how can you make sure the code your agents just wrote doesn’t break everything? The best QA are the ones who know everything about your product. Today, we give that knowledge to Momentic’s most capable agent. Every Linear ticket, every PRD in Notion, every PR - Momentic knows all. In just the past few weeks, our agents analyzed 70k+ test failures and created 600 tests. Achieving a 73% PR merge rate, and it’s growing fast. We’re grateful for the trust of our customers, including Notion, Xero, Webflow, Retool, Runway, Bilt, and more.
80
41
35
409
974,741
shape is the new delve
30
Andrew Tate retweeted
If you are using Postgres, you already have a vector database. We built a RAG app that peps us up with inspirational quotes using LLama 3.1, Neon, and @OctoAICloud 👇 neon.tech/blog/building-a-ra…
2
13
1,339
Big @neondatabase news: We’re expanding to Azure, first with Azure regions in Neon console, deeper integration after that. This is backed by a strategic investment from M12 (MSFT VC arm) Full story in thread:
11
26
8
169
33,406
2,751
12,037
4,200
94,017
10,128,434
4
36
3
386
17,152
Andrew Tate retweeted
we teamed up with @CausalHQ to dashboard the generative ai market, breaking it down to show the fastest growing segments and companies. the modeling and dataviz in causal helped bring the data to life. check it out below.
The folks at @sacrainc have the juiciest proprietary data on private tech companies, and we've worked with them to share some of it with you! Here's a rundown of the Gen AI market, ft. @OpenAI, @AnthropicAI, @huggingface, and more 👇 go.causal.app/ai
6
19
2,759
Andrew Tate retweeted
The folks at @sacrainc have the juiciest proprietary data on private tech companies, and we've worked with them to share some of it with you! Here's a rundown of the Gen AI market, ft. @OpenAI, @AnthropicAI, @huggingface, and more 👇 go.causal.app/ai
2
10
2
39
8,783
EFL GOAL OF THE SEASON, scored by Wes Burns of Ipswich. 🚜 Thirteen passes. Eight players. One human truth of a shot. 😱
25
234
49
3,181
270,789
Andrew Tate retweeted
i'm a big fan of negative visualisation: pick whatever terrifies you, imagine the worst possible scenario, and try to develop a plan let's run the process with generative AI: --- worst case scenario: ~ all content becomes (to some extent) AI generated ~ willingness-to-pay for content plummets ~ vast reduction in content roles/salaries ~ massive saturation of online content ~ massively diminished returns from SEO ~ [AGI develops, humanity squashed like bugs] plan: ~ ride the wave. it's still early: there is leverage to be found through early adoption. become a vocal proponent of the tech; help others understand and use it; explore novel use cases; help companies get a headstart in the AI race ~ move up the value chain. content writing is a small portion of a great content marketer's total skill set. focus on the higher-leverage stuff: data analysis, distribution, research, technical SEO, editorial. think like a marketer, not "just" a writer ~ get strong where AI is weak. think second-order effects: if AI-generated content is everywhere, how could you differentiate? my bet: get good at thought leadership, industry analysis, data content ~ diversify. the future is very uncertain: be as flexible as possible and willing to explore new skills, interests, marketing channels. strive to be less wedded to your current situation - it's probably going to change ~ [buy a plot of land in rural scotland. learn to homestead. develop a taste for watery porridge and life without internet access]
4
3
2
33
6,237
What a turnaround for @Docker. Left for dead a few years back now reports they're well over $100M in ARR 🤯 @sacrainc has a great write up on the comeback. Never quit! sacra.com/research/scott-joh…
1
5
1
11
3,502
Andrew Tate retweeted
jan is basically the best in the biz these days lots of folks are surprised to learn that sacra is only 3 people with @SeekingN0rth it's being long term in business, relationships etc that's enabled us to build this company
2
1
15
1,616
I can pick a winner
we hired the first ~10 team members at animalz exclusively off of craigslist that's how i met @janerikasplund and now we've worked together over the last 7 years at @animalzco and now as co-founders at @sacrainc
1
2
1,103
Could murder a pizza rn.
5
1,714
Andrew Tate retweeted
4 of 7 execs & vps at @dbt_labs worked together before at analytics pioneer rjmetrics @jthandy ceo @connormcarthur cto @janessalantz marketing vp erin vaughan cust success vp a powerful advantage and rare in the increasingly professionalized & mercenary world of tech leadership
2
9
6
77
Andrew Tate retweeted
superpath gets shouted out by the @WSJ today wsj.com/articles/how-to-netw…
5
3
44
Andrew Tate retweeted
Things you can do in 3️⃣0️⃣  days: 🏊 learn to swim 🦮 train your dog ...build a content strategy!? 🤯 But in 30 days? 🤔 Join us tomorrow on LinkedIn @ 10 AM EST (4 PM CEST) for this hot topic hosted by our own @TWadeinTO & @AnimalzCo 's @mrog91588. linkedin.com/video/event/urn…
1
12
2
14
Andrew Tate retweeted
here’s how the Animalz crew helped vertical saas company @simple_legal increase their traffic by 515% the secret: a rock-solid content refreshing strategy
2
19
2
56