Three months since adopting software factories, we’re at 40% more PRs merged and 67% lower cost per PR. We got there by measuring agent quality, benchmarking models, and using self-improvement loops to make our factory better. Here's how you can do this too in Warp Factories:

Building Evals on Production Data: 🦄 AI That Works #79 xitter.cf/i/broadcasts/1oJMvNZWv…

Software engineers have been through a lot of change in the past two years, transitioning from writing code by hand to steering agents via local interactive prompting, the current paradigm. The next paradigm, software factories (where agents do more and more autonomous work) is particularly challenging. The metaphor of “factory” is kind of a bummer. It feels like your job as an engineer is either working on an assembly line, or building the assembly line that obviates the need for your talents. If you use the “AI teammate” metaphor (which I also dislike), then it feels like engineers are becoming managers of sycophantic and somewhat inept junior engineers. For folks who take pride in the craft of building, this all feels like commoditization of software creation. I disagree with this outlook. On our team, engineering involves two very important challenges today: building the product, and now, building the factory that builds the product. First, our engineering team is still responsible for the quality and usefulness of the product – this cannot be delegated to agents. Humans are uniquely positioned to know what to build, how the pieces should fit together, what the product experience ought to be, what constitutes good design. There are two caveats: Some types of product work that don’t need any human input at all and can be purely automated. (Fixing server crashes, simple bugs, dependency upgrades, etc.) The second caveat is that product engineering will work differently. Rather than engineers working locally with their own bespoke setups, they will hand off as much of the software lifecycle as possible to the factory. All of this work will be inherently visible and measurable, and that’s a feature, not a bug. Developers should expect to measure and iterate on other aspects of how they use the factory, including how much their agents ship and at what cost. Engineers could view this as a drag, but they also can view it as a game, and winning that game means shipping more for less, using an engineering mindset to improve. This brings me to the second job of the engineer, which is to design the factory itself. We call this factory engineering. It consists of measuring the performance of your factory. This needs to be done with an engineering mindset – you don’t adjust your factory on vibes; you measure, test, benchmark and then improve. At Warp we make it explicit that the expectation of engineers is not just to build products, but to improve factories. We envision both jobs as being the responsibility of all engineers to make it clear that factory engineering is now part of software engineering. On our team, the folks who like it most are senior engineers who love hard systems problems, who enjoy optimizing and debugging performance issues. It requires a lot of thought and testing and patience. We lean into it as a new skill to learn. Ultimately, this lets us ship better products more quickly, which has always been our mission.

Late to the party here, but @dexhorthy's /show-me skill is awesome. It gives a short, token-efficient overview of the changes in a pseudo-code format. Quick to generate and easily embedded in a PR description. It's composable too. Want more on InboxView? /show-me the InboxView. Want more on the preview batching mechanism? /show-me the preview cache

Here's the multi-agent system driving our software factory right now. Core pieces: - We say what to build (sometimes Linear issues, mostly Slack threads) - Triage agent researches and decides whether to implement or to ask questions (think /grill-me but smarter) - Implementation agent builds, maybe collaborating with us on a spec with a subagent first - Verification subagent tests the implementation agent's work e2e - Code review agent cycles with the implementation agent for a few rounds before human review - Human reviews and ships - Monitoring automation responds to alerts and files issues (our first step to closing the factory loop) There's room to simplify this. We're benchmarking whether we need separate agents for steps like triage vs. implementation, or if a single high-intelligence model can do most of it. The core loop remains the same though. Curious how this compares to factories others are deploying

Here's my take on what a software factory really is This is my talk from @aiDotEngineer NYC, and I decided to record a version for everyone to watch on X. tl;dr - software engineering is becoming factory engineering, and there's new work (especially factory benchmarking) that engineers should start focusing on.