
Ambassador @FactoryAI Building Annular, 300+ PRs a month with Droid and Devin Honest tests of AI coding tools and models
Joined December 2021
- Tweets787
- Following196
- Followers98
- Likes325
Droid saves me 15–20 hours a week on acceptance testing. Here's the issue workflow behind Annular, the app I'm building right now. More infos soon :)
1. Every finding becomes a GitHub issue: from a desk run, a code review or a test. Each one gets labels and a decision on the spot: does it block the current goal, or does it wait?
2. Simple ones get resolved by automation.
3. Anything that touches windows, input, displays or live sync does NOT close on green CI. Those PRs say "Refs", not "Closes". Learned that the hard way: two fixes closed on green CI and failed at the desk two days later.
4. For those, I sit down with Droid and explain the case and the pass rule up front, e.g. "fullscreen window on another Space must show up in the list, 10 out of 10 tries". Droid writes a script for exactly that case.
5. Droid runs on my PC and drives the Mac over SSH, so cross-device cases get measured on both machines in the same run. Tests that need a quiet desk wait until I'm away and abort on my first keystroke.
6. Every run leaves a record in the repo: build commit on each device, exact commands, numbers with n, verdict, and what didn't match. Raw evidence gets zipped to a backup with checksums.
7. I read the results. The issue only closes once that record is on main.
Same setup covers acceptance of every new feature, not just bugfixes: the pass cases are written before the code.
Thinking about turning this into an open source skill where you plug in your own devices and test cases. Would you use it?
@FactoryAI @droid
That is why I use Fusion at the moment.
It works great, it is faster and cheaper than just using the model alone.
Give Fusion a shot you won’t regret it.
Devin Fusion with Claude Opus 5.5 (max) and a GPT-6 Luna (high) sidekick scores 68 on the Coding Agent Index, 2 points above Claude Code with Opus 5.5 (max) at 31% lower cost per task
@cognition's Devin Fusion runs a frontier lead model alongside a lower-cost sidekick model. The two work in parallel with their own context and tools, with the lead keeping planning, delegation and final review. We benchmarked Devin Fusion CLI with Claude Opus 5.5 (max) + GPT-6 Luna (high).
It scores 68.1 on the Artificial Analysis Coding Agent Index, closely matched at the top with Claude Code with Claude Sonnet 5.5 (max) at 68.4 and Muse Code with Muse Wonder Full (max) at 68.0. Against Claude Code with Claude Opus 5.5 (max) at 66.0, it costs $9.01 vs. $13.04 per task and averages 55 vs. 65 minutes per task.
Across the three benchmarks, the Fusion configuration scores higher on DeepSWE v1.1 (72.6% vs. 68.4%) and SWE-Atlas-QnA (69.6% vs. 66.4%) and slightly lower on Terminal-Bench 4.0 (62.1% vs. 63.1%). Its results reflect both models working together, including their combined API cost. Costs are pay-per-token API costs, not subscription prices.
the difference between gemini 4 argon and opus 5.5 are two diffrent worlds
Mads retweeted
Environments remove the cold start problem associated with running agents in sandboxes. Happy building!
Promised a rerun, so here it is with Opus 5.5 High.
Same prompt in @DevinAI and @droid, no skills, no subagents, no files.
Devin: 43 min, $6.42
Droid: 38 min, $7.30
Droid was faster, Devin was cheaper and the cut views came out cleaner. Honestly both are crazy
Gave Gemini 4 Argon (high) the same prompt in @DevinAI and @droid: build a 3D anatomical heart
Devin: 8:11 min
Droid: ~19 min
I like Devins result more. Droid felt like it had latency issues all day today, so not sure if the long time comes from that, its not usual for me. Wanted to add Opus 5.5 too but in Droid it aborted after exactly 2407s of reasoning, 4th time today
planning more harness comparisons like this, let me know if you want more
turned this into a skill today, open source now xitter.cf/Laz01L/status/21115815…
Spent the whole day turning my desk acceptance workflow into a skill, built with @droid.
Green CI never meant it works at my desk. With Drayl the agent runs the live check on the real machines, and every run ends as pass, fail or interrupted, with the raw evidence saved. The PR side is in there too.
Its built on months of my own workflow. Found a bug or have an idea, open an issue. I read every single one.
github.com/mxdsl/drayl-skill…
Spent the whole day turning my desk acceptance workflow into a skill, built with @droid.
Green CI never meant it works at my desk. With Drayl the agent runs the live check on the real machines, and every run ends as pass, fail or interrupted, with the raw evidence saved. The PR side is in there too.
Its built on months of my own workflow. Found a bug or have an idea, open an issue. I read every single one.
github.com/mxdsl/drayl-skill…
this is the workflow from my post yesterday xitter.cf/Laz01L/status/21111627…
Droid saves me 15–20 hours a week on acceptance testing. Here's the issue workflow behind Annular, the app I'm building right now. More infos soon :)
1. Every finding becomes a GitHub issue: from a desk run, a code review or a test. Each one gets labels and a decision on the spot: does it block the current goal, or does it wait?
2. Simple ones get resolved by automation.
3. Anything that touches windows, input, displays or live sync does NOT close on green CI. Those PRs say "Refs", not "Closes". Learned that the hard way: two fixes closed on green CI and failed at the desk two days later.
4. For those, I sit down with Droid and explain the case and the pass rule up front, e.g. "fullscreen window on another Space must show up in the list, 10 out of 10 tries". Droid writes a script for exactly that case.
5. Droid runs on my PC and drives the Mac over SSH, so cross-device cases get measured on both machines in the same run. Tests that need a quiet desk wait until I'm away and abort on my first keystroke.
6. Every run leaves a record in the repo: build commit on each device, exact commands, numbers with n, verdict, and what didn't match. Raw evidence gets zipped to a backup with checksums.
7. I read the results. The issue only closes once that record is on main.
Same setup covers acceptance of every new feature, not just bugfixes: the pass cases are written before the code.
Thinking about turning this into an open source skill where you plug in your own devices and test cases. Would you use it?
@FactoryAI @droid
Happy to announce that I joined the @FactoryAI Guild 😁 Looking forward to building with Droid, giving feedback and helping make the product better
Wow thats Crazy, when my limits are back ill test how fast fast really is 😂
Let’s go @DevinAI
Two exciting updates for SWE-2: we're extending the **free** promo for pro/max/teams users to the end of the month and we're introducing SWE-2 Lightning as a paid option to run SWE-2 even faster.
Gave Gemini 4 Argon (high) the same prompt in @DevinAI and @droid: build a 3D anatomical heart
Devin: 8:11 min
Droid: ~19 min
I like Devins result more. Droid felt like it had latency issues all day today, so not sure if the long time comes from that, its not usual for me. Wanted to add Opus 5.5 too but in Droid it aborted after exactly 2407s of reasoning, 4th time today
planning more harness comparisons like this, let me know if you want more
Every week I had around 15 hours of physical acceptance tests at my desk, on Mac and on Windows.
Now Droid runs most of them. It controls my Mac over SSH and my Windows PC with Computer Use, both at the same time, and does the measurements too.
A few steps still need me, but it saves me around 15 hours a week.
Anyone else automating tests like this ?
Two weeks with @droid from @FactoryAI as my daily driver. Last week I shared my first impressions, now here is the full review
I got Droid Max for free, so I don't pay for it yet. But I will keep paying for it once the free period ends, because it gives me so much value that I can't see my stack without it anymore.
How I use Droid
I always plan with Droid because the planning is really good. I also run my tests and measurements with it. Devin still does my coding, debugging and CI, and Droid does the planning, testing the builds and the rest.
My setup is the same for missions and normal sessions. Opus 5.5 High as orchestrator, because it's by far the best model for orchestration and honestly the best model out right now. Sonnet 5.5 Medium as worker, and once Haiku 5.5 is in Droid I'll switch to it because it's cheaper. GPT 6.1 Sol as validator, because I want a model from a different provider for a second opinion, and it's good and cheap.
Usage and limits
There is the 5h limit and there is a weekly limit. And this is the part I really don't like. 100% of the weekly limit is 50% of the monthly limit. So if you use your full weekly limit every week, you only get two weeks of usage per month and then you have nothing for the other two weeks.
That shocked me when I found out. Now I have to plan with only 50% of my weekly limit per week, just so I still have usage at the end of the month. For me that's the biggest pain point of Droid right now.
The good part is the two usage pools. When my standard usage is gone I still have Droid Core. I use it for private stuff that has nothing to do with my project, where I don't need the absolute top model.
To be fair, my limit problem comes from how I use it. I only use Claude models, so everything runs on the standard usage. With the open weight models in Droid Core, for example DeepSeek 4.1 Flash for implementation and GLM 5.3 for debugging, you basically can't run out. They are not the best models, but they get the work done. With a setup like that you have more than enough usage for the month.
Mission Mode
Mission Mode is crazy for my long tasks and I really like it.
A while ago I posted that a mission had been running for 21 hours flawlessly. In the end it ran 24.5 hours, with 3 PRs and 81 commits, without any problems. It was a huge fix block in my codebase, because physical acceptance tests had found some bugs in my program. All of them are fixed now.
The mission cost me around 25% of my weekly limit. For that amount of work it's okay for me.
Acceptance tests with SSH and Computer Use
This is my favorite new workflow. For my project I have to do physical acceptance tests at my desk, on Mac and on Windows. That was around 15 hours of work every week.
Now Droid controls my Mac over SSH and my Windows PC with Computer Use and automates a big part of these tests. It also does the measurements perfectly, and the tests can run in parallel on Mac and Windows at the same time. A few steps still don't work without me, but it saves me around 15 hours of work per week.
If you do tests like that, I really recommend trying to automate them with Computer Use.
The harness
This is honestly what makes Droid feel so different for me. The agents don't just start building. They first make sure they understood what I want and that they are doing the right thing, and only then they touch the code. That's also why the planning is so good.
When something is unclear they ask me first instead of guessing.
The models also follow instructions a lot better in Droid than in other harnesses. GPT models that like to do their own thing somewhere else actually listen here.
And it stays like that when I switch models. Claude, GPT or the open weight models in Droid Core, they all feel solid in it. A lot of harnesses only work well with one model family.
On long tasks it keeps the goal in mind and doesn't slowly forget what I told it at the start. My 24.5 hour mission is the best example for that.
I also like that I can let it plan first without touching any files, and the code only gets written after I approve the plan.
The desktop app
Honestly the nicest and best desktop app I've used so far. It barely uses any RAM or performance even with a mission running, other IDEs use way more just sitting idle.
Voice mode
Voice mode is great for brainstorming and I use it all the time. Talking through a big prompt takes me a few minutes, typing the same thing out would take way longer.
Even long voice prompts work fine and the text is there almost right away.
It also gets technical terms and model names right most of the time.
Two things bother me though. When I use the shortcut I can't hit send directly. And once the standard usage is gone voice mode stops working, even though I still have Droid Core usage. More on both below.
What I don't like so far
The weekly limit is 50% of the monthly limit, see the usage part. Remove it or match it to the monthly limit.
The 5h limit just gets in the way, I don't like it in any subscription. I would rather have no 5h limit at all.
In missions you can't see which model a worker uses. If you change the model in the mission selection, every agent chat suddenly shows that model. A clear model label per worker would fix it.
Mission planning uses the default agent delegation instead of the agents I already picked. It should just use my selection.
Voice mode stops after the standard usage is gone. It should keep working as long as I have Droid Core usage left.
With the voice shortcut I can't send directly. A send button or key for it would fix it, probably an easy one.
There is no proper PR section, so I still use Devin for my PRs and the after review. A PR tab next to new session, Mission Control and automations would be great.
All of these still exist as of today.
Final thoughts
Droid has its flaws, like every app. But what matters a lot to me is that the devs listen to their community here on X and actually take feedback. It became one of my favorite apps and I'm very, very satisfied with it.













