Chryssa Aliferi

Final day of the DevBox series. So far it was about how we built it. This one is about how we know it works. For months, we judged changes by running a few tasks and eyeballing the PRs. With a system that gives different output every run, that tells you very little. So we built golden-task evals. Take a real Linear issue with a good merged PR. Reset the repos to the commit before it, so the agent can’t see the answer. Run DevBox and capture everything: duration, tokens, retries, infra failures, PRs opened. LLM judges compare the diff with the real one on requirements, bugs, tests, maintainability, scope creep and merge safety. Current release vs candidate, several runs per task. If the candidate wins, it becomes the baseline. DevBox is good at well-specified work that can run in background. A scaling fix, a CLI command mirroring an existing one, a client from an API spec. It’s bad at vague tickets, big features with no spec, and anything needing live third-party APIs. One “consolidate these flags” ticket became an 18-file PR. We closed it. Output quality tracks ticket quality almost exactly. Writing a good spec is now one of the most valuable skills on the team. In September, about 3 in 10 PRs across our GitHub org came from DevBox. Over the last 3 months, its top two categories were bug fixes (340) and features (296). Merge rate on decided PRs went from 57% in May to 76% in September. A lot changed in between, so I won’t credit one fix. That’s why we built the evals. About half the non-release commits in DevBox’s own repos were written by DevBox. It’s for routine background work. For hard problems, our engineers still pair with Claude Code or Codex. In last 6 months, what majorly helped: 1. Orchestration in code, reasoning in the model 2. Make the environment safe, not the model 3. Load context deterministically 4. Measure before you believe Thanks for following along!

Day 9 of building DevBox, our internal coding agent. The biggest jump in quality didn’t come from a better model. It came from fixing how the agent reviews its own work. DevBox already reviewed its diff before opening a PR. But humans were still catching bugs it should have caught. So we looked at the traces. The reviewer ran for about 2 minutes, never executed any code, and came back with a short list of opinions. And the fixer was quietly rejecting medium-severity findings by default, even when the planner had flagged the same risk at the start. We did try a model upgrade. It caused session errors and we rolled it back. That made it pretty clear the problem was the workflow. So we rebuilt review. Each reviewer is now a separate agent that never sees the implementer’s reasoning. It only gets the task, the plan and the diff. The implementer already believes its code is right, so you want a reviewer who wasn’t in the room. The harness runs reviewers in parallel. A general code review always runs. The planner adds others depending on the change: a bug hunter that traces risky paths to an actual failure, a bird’s-eye reviewer that looks at callers, dependencies and the end goal, plus security and code quality. A config tweak gets one reviewer. An auth change might get security and the bug hunter too. Reviewers get a scratch space where they can run tests and repro scripts. If they say something is broken, they have to show it. A merge agent then dedupes the findings (same root cause, worded differently by different reviewers) and writes one report. Some rules live in code, not in a model. Critical/high findings without evidence get downgraded, there’s a cap of 8 findings per repo, and the harness decides the final verdict. Then a fresh fixer agent, with the implementer’s design notes, goes through each finding and either fixes it or writes down why not. The loop has a limit, and anything left over goes into the PR description so the human reviewer sees it. Our takeaway: when agent quality is bad, look at the process before you swap the model. A smarter model doing a 2-minute review without running anything is still a 2-minute review without running anything. Tomorrow, the last one: how we measure all of this.

Day 8 of building DevBox, our internal coding agent. A PR that says "tests pass" doesn't earn much trust. Today: how we got the agent to prove its change works. Most of our interesting changes span services: a backend change, a UI change, a config change. Unit tests passing told reviewers little. They'd pull the branch, spin up the stack locally and click around. Sometimes that cost more than writing it themselves. We already had "mini", an internal tool that boots a minimal @RudderStack stack locally. So we made it run headless inside the sandbox. The agent picks the services a change needs, brings them up, seeds data and checks health. During planning, a verification-spec step decides whether end-to-end verification even applies, and which surface: API, webapp or live events. Most tasks don't need it. The ones that do get a concrete plan instead of "test it somehow". For UI changes the agent drives the real web app with Playwright and captures screenshots. Those get hosted and embedded inline in the PR body, so reviewers see the result without checking anything out. Key rule: verification is honest. If the stack doesn't come up, the PR says "skipped because <reason>". Never a faked pass. A missing screenshot is much better than a misleading one. We also added live previews: a running copy of the changed stack a reviewer can open over VPN, with the link posted to Slack. Cold starts could take ~5 minutes, so the harness warms previews up ahead of time and re-warms them if the pod restarts. What didn't work yet: generic seed data that didn't exercise the change, missing service config, and screenshots of loading spinners. Evidence is only useful when it's specific to the change. Where it's going: we're turning verification into its own session-based service with an MCP interface, so any agent (DevBox, our assistant or an engineer's local Claude Code) can spin up a stack and verify against it. Tomorrow: fixing self-review.

Day 7 of building DevBox, our internal coding agent. We started on Codex CLI. Over the summer we moved stage after stage to Deep Agents. Here's why, and what it taught us about who should own orchestration. Codex CLI got us to a working product fast. It's a great coding agent. But we were driving it as a subprocess, and as the workflow grew we kept losing control of three things: what it read, which session it resumed, and whether each step actually ran. Examples: • Resume "last session" could pick a sub-agent's session when several shared a folder • Prompt updates weren't picked up on resumed rounds • An OOM-killed child process left the pod in an error state we kept polling We patched around them: resume by marker instead of "last", reinstall the system prompt on resume, detect dead pods. Each fix worked, but we were building an orchestrator out of workarounds around someone else's CLI. So we flipped it: Python owns orchestration, the model only does reasoning. With Deep Agents (LangGraph underneath), each repo gets its own in-process agent with its own context, checkpointed to SQLite, so it resumes exactly where it stopped. What moved from prompts into code: • Which files and skills each agent starts with • Retry and backoff, including rate limits with jitter so parallel workers don't retry in lockstep • Schemas for every handoff, enforced with structured output It wasn't a big-bang migration. Each piece shipped behind an experimental flag we could turn on per task: implementation first, then distillation and review. Old and new ran side by side until the new path won. One win, previously Codex agents would often ignore the instructions to start per repo subagents unless reasoning was set to high. This decision now moved to Python orchestration. Deepagent planner reports which repos are involved, harness owns starting implementation agents. Consistent behavior without unnessary reasoning tokens. Lesson: let the model reason and let code orchestrate. Every time we asked a model to "remember to do X next", it eventually didn't. Every time we moved X into code, it stopped being a problem. Tomorrow: proving the change actually works.

Day 6 of building DevBox, our internal coding agent. Our v1 coding agent opened a PR and disappeared. Nobody merges a PR like that. Here's how it became something you can have a conversation with. The shift: from a pipeline to an event loop. Every sandbox runs a daemon with an inbox. The control plane drops events into it: new assignment, PR comment, review, CI failure. The daemon handles them one at a time and writes events back. Communication between control plane and agent is just files: an inbox of events in, an append-only events log out. Boring, easy to inspect, and it survives restarts. Every task leaves an artifact trail you can archive and replay. PR comments, v1: act on every comment. Merge rate improved, but it was risky. A reviewer discussing an approach with a colleague could find the agent had already "fixed" it. v2: the agent acts only when explicitly "@"mentioned. But when it does act, it fetches the whole PR discussion (every thread, new since its last round) so it doesn't answer one comment and ignore the three that explain it.

Day 5 of building DevBox, our internal coding agent. Context was a bigger bottleneck than the model. Here's how we handle it across a multi-repo codebase. A "simple" RudderStack change can touch a backend service, the web app and a config repo. Planning question #1 is always: which repos, and why? Early on the agent sometimes picked wrong, and one task landed in the wrong repos entirely. So we built a shared knowledge graph: a repo that describes our architecture, services and how they connect. It's mounted into every sandbox. Planning starts with an orientation step: discover → evaluate → adapt. When the graph disagrees with the code, the agent doesn't silently trust either. It logs a "knowledge drift" note. Those notes are how we find out the docs/graph are stale.

Day 4 of building DevBox, our internal coding agent. This is an important one. We assumed the agent will eventually be prompt-injected. What can it do then? It reads tickets, PR comments, docs and web search results, any of which could carry instructions. And it runs arbitrary code as root inside its sandbox. So the question is "what can it reach". Our rule: no real secret ever enters the sandbox. All egress goes through a proxy. Inside the sandbox, the GitHub token, model API key and search key are literally dummy strings. The proxy swaps in real credentials on the way out. Thanks to the team behind iron proxy (@mslipper). Even git works this way. Bootstrap rewrites every GitHub URL to HTTPS and adds a header with a dummy token. The proxy replaces it with a short-lived GitHub App token. A compromised agent can dump its whole environment and find nothing useful. Same for web search: the agent calls a credential-free MCP URL and the proxy injects the API key. We also allowlist which search tools the agent can use. We found out the hard way that a typo in the tool names silently allowed everything.

Day 3 of building DevBox, our internal coding agent. Yesterday: where it runs. Today: getting code into the sandbox quickly, and what happens when sandboxes misbehave. Our tasks touch several big repos. Cloning them at the start of every task burned time before the agent did anything useful. First idea: one shared volume with a git worktree per task. That broke fast. Concurrent sandboxes all wrote to the same .git directory and stepped on each other's locks and refs. Shared mutable state + parallel agents = pain. What we run now: A Kubernetes CronJob clones every repo onto an EBS volume every 30 min and takes a VolumeSnapshot. When a task starts, the control plane mints a fresh PVC from the latest snapshot and mounts it. Attaching takes seconds. Nice properties: • Every task gets its own copy, so no shared .git • If a clone fails, the job doesn't publish, and the last good snapshot stays live • Adding a repo is a one-line ConfigMap change, no redeploy • Old snapshots get pruned to the last 3 30 minutes is still stale for a fast-moving repo. So the harness also syncs each planned repo from origin before planning and again before committing. The snapshot makes it fast, and the sync keeps it correct.

Day 2 of how we built DevBox, our internal coding agent. This one is about where the agent actually runs. 🧵 The agent runs arbitrary code as root, so we needed real isolation. We also needed Docker + Compose for integration tests, our own network policy, and to run on our own Kubernetes, near our data. We looked at hosted sandbox providers like E2B, Daytona and Modal. For some, Docker support was still experimental. We also wanted strict path-based control over network egress and flexible credential injection that supports rotation. E2B was the closest fit but did not meet that last requirement. So we picked OpenSandbox, self-hosted on our EKS cluster. Each sandbox is a pod running inside a Kata Containers microVM. That VM is the security boundary. Inside it, the agent gets full access: no approval prompts, no nested sandboxing. We trust the box.

Over the last 6 months, we built DevBox, an internal coding agent at RudderStack. Delegate a Linear ticket to it and it plans, writes the code, runs tests, reviews its own work and opens a draft PR. Over the next weeks I'll share how we built it, one problem at a time 🧵 1/10