Tensorlake’s sandbox scheduler uses a replicated state machine (RSM) to replicate cluster and sandbox metadata across multiple machines for high availability. Recently, we increased our throughput for scheduling sandboxes by 12x. RSMs linearize writes across nodes to provide strong consistency. As we scaled our sandbox infrastructure, we found that creating a burst of 100k sandboxes concurrently competed for the replication pipeline with scheduler updates that place sandboxes on dataplanes. Worse, dataplane updates which inform users that their sandboxes have started were jammed behind create and placement messages across nodes. To solve this problem, we borrowed ideas from massively distributed databases such as TiKV and CockroachDB and implemented Multi-Raft. Among other use cases, these databases use Multi-Raft to replicate databases independently of one another. We now have three replication lanes across scheduler nodes: 1. Intent to create/delete/update sandboxes, driven by users 2. Scheduler placement updates 3. Dataplane updates Multi-Raft concurrently transfers messages across these three lanes. Both (1) and (3) drive the scheduler, so the scheduler is never blocked waiting for more messages.

Tensorlake is leading the @computesdk benchmarks. We have been focused on building a best-in-class sandbox infrastructure platform for agents, pairing high performance with the flexibility to support everything from agent harnesses and RL environment simulation to stateful VMs for coding agents. We built a cluster scheduler optimized for stateful sandboxes that can make thousands of placement decisions in under 5 milliseconds. It achieves this performance using copy-on-write indexes and replicated state machines. The remaining ~200–250 milliseconds are spent issuing a command to the data plane, creating a disk, and bootstrapping the rest of the VM infrastructure. Tensorlake is also exceptionally fast at resuming suspended sandboxes, waking from sleep in 1–2 seconds. This enables coding agents to install dependencies, do their work, suspend to release compute resources, and resume quickly to start new tasks without reinstalling dependencies or re-downloading artifacts. Tensorlake also includes a high-performance sandbox file system, which speeds up compilation for CI and coding workflows, and improves simulation performance for environments that run databases and similar services inside the sandbox.