Models currently lack the skills to achieve true software engineering automation. Emulated is a data company aiming to close the gaps in the software development loop. We're releasing the Software Development Automation Benchmark, a benchmark focused on model abilities beyond code generation to cover the full software development lifecycle. Read more here: emulated.so/sdab

Nearly all benchmarks focus on codegen-only tasks. Emulated's environments contain running services serving real traffic. We not only mock integrations with Slack, Linear, Sentry, Prometheus, and CI/CD, but we also emulate full cloud services like AWS, GCP, and Azure. These let us build incredibly rich tasks for distributed databases, training clusters, and network infrastructure!