We introduce MetagenomicsBench, a benchmark for testing whether AI agents can analyze microbial communities and make scientific decisions from experimental data. 100 evaluations span five domains: (1) community structure (2) host-microbiome associations (3) microbial function (4) longitudinal dynamics (5) microbiome interventions Tasks draw on published datasets from human, animal, and environmental microbiomes. Metagenomic analysis requires interpreting microbial abundance alongside functional potential, host biology, and experimental design. Sequencing depth, taxonomic resolution, and confounding all affect observed results. Some concrete examples: - A higher abundance of antibiotic-resistance genes may reflect a change in community composition rather than a greater burden of acquired resistance. - A decline in one annotated butyrate pathway may not imply a loss of overall butyrate-synthesis capacity. - An association between bacterial abundance and reduced host-cell proliferation may reflect tissue context rather than a direct microbial effect. Agents received experimental data and worked without internet access. Across 34 model–harness configurations, Claude Sonnet 5.5 achieved the highest mean pass rate at 60.0% with both Claude Code and Pi, followed by Claude Opus 5 with Pi at 54.0%. GPT-6 Astra with OpenAI Codex was the highest-scoring non-Anthropic configuration at 49.7%. Performance also varied by scientific question. Host-microbiome associations were generally harder for the Opus and GPT families, while longitudinal dynamics and microbial function had higher pass rates across model families. Across judged failures, 47.1% involved unverified assumptions about task interpretation, thresholds, or metadata. Agents often produced plausible analyses while making choices that changed correct end biological conclusions. Manuscript: latch.bio/metagenomicsbench GH: github.com/latchbio/metageno… Blog: blog.latch.bio/p/metagenomic…

Hiring engineers to work on infrastructure for AI x biology. Our mission is to advance humanity's ability to engineer biology. We approach this by understanding and improving AI’s application to biology use cases such as therapeutics and biosecurity. We are a group of engineers, scientists, and operations specialists who are driven by improving capabilities and interpretability. Some of our work is showcased on benchmarks.bio. You will be working on problems including agent infrastructure and orchestration, ML infrastructure, model benchmarking at scale, full stack web development, developer and user experience, and process optimization. Infrastructure challenges span container orchestration, sandboxing, distributed filesystems, cloud architecture, and database systems. Our engineering team is in-person in San Francisco. We look for driven engineers who are excited to take ownership on hard problems and ship creative, useful solutions. Our process is typically completed within two weeks. • Round 1: Introduction • Round 2: Takehome Coding Project • Round 3: Technical • Round 4: Final/ 2-Day Paid Onsite Project • Offer Please email [email protected] if interested.

Results suggesting Kimi-K3 often tries to guess what the benchmark expects instead of solving the scientific problem directly. This behavior makes it slower, less reliable, and raises questions about how much its strong benchmark scores reflect genuine reasoning.