
co-founder & president @LatchBio, intelligence infrastructure for biology
San Francisco, CA
Joined January 2020
- Tweets369
- Following579
- Followers716
- Likes3.8K
Kyle Giffin retweeted
Lux family co @LatchBio
a tremendous opportunity to join for fast growing AI/Bio future...
We are growing at a breakneck clip and have ambitious engineering goals in therapeutics. Expanding our MTS team across chemistry, assay dev, translation and the clinic:
- MTS, Chemistry
- MTS, Medicinal Chemistry
- MTS, Molecular Design
- MTS, Experimental Biology + Assay Development
- MTS, Translational Science
- MTS, Clinical Development
- MTS, Clinical Operations
- MTS, Clinical Data + Statistical Programming
- MTS, Biostatistics + Trial Design
High-intensity, founder-type roles. Translate a deep understanding of a biological domain into a roadmap of benchmarks, then build and manage your own scientific team. Our MTS have built companies, run drug programs and scaled complex technical projects.
In person in Mission Bay, San Francisco. We keep meetings to a minimum. Tight deadlines and customer commitments sometimes require late hours.
350K–500K+ OTE + equity. Visa sponsorship and relocation assistance. Our process is typically completed within two weeks.
Frontier AI labs/pharma use our benchmarks to assess biological capabilities and safeguards. Our work appears in research announcements and model cards, including:
- GDPx biology: gdpx.semianalysis.com/domain…
- Fireworks SII: fireworks.ai/specialized-int…
- Gemini 4 Hydrogen: blog.google/innovation-and-a…
- Claude Fable 5.1/Mythos 5.1 system card: www-cdn.anthropic.com/0339e6…
- Claude Opus 5.5 system card: www-cdn.anthropic.com/fc1b44…
- Grok 4.7 model card: media.x.ai/v1/website/card4p…
- xAI Biosecurity report: x.ai/news/biosafety-at-the-f…
Apply at jobs.ashbyhq.com/latchbio. Or send an email to kenny[at]latch[dot]bio.
Kyle Giffin retweeted
Introducing AIR-O, a model that decodes what a person is attending to directly from brain scans.
Using only noninvasive brain data, we can generate an image from scratch in real time that depicts where a person is focusing their attention.
This is the first of a new generation of algorithms that can read the shape of thoughts directly from brain activity.
Kyle Giffin retweeted
Occasionally I take a look at #general and wonder who all these people are. Latch has grown so much so fast and is only growing faster. Throughout this growth it has stayed a place where a love for science and an awareness of how science can improve lives has stayed constant.
To support these efforts, we're expanding the MTS team across chemistry, assay development, and clinical science, with roles open in:
- MTS, Chemistry
- MTS, Medicinal Chemistry
- MTS, Molecular Design
- MTS, Experimental Biology + Assay Development
- MTS, Translational Science
- MTS, Clinical Development // Clinical Operations // Clinical Data // Trial Design
Some Latch shill on where we've had an impact:
- GDPx biology: gdpx.semianalysis.com/domain…
- Fireworks SII: fireworks.ai/specialized-int…
- Gemini 4 Hydrogen: blog.google/innovation-and-a…
- Claude Fable 5.1/Mythos 5.1 system card: www-cdn.anthropic.com/0339e6…
- Claude Opus 5.5 system card: www-cdn.anthropic.com/fc1b44…
- Grok 4.7 model card: media.x.ai/v1/website/card4p…
- xAI Biosecurity report: x.ai/news/biosafety-at-the-f…
Apply at jobs.ashbyhq.com/latchbio. DMs open
Kyle Giffin retweeted
We are growing at a breakneck clip and have ambitious engineering goals in therapeutics. Expanding our MTS team across chemistry, assay dev, translation and the clinic:
- MTS, Chemistry
- MTS, Medicinal Chemistry
- MTS, Molecular Design
- MTS, Experimental Biology + Assay Development
- MTS, Translational Science
- MTS, Clinical Development
- MTS, Clinical Operations
- MTS, Clinical Data + Statistical Programming
- MTS, Biostatistics + Trial Design
High-intensity, founder-type roles. Translate a deep understanding of a biological domain into a roadmap of benchmarks, then build and manage your own scientific team. Our MTS have built companies, run drug programs and scaled complex technical projects.
In person in Mission Bay, San Francisco. We keep meetings to a minimum. Tight deadlines and customer commitments sometimes require late hours.
350K–500K+ OTE + equity. Visa sponsorship and relocation assistance. Our process is typically completed within two weeks.
Frontier AI labs/pharma use our benchmarks to assess biological capabilities and safeguards. Our work appears in research announcements and model cards, including:
- GDPx biology: gdpx.semianalysis.com/domain…
- Fireworks SII: fireworks.ai/specialized-int…
- Gemini 4 Hydrogen: blog.google/innovation-and-a…
- Claude Fable 5.1/Mythos 5.1 system card: www-cdn.anthropic.com/0339e6…
- Claude Opus 5.5 system card: www-cdn.anthropic.com/fc1b44…
- Grok 4.7 model card: media.x.ai/v1/website/card4p…
- xAI Biosecurity report: x.ai/news/biosafety-at-the-f…
Apply at jobs.ashbyhq.com/latchbio. Or send an email to kenny[at]latch[dot]bio.
Kyle Giffin retweeted
a new way to contribute to scientific progress is emerging
until recently, there were effectively two paths for scientists:
- stay in academia: get to work on more or less whatever you want, but be resource limited and underpaid
- go into industry: be well resourced and well paid, but have to work on narrow, research programs that increase shareholder value
there’s now a third option: contributing to improving the scientific capabilities of AI. this has some of the best aspects of both traditional industry and academia
- you get freedom to pursue the research that interests you
- your work is very high leverage, helping accelerate research of all scientists who use AI
- you are resourced and paid much better than academia
interestingly, what makes it possible for this third path to exist is that the frontier labs actually mean it when they say they want to “cure all disease in 5 years.” whether or not you think this is plausible, there is strong conviction and abundant capital trying to make it happen
i’ve joined @latchbio and i’m very excited to be part of this new wave of scientists striving to ensure AI can contribute to revolutionary scientific discovery
@latchbio we build benchmarks and evaluations that frontier labs and pharma use to measure and improve their models
i am building out a team focused on Aging & Longevity whose goal is to give AI (1) a deep understanding of aging biology, (2) the ability to develop next generation biomarkers of aging, and (3) the capability to discover and develop anti-aging interventions.
if you’re interested in helping accelerate research across the aging & longevity ecosystem my DMs are open
latch.bio/careers/ai-biologi…
Kyle Giffin retweeted
New agentic benchmark covering vaccines and other treatments for respiratory viruses.
Prior work in respiratory surveillance focused on detecting and tracking threats. BioSecBench-RespiratoryVirus-Countermeasures (will work on our names) extends task distribution to understanding + treating disease.
64 evaluations cover the following groups:
- viral structure and function
- immunity and antigenic variation
- host response and disease biology
- antiviral activity and resistance
- clinical and population outcomes.
There are three public tasks for inspection spanning antibody occupancy, neutralization of Omicron and vaccine safety reporting.
Gemini 4 Carbon with Pi had the highest overall pass rate at 30.2%, followed by Claude Sonnet 5.5 with Pi at 18.8% and Claude Fable 5.2 with Claude Code at 17.7%. Refusals and errors counted as failures.
Note: Low scores were actually not solely due to refusals. eg. clinical and population outcome tasks had no refusals
Strong work from @harm0n , @evanseeyave, Gage Moreno, Yuyi Liu, and the biosecurity team.
- Leaderboard: benchmarks.bio/respvirus-cou…
- Paper: latch.bio/biosecbench-respvi…
- GitHub: github.com/latchbio/biosecbe…
- Blog: blog.latch.bio/p/can-ai-agen…
Kyle Giffin retweeted
Proud to see scientists from our biologics development team featured in the Gemini 4 Hydrogen launch.
If you're interested in working on frontier biology evaluation, where engineers and scientists work side by side to improve these systems, shoot me an email at kenny[at]latch[dot]bio.
blog.google/innovation-and-a…
Kyle Giffin retweeted
Proud to see scientists from our biologics development team featured in the Gemini 4 Hydrogen launch.
If you're interested in working on frontier biology evaluation, where engineers and scientists work side by side to improve these systems, shoot me an email at kenny[at]latch[dot]bio.
blog.google/innovation-and-a…
Kyle Giffin retweeted
Introducing BioSecBench-Structure, a structural reasoning benchmark for biosecurity tasks.
We measure agentic interpretation of molecular structures to assess pathogen targets and support the development of vaccines, antibodies, and drugs.
Ex:
- Coronavirus spike conformations. Nine cryo-EM structures capture receptor-binding domains in open and closed states, which affect receptor engagement and antibody access. Agents determine how many domains are open in each structure
- Inhibitor screening against a bacterial pathogen. Two structures of a protein involved in pilus assembly are being considered as templates for screening inhibitors. Agents compare the protein’s subunits to determine whether screening can use a single structural template or needs an ensemble.
- Candidate drugs targeting a Pseudomonas aeruginosa enzyme. Two compounds differ by a small chemical modification. Agents examine crystal structures and electron-density maps to determine whether the modification increases the number of occupied binding sites before it is carried into the next design round.
The benchmark comprises 58 evaluations built from crystal and cryo-EM structures, predicted models, and experimental measurements. Thirty-three tasks concern threat characterization, 22 concern countermeasure design, and three concern threat detection.
We tested 20 models in 35 model–harness configurations, with three offline runs per evaluation. With refusals enabled, Claude Fable 5.2 with Claude Code had the highest overall pass rate at 43.7%, followed by Fable 5.2 with Pi at 41.4% and Claude Opus 5.5 with Claude Code at 31.0%. Refusals and errors counted as failures.
The leaderboard also includes Anthropic’s safety-filter-exempt evaluation variants (LSVP). Fable 5.2 LSVP with Pi reached 52.3%, and Opus 5.5 LSVP with Pi reached 39.7%.
Refusals concentrated on defensive countermeasure work. In reruns with refusals disabled, countermeasure pass rates increased by 14 percentage points for Fable 5.2 and 15 for Opus 5.5, with harnesses pooled. Their threat-characterization scores barely changed.
We think structural reasoning will be an important capability for agents supporting biosecurity. Important work from John Wang, Nick Keul, Jonathan Felix, Qian Xu, and team.
Benchmarks[dot]bio: benchmarks.bio/biosec-struct…
Paper: latch.bio/biosecbench-struct…
GitHub: github.com/latchbio/biosecbe…
Kyle Giffin retweeted
Microscopy evals are quite fun and its often easy to quickly understand the spirit of the task
Microscopy lets scientists literally see the biological world. We think giving agents the same ability will be an important source of progress in science.
Today we introduce TissueMicroscopyBench, our first release in a new microscopy series covering tissues, cells, and in vitro systems.
The benchmark comprises 49 evaluations built from four public imaging datasets. Agents must locate structures, trace their boundaries, reason through three-dimensional images, and follow changes over time.
Some of the work:
- Colorectal cancer sections stained with H&E and multiplexed immunofluorescence. Tasks include outlining mucosa, fat, and nerves, and locating evidence of tumor invasion.
- Liver metastasis biopsies taken before and after treatment. Agents compare tumor compartments, PD-L1 staining, and T-cell-rich regions across the images.
- Light-sheet recordings of living organoids. Overlapping structures and densely packed nuclei require agents to distinguish individual objects and draw their boundaries.
- A regenerating leg recorded in three dimensions over time. Tasks examine the injury site, local cell divisions, tissue retraction, and the sequence of regeneration.
Compared with our previous benchmarks, these evals are especially visually striking. The rollouts are dense, interleaving image manipulation (including hand-rolled segmentation and CV in Python) with reasoning.
Some slides contain billions of pixels. Agents have to decide where to look, examine the relevant regions at sufficient resolution, and return precise outlines, tracks, or volumes. These are scored against expert annotations. Every task was also solved independently by a human as a check on task quality.
We tested 22 models in 30 model–harness configurations, with three attempts per task. Claude Fable 5.2 (LSVP) with Pi had the highest pass rate at 34.0%, passing 50 of 147 attempts. Timeouts and invalid answers counted as failures. Twenty-three of the 49 tasks were never solved by any configuration.
We found that agents can be shockingly bad at localization, often finding the general region of interest but drawing boundaries that included surrounding tissue or random patches of empty space. Small structures and changes across time were particularly difficult.
We think visual perception and spatial reasoning will become core capabilities for scientific agents. It’s also just a super fun space to think creatively about evals (h/t Ian Diks). Excited to release more in this series and watch agents improve here.
- Blog: blog.latch.bio/p/tissue-micr…
- GitHub: github.com/latchbio/tissuemi…
- Leaderboard: benchmarks.bio/tissuemicrosc…
Kyle Giffin retweeted
Microscopy lets scientists literally see the biological world. We think giving agents the same ability will be an important source of progress in science.
Today we introduce TissueMicroscopyBench, our first release in a new microscopy series covering tissues, cells, and in vitro systems.
The benchmark comprises 49 evaluations built from four public imaging datasets. Agents must locate structures, trace their boundaries, reason through three-dimensional images, and follow changes over time.
Some of the work:
- Colorectal cancer sections stained with H&E and multiplexed immunofluorescence. Tasks include outlining mucosa, fat, and nerves, and locating evidence of tumor invasion.
- Liver metastasis biopsies taken before and after treatment. Agents compare tumor compartments, PD-L1 staining, and T-cell-rich regions across the images.
- Light-sheet recordings of living organoids. Overlapping structures and densely packed nuclei require agents to distinguish individual objects and draw their boundaries.
- A regenerating leg recorded in three dimensions over time. Tasks examine the injury site, local cell divisions, tissue retraction, and the sequence of regeneration.
Compared with our previous benchmarks, these evals are especially visually striking. The rollouts are dense, interleaving image manipulation (including hand-rolled segmentation and CV in Python) with reasoning.
Some slides contain billions of pixels. Agents have to decide where to look, examine the relevant regions at sufficient resolution, and return precise outlines, tracks, or volumes. These are scored against expert annotations. Every task was also solved independently by a human as a check on task quality.
We tested 22 models in 30 model–harness configurations, with three attempts per task. Claude Fable 5.2 (LSVP) with Pi had the highest pass rate at 34.0%, passing 50 of 147 attempts. Timeouts and invalid answers counted as failures. Twenty-three of the 49 tasks were never solved by any configuration.
We found that agents can be shockingly bad at localization, often finding the general region of interest but drawing boundaries that included surrounding tissue or random patches of empty space. Small structures and changes across time were particularly difficult.
We think visual perception and spatial reasoning will become core capabilities for scientific agents. It’s also just a super fun space to think creatively about evals (h/t Ian Diks). Excited to release more in this series and watch agents improve here.
- Blog: blog.latch.bio/p/tissue-micr…
- GitHub: github.com/latchbio/tissuemi…
- Leaderboard: benchmarks.bio/tissuemicrosc…
Kyle Giffin retweeted
Microscopy lets scientists literally see the biological world. We think giving agents the same ability will be an important source of progress in science.
Today we introduce TissueMicroscopyBench, our first release in a new microscopy series covering tissues, cells, and in vitro systems.
The benchmark comprises 49 evaluations built from four public imaging datasets. Agents must locate structures, trace their boundaries, reason through three-dimensional images, and follow changes over time.
Some of the work:
- Colorectal cancer sections stained with H&E and multiplexed immunofluorescence. Tasks include outlining mucosa, fat, and nerves, and locating evidence of tumor invasion.
- Liver metastasis biopsies taken before and after treatment. Agents compare tumor compartments, PD-L1 staining, and T-cell-rich regions across the images.
- Light-sheet recordings of living organoids. Overlapping structures and densely packed nuclei require agents to distinguish individual objects and draw their boundaries.
- A regenerating leg recorded in three dimensions over time. Tasks examine the injury site, local cell divisions, tissue retraction, and the sequence of regeneration.
Compared with our previous benchmarks, these evals are especially visually striking. The rollouts are dense, interleaving image manipulation (including hand-rolled segmentation and CV in Python) with reasoning.
Some slides contain billions of pixels. Agents have to decide where to look, examine the relevant regions at sufficient resolution, and return precise outlines, tracks, or volumes. These are scored against expert annotations. Every task was also solved independently by a human as a check on task quality.
We tested 22 models in 30 model–harness configurations, with three attempts per task. Claude Fable 5.2 (LSVP) with Pi had the highest pass rate at 34.0%, passing 50 of 147 attempts. Timeouts and invalid answers counted as failures. Twenty-three of the 49 tasks were never solved by any configuration.
We found that agents can be shockingly bad at localization, often finding the general region of interest but drawing boundaries that included surrounding tissue or random patches of empty space. Small structures and changes across time were particularly difficult.
We think visual perception and spatial reasoning will become core capabilities for scientific agents. It’s also just a super fun space to think creatively about evals (h/t Ian Diks). Excited to release more in this series and watch agents improve here.
- Blog: blog.latch.bio/p/tissue-micr…
- GitHub: github.com/latchbio/tissuemi…
- Leaderboard: benchmarks.bio/tissuemicrosc…
Kyle Giffin retweeted
Rare diseases collectively affect an estimated 300 million people worldwide. For patients and families, reaching a diagnosis can take years of referrals and testing. One place AI agents could help is in interpreting each new sequencing result or functional assay alongside the patient’s symptoms, family history, and earlier findings.
We introduce RareDiseaseBench, a benchmark of 35 diagnostic workups reconstructed from published rare disease cases. Each agent starts with the patient’s presentation and part of the medical record. Over one or more visits, it must decide which tests to request, explain why, analyze the files it receives, and propose a diagnosis and management plan.
Five cases from the benchmark:
- A 13-year-old boy has growth restriction, hearing loss, and severe chilblains. An MN1 variant explains several developmental findings. Long-read sequencing reveals a translocation disrupting SAMHD1 that accounts for his inflammatory symptoms.
- A 21-year-old man has heterotaxy with right atrial isomerism. A variant just beyond the standard ZIC3 transcript falls within an intron of an alternative transcript, causing 57 extra bases to be spliced into the RNA.
- A boy develops progressive scalp hair loss while his eyebrows and body hair are preserved. A scalp biopsy shows miniaturized follicles, and sequencing reveals a 357-kb duplication spanning TAB3 and neighboring genes.
- A 16-year-old boy with typical growth has a language disorder, learning disability, and autism. He carries two inherited variants in RNU2-2, a small nuclear RNA gene outside the coverage of his exome sequencing.
- A boy with delayed motor milestones and enlarged calves has a creatine kinase level of 10,100 U/L. At a second visit, an outside lab reports a DMD inversion. Targeted long reads show duplicated segments containing exons 1 and 2 joined head to head.
Each configuration made three attempts on each of the 35 cases. Gemini 4 Argon with Pi passed 46 of 105 attempts (43.8%). Claude Opus 5.5 with Claude Code and GPT-6 Astra with Pi each passed 35 of 105 (33.3%). A pass required the diagnosis and management plan to meet every criterion in the case-specific rubric. Timeouts counted as failures.
Among the 14 models that passed at least one case, 2,097 runs failed. Data acquisition and diagnostic workup was the most frequent primary failure category (36%). In 15% of all analyzed failures, the agent failed to pursue the type of data the case required. In another 12%, it omitted a necessary non-sequencing investigation, such as a tissue stain, imaging study, or laboratory test.
Paper: latch.bio/raredisease
Leaderboard: benchmarks.bio/raredisease
Blog: blog.latch.bio/p/raredisease…
We think rare disease diagnosis is one of the areas where AI systems have the most potential to make rapid progress and directly benefit patients over the next few years.
Incredible work from @chriswzou and team.
Kyle Giffin retweeted
Our Antibody Discovery Benchmark asked whether agents could make the decisions that produce a promising candidate. Now we’re following what happens on the way to its first human dose.
We introduce TxBench–Antibody Development, a verifiable benchmark of 60 evaluations across seven stages of therapeutic antibody development.
Choosing a molecule opens another set of scientific questions. Can it be manufactured and stored reliably? Did a process change alter its properties? How should animal exposure, immune responses and safety findings inform the decision to enter humans?
Some concrete examples:
- A manufacturing change may affect the drug’s quality even when its sequence stays the same.
- Low antibody exposure in an animal may reflect a dosing artifact rather than rapid clearance.
- Choosing a first human dose requires reconciling pharmacology and safety evidence across experimental systems.
Agents received records from real development programs, drawn from public patents, regulatory reviews, published studies and trial records. Each evaluation asks for a decision and its quantitative justification, graded against a fixed answer key.
Across 34 model–harness configurations, Grok 4.7 with Grok Build led at 39.4%, followed by Opus 5 with Claude Code at 37.2% and Sonnet 5.5 with Pi at 36.7%. Averaged across configurations, toxicology and first-dose decisions each passed only about 13% of the time.
We also connected five stages into one development program, asking agents to carry four candidate antibodies through successive decisions while building on their earlier answers. Only Opus 5.5 completed the full sequence, in one of five runs.
Development makes the dependencies between scientific decisions explicit. An interpretation of an animal study can change which candidate advances and the dose proposed for humans. We’re extending our benchmarks along that path, toward the increasingly consequential decisions required to turn a promising molecule into a medicine.
Manuscript: latch.bio/txbench-ab-dev
Leaderboard: benchmarks.bio/txbench-abdev
Blog: blog.latch.bio/p/txbenchanti…
Kyle Giffin retweeted
Stoked to be SemiAnalysis’s exclusive biology partner for GDPx. Very early days for AI in biology.
Recommend digging into the trajectories: gdpx.semianalysis.com/domain…
Read more about our contributions: gdpx.semianalysis.com/domain…
Exceptional folks working on this project @arjunomics @maxkan @AndrewMegalaa
Kyle Giffin retweeted
Incredibly excited that Latch was part of this (awesome work from @arjunomics ). Incredibly exciting too to see how biology has become one of the most talked about applications of LLMs.
While it's clear that LLMs are improving very quickly, work like GDPx makes it clear that their knowledge and practice of it are very spiky. Not only that, but as LLM capabilities in biology improve we still need better tools and benchmarks for safe deployment. Some of the tasks included in GDPx originated from biosecurity work at Latch.
Latch is hiring extremely aggressively across science, engineering, and ops roles. If you're interested in being part of more initiatives like this one, check out jobs.ashbyhq.com/LatchBio
Kyle Giffin retweeted
In science, model capabilities have no obvious ceiling. Economic utility can keep increasing as impact jumps from improvements in constituent steps of research programs to outcomes legible to consumers: cures and enhancements.
Single evaluations might take months, millions of dollars and large teams of the world’s best talent. Benchmarking is one of the few durable components of the AI x science ecosystem, requiring compounding technical knowledge and human creativity.
As these markets are being built with unknown growth potential, it’s important to use your imagination.
Kyle Giffin retweeted
Our multi-omics exercise paper is out in Cell!
We all know exercise is good for us. But what actually happens at the molecular level when we exercise regularly?
In this study, we set out to understand this by analyzing bulk RNA-seq data from 22 mouse tissues after 8 weeks of voluntary exercise. We also generated single-nucleus RNA-seq, single-nucleus genome-wide DNA methylation, proteomics, and metabolomics data to investigate tissue-, sex-, and time-of-day-dependent responses.
Some interesting findings:
1. We observed widespread molecular changes across multiple organs, with responses varying by tissue, sex, and time of day.
2. By profiling ~36,000 single-nucleus methylomes alongside ~268,000 single-nucleus transcriptomes, we identified cell-type-specific DNA methylation changes associated with muscle adaptation that bulk measurements cannot resolve.
3. We found that males and females respond differently to exercise, with distinct transcriptional and DNA methylation signatures.
4. We identified a 51-gene signature that distinguishes exercise-trained from sedentary states across independent datasets, including human and rodent studies.
5. We identified 180 differentially expressed transcription factors and 4,633 predicted target genes, revealing cell-type-specific regulatory networks associated with muscle adaptation to exercise.
Beyond these findings, we've made the datasets available as a resource for the research community to explore the molecular mechanisms of exercise.
It was a fantastic collaborative effort across multiple labs and disciplines. Thanks to everyone involved in making this work possible!
Paper: doi.org/10.1016/j.cell.2026.…
#MultiOmics #Exercise #Epigenetics #SingleCell
Kyle Giffin retweeted
Lux family co @LatchBio launched OncologyBench...
AI x Cancer...
We introduce OncologyBench, a verifiable benchmark of 25 multistep oncology evaluations that tests the ability of multi-agent systems to produce clinically relevant decisions in cancer.
25 evaluations, built on 92 component tasks, span six stages of an oncology journey:
(1) Translational and basic research
(2) Risk and detection
(3) Diagnosis and tumor characterization
(4) Treatment
(5) Response and resistance
(6) Outcome
Evaluations draw on published datasets, case studies, public databases, and patient data graciously donated to Latch for the express purpose of benchmarking.
Clinical and translational decisions in oncology are rarely made by one person working from one dataset. They are built on several analyses, each carried out by a specialist on a given data type, whose reports are then weighed by someone who does not typically repeat the underlying work.
OncologyBench is built to benchmark agents working in that setting. Agents received experimental data and worked without internet access. The strongest of the 31 model-harness combinations we tested were Claude Opus 5, Opus 5.5 and Sonnet 5.5, passing 29 to 40% of attempts, followed by Kimi K3 at 24%.
We found that the models with the strong interpretation and analysis skills still showed weakness in reasoning. Two-thirds of the attempts that got every analysis right still failed the final decision. In a separate experiment we found decision and answer quality varied depending on the persona of the question asker.
An example evaluation:
A 76-year-old woman has undergone resection of stage II mucinous cecal cancer. The evaluation asks whether she should receive adjuvant chemotherapy.
Five specialist agents analyze her tumor mutations, pharmacogenomics, gene expression, T-cell infiltration and clinical record. A decision agent receives their reports without access to the underlying data.
The tumor is MSI-high, which predicts no benefit from 5-FU monotherapy. CDX2 loss and low T-cell infiltration do not establish that chemotherapy would improve her outcome. DPYD*2A is detected in 6 of 58 tumor-sequencing reads, but these data neither confirm nor exclude an inherited variant associated with severe 5-FU toxicity. BRAF V600E and MLH1 silencing support a sporadic origin.
The expected recommendation is observation, with missing pathology requested and germline DPYD testing before any future 5-FU.
---
Incredible work from researchers + former biotech founders at Latch. Gratitude to the patients that donated their data. More to come.
Kyle Giffin retweeted
We introduce MetagenomicsBench, a benchmark for testing whether AI agents can analyze microbial communities and make scientific decisions from experimental data.
100 evaluations span five domains:
(1) community structure
(2) host-microbiome associations
(3) microbial function
(4) longitudinal dynamics
(5) microbiome interventions
Tasks draw on published datasets from human, animal, and environmental microbiomes.
Metagenomic analysis requires interpreting microbial abundance alongside functional potential, host biology, and experimental design. Sequencing depth, taxonomic resolution, and confounding all affect observed results.
Some concrete examples:
- A higher abundance of antibiotic-resistance genes may reflect a change in community composition rather than a greater burden of acquired resistance.
- A decline in one annotated butyrate pathway may not imply a loss of overall butyrate-synthesis capacity.
- An association between bacterial abundance and reduced host-cell proliferation may reflect tissue context rather than a direct microbial effect.
Agents received experimental data and worked without internet access. Across 34 model–harness configurations, Claude Sonnet 5.5 achieved the highest mean pass rate at 60.0% with both Claude Code and Pi, followed by Claude Opus 5 with Pi at 54.0%. GPT-6 Astra with OpenAI Codex was the highest-scoring non-Anthropic configuration at 49.7%.
Performance also varied by scientific question. Host-microbiome associations were generally harder for the Opus and GPT families, while longitudinal dynamics and microbial function had higher pass rates across model families.
Across judged failures, 47.1% involved unverified assumptions about task interpretation, thresholds, or metadata. Agents often produced plausible analyses while making choices that changed correct end biological conclusions.
Manuscript: latch.bio/metagenomicsbench
GH: github.com/latchbio/metageno…
Blog: blog.latch.bio/p/metagenomic…




















