I agree, I am actually researching how to best do that with agents: openreview.net/forum?id=xjCJ…

From my experience of doing post publication review: prepare to be ghosted

Benefit of the doubt for the reviewers: 1. Finding this kind of stuff pre-agentic-AI is hard 2. I think it is not the reviewers' job to check for these kinds of data integrity issues. Science is also based on trust. (and the consequences of misuse of this trust should be severe imo) Does Nature usually engage when you raise problems like this, or ghost you?

This was just a prompting issue? Did you use agentic AI? It could just download a model from Hugging Face and use it?

There is an obvious confounder here: age

Name one confounder of this study.

What if I get an idea for an analysis from an arxiv and use it in my paper? You propose I shouldn't cite?

vibe researchers be like

I want more AI in the review of code behind publications and less in the text/ideas.

First guy really looks a bit like a buff, younger version of Richard Sutton. But he does not seem to be doing Richard Sutton things in the pic...

Quick paper overview video: youtube.com/watch?v=QiP65fft… The paper: openreview.net/forum?id=xjCJ… Try the beta version of our tool for free at auditowl.ai. We'd be super interested in your feedback!

Code is the blind spot of peer review, and many serious errors that may invalidate a paper's conclusions can stay hidden in scientific repositories. We study whether coding agents can help make code review feasible for science. Check out our paper, where we apply agentic scientific code review to 100 NeurIPS 2025 papers (link below), and meet us at NeurIPS 2026! We find that codebases are often missing, incomplete, or contain methodological problems. We think that AI-auditing can make computational science more rigorous and reproducible! Thanks to my co-first @ferbsx, and co-authors Katharina Baum and Bernhard Renard.

This is similar to what we're doing: auditing the codebases behind computational scientific publications to check the claims. (TMLR, Neurips2026 Repro Track: openreview.net/forum?id=xjCJ…) However, one hurdle is the true-but-trivial rate. Even in good papers, an AI flags small methodological nuances as critical failures or inconsistencies. It is the same in software, where a "find bugs" prompt will never yield "no bugs", even for very good code and even after many fixing-iterations. How are you calibrating?

Bulk cancer cell line screens often span fewer than 1k cell lines. TCGA has like 10k patients. Massive datasets means high n, not high p. And single-cell datasets are just pseudoreplicated from a low number of humans. Which massive datasets are you talking about?

The problem is that the massive datasets are missing. If these companies produce the data, they will revolutionize the field. You have not seen good evidence because it takes time to build industrial labs and generate useful bio data at scale.

I get the academic side of this discussion is interesting, but I hope people aren't taking this seriously. A human is not conscious, or even aware. It is a bunch of atoms following the laws of physics to decide what comes next.

In principle, easy: zero-knowledge proof. Formalize in Lean, ZK-prove it checks, post the ZK proof + hash. Start polishing

It's a self fulfilling paper!

I usually use an easy method to get rid of much of this: Ask for a classification of flaws by severity mode: low, medium, and high severity. Then throw away the low- and maybe medium-severity stuff afterwards. Not perfect because it shifts the problem to the calibration of the severity modes, but all in all, I just get more actionable reports from this. The agents just love to work, so on great papers, they can go find the nitpicks, classify low severity, and I trash them, without wasting mental capacity on them