New essay: ML seems to promise discovery without understanding, but this is fool's gold that has led to a reproducibility crisis in ML-based science. aisnakeoil.com/p/scientists-… (with @sayashk). In 2021 we compiled evidence that an error called leakage is pervasive in ML models across scientific fields. In our most recent survey the number of affected fields has climbed to 30. reproducible.cs.princeton.ed… Leakage is only one of many reasons for reproducibility failures. There are widespread shortcomings in every step of ML-based science, from data collection to preprocessing and reporting results. reforms.cs.princeton.edu/app… Root causes The reasons for pre-ML replication crises, such as publication bias, also apply to ML. But a new and important reason for the poor quality of ML-based science is pervasive hype, resulting in the lack of a skeptical mindset among researchers, which is a cornerstone of good scientific practice. We’ve observed that when researchers have overoptimistic expectations, and their ML model performs poorly, they assume that they did something wrong and tweak the model, when in fact they should strongly consider the possibility that they have run up against inherent limits to predictability. Conversely, they tend to be credulous when their model performs well, when in fact they should be on high alert for leakage or other flaws. And if the model performs better than expected, they assume that it has discovered patterns in the data that no human could have thought of, and the myth of AI as an alien intelligence makes this explanation seem readily plausible. This is a feedback loop. Overoptimism fuels flawed research which further misleads other researchers in the field about what they should and shouldn’t expect AI to be able to do. aisnakeoil.com/p/scientists-… Glimmers of hope Researchers should in principle be able to download a paper’s code and data, review it, and check whether they can reproduce the reported results. And the vast majority of errors can be avoided if the researchers know what to look out for. So we think that the problem can be greatly mitigated by a culture change where researchers systematically exercise more care in their work and reproducibility studies are incentivized. We have led a few efforts to change this. First, our leakage paper has had an impact. Many researchers have used it to avoid leakage in their own work and to check previously published work. reproducible.cs.princeton.ed… Beyond leakage, we led a group of 19 researchers across computer science, data science, social sciences, mathematics, and biomedical research to develop the REFORMS checklist for ML-based science. It is a 32-item checklist that can help researchers catch eight kinds of common pitfalls in ML-based science. It was recently published in Science Advances. Of course, checklists by themselves won’t help if there isn’t a culture change, but based on the reception so far, we are cautiously optimistic. reforms.cs.princeton.edu/ A tool, not a revolution Of course, AI can be a useful tool for scientists. The key word is tool. AI is not a revolution. It is not a replacement for human understanding — to think so is to miss the point of science. AI does not offer a shortcut to the hard work and frustration inherent to research. AI is not an oracle and cannot see the future. We are at an interesting moment in the history of science. Look at these graphs showing the adoption of AI in various fields (by Duede et al. arxiv.org/abs/2405.15828): These hockey stick graphs are not good news. They should be terrifying. Adopting AI requires changes to scientific epistemology. No scientific field has the capacity to accomplish this on a timescale of a couple of years. This is not what happens when a tool or method is adopted organically. It happens when scientists jump on a trend to get funding. Given the level of hype, scientists don’t need additional incentives to adopt AI. That means AI-for-science funding programs are probably making things worse. We doubt the avalanche of flawed research can be stopped, but if at least a fraction of AI-for-science funding were diverted to better training, critical inquiry, meta-science, reproducibility, and other quality-control efforts, the havoc can be minimized. aisnakeoil.com/p/scientists-… P. S. Our book AI Snake Oil is all about how to separate real AI advances from hype. It's now available to preorder (and we're told preordering makes a big difference to the book's success). amazon.com/Snake-Oil-Artific… bookshop.org/p/books/ai-snak…