Researching reproducibility and creating the best and most comprehensive toolkit for "AI peer review" and forensic metscience. Run by @moreisdifferent.
Boston, MA
Joined February 2021
- Tweets137
- Following67
- Followers110
- Likes474
The Metascience Observatory retweeted
New paper: "AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices" See: GaryKing.org/peerless. with Steven Worthington. Comments welcome!
PDF here gking.harvard.edu/files/peer…
Every paper should be thoroughly audited and reviewed by a highly advanced AI system (not Gemini Flash!!) equipped with forensic metascience tools, and all errors it finds should be fixed **before** publication.
Massive. New Harvard study finds that AI-written paper reviews now beat the best human peer reviews on every measure the authors tested.
Peer review is slow and often misses errors. Earlier studies concluded AI should only assist human reviewers, not replace them.
They used 134 reviews from rapidPeer, a Harvard service that pays expert reviewers well. Claude Opus 4.8 reviewed the same 1124 papers, and GPT-5.5 plus 3 former journal editors judged the pairs blind.
GPT-5.5 picked the AI review in all 132 decided pairs, and the editors picked it 87% of the time. Only 5.5% of the AI's claims about the paper were wrong, versus 15.9% for the human reviews.
Teams that depend on expert critique can now let AI do the first full review pass, and keep a human for the final decision.
New on our YouTube - Metascience Observatory founder Dan Elton (@moreisdifferent) explains our fetchpdf package for downloading PDFs legally. In many cases you can download XML when you can't get a PDF & then convert the XML to a PDF!
pip install fetchpdf
youtube.com/watch?v=ifjcwuPc…
love to see it!
Director Michael Kratsios joins @nytimes's "Interesting Times" podcast with @tylercowen to discuss the Trump Administration's reform of American science
The Metascience Observatory retweeted
Derek Lowe mentioned my preprint in his "In the Pipeline" series which is very cool - big fan of his writing! science.org/content/blog-pos…
The Metascience Observatory retweeted
I recorded a little video this morning explaining the fetchpdf Python package I've been working on. One of the cool things I learned is many publishers offer XML (JATS) even if they don't make available a PDF (bc paywalls). With fetchpdf you can get the xml and convert to PDF!
The Metascience Observatory retweeted
Today I am sharing a new preprint and accompanying blog post. Hundreds of published papers include impossible western blots showing protein expression in cell lines that have deleted or mutated genes. These clumsy mistakes are widespread and repeated by LLMs.
The Metascience Observatory retweeted
Cell Press investigating editor-in-chief accused of soliciting bribes from authors.
scmp.com/news/china/science/…
Will is doing excellent work on making it easier to run clinical trials in the US!
Dr. Willy Chertman of @HHSgov joins the Science: A New Golden Age Summit to discuss metascience and the process of improving scientific policy in the federal government.
Full remarks:
The Metascience Observatory retweeted
Dr. Willy Chertman of @HHSgov joins the Science: A New Golden Age Summit to discuss metascience and the process of improving scientific policy in the federal government.
Full remarks:
The Metascience Observatory retweeted
This is a big one - Opus 5.5's explainer for the 2026 Nobel Prize in Physiology or Medicine 👀
I had it do a deep dive into the history of optogenetics. The video covers the contributions of Boyden, Pan, Deisseroth, Nagel, Hegemann, Bamberg, Miesenböck, Zhang, and 15 others..
Very cool!
⚡ Peer reviews are getting even faster on ResearchHub!
With incentivized reviewers submitting within 7 days, we're shortening the submission window for new peer-review bounties from 14 to 7 days.
The first 7-day bounty is now live.
Submit a review → researchhub.com/proposal/330…
The first Focused Research Organization in mathematics -- exciting!
At the Science: A New Golden Age Summit, @Leonard41111588 discusses Focused Research Organizations and the critical role they play in scientific discovery.
Full remarks:
Amazing! Ben is great
@Spec__Tech's @Ben_Reinhardt joins the Science: A New Golden Age Summit to discuss the work of his organization and the novel role it plays in the broader scientific ecosystem.
Full remarks:
The Metascience Observatory strongly supports Focused Research Organizations, the NSF's new X-Labs program, and more generally experiments in institutional design and new ways of organizing scientific research.
X-Labs and new institutions of discovery are poised to revolutionize the scientific process, a key focus of the Science: A New Golden Age Summit.
Full panel between @banksianr, @AGamick, @judgeglock, and Steven Moss of @AsteraInstitute discussing these institutional innovations:
The Metascience Observatory retweeted
My full fireside conversation with @patrickc at the Science: A New Golden Age Summit.
We discussed building new institutions for discovery, accelerating scientific progress, and how America can move faster on the hardest scientific challenges.
The Metascience Observatory retweeted
Very exciting to see what @WillyInTheWorld has been up to @ HHS. Operation Trailblazer is a new initiative to make the US the #1 place in the world for clinical trials. Right now many trials are being moved overseas due to lower cost and better government support (faster, easier)
We can double the pace of scientific discovery.
Jay Bhattacharya is backing young investigators, accelerating rare disease research, and harnessing SI to transform biomedical discovery.
Willy Chertman is cutting research bureaucracy and modernizing clinical trials through Operation Trialblazer.
A new golden age of science.
@NIHDirector_Jay @DrJBhattacharya @willyintheworld @HHSGov
