🔬 Calling scientists across disciplines: help build SciVeri-Bench v1.0! After showcasing SciVeri-Bench v0.1 at the Frontier Data Summit hosted by @SnorkelAI, we’re now building v1.0 and inviting scientists from a wide range of fields to contribute scientific tasks. 🤔 What is SciVeri-Bench? SciVeri-Bench is a deploy-first, scientist-driven benchmark for evaluating AI agents’ scientific verification capabilities on open-ended, novel scientific tasks. 🎯 Why are we building it? - Stop to chasing maximing score in benchmark, rather we want to understand whether AI agents verify whether AI agent doing good science as domain scientist do - Domain scientists only verify whether the science is good or bad, while agents do a lot of execution and exploration, but we really want to show that scientists can make novel discoveries 📝 How are tasks collected? All task submissions and communication take place on OpenReview, so domain scientists can focus on proposing open-ended & novel scientific tasks and verifying agent outcomes. 🤝 What do scientists receive? - All costs of running the agents are covered by the SciVeri-Bench team. - Intellectual property rights to scientific discoveries made by AI agents belong to the scientist who proposed the task. - Co-authorship on the SciVeri-Bench paper. How can you contribute? (1) Task Proposal: Propose a scientific task for SciVeri-Bench. (2) Task Review: Review scientific tasks proposed by other scientists. Interested in joining us? 👉 Sign up: forms.gle/bv9KZK6mW6ubBJXZ9 🌐 Website: sciveri-bench.com/ 🙏 Huge thanks to our team, contributing scientists, advisors, and sponsor! AI researchers: @seungonekim @jinheonbaek @yoonholeee @JiayiiGeng Task proposers: 16 scientists who have joined so far AI advisors: @StevenDillmann @SoojungYang2 @AkariAsai @gneubig @wellecks @dongyeopkang Sponsor: openrefinery.ai/ We’re also looking for more sponsors and senior scientists! Note: This work does not relate to any work conducted at Amazon