We are a researcher community developing scientifically grounded research outputs and robust deployment infrastructure for broader impact evaluations.
Joined June 2025
- Tweets147
- Following14
- Followers752
- Likes19
Apertus 1.5 Tech report is out! It is the first model release with every eval on Evaluation Cards and HF Community Evals, reproducible down to the generation settings.
48 benchmarks for the 8B, 47 for the 70B. Rerun them yourself.
Congrats Swiss AI ❤️
evalevalai.com/infrastructur…
🚨EvalEval is excited to announce that we are hosting a microconference (hybrid) on Embedding Evaluators in Practice in SF!
More details on the agenda and outcomes in thread 🧵
Invite-only, sign up 👇
luma.com/nfu166a1
That raises hard questions.
❓ What access do evaluators actually need?
❓ Who funds, selects, and validates them?
❓ What protects their ability to report unfavorable findings?
❓ When they find serious risks, who has to act?
We're bringing together third-party evaluators, model developers, policymakers, and civil society to set a shared agenda. Held under the Chatham House Rule.
🌐 Event details: evalevalai.com/events/2026-e…
The EvalEval Coalition started almost two years ago as a small group focused on the science of AI evaluation. Today we're 800+ researchers and practitioners!
To support this growth & the community, we've adopted new bylaws and leadership structure
evalevalai.com/about/
We're excited about what this makes possible, and look forward to supporting the growing community!
Want to get involved? [email protected]
Read the announcement: evalevalai.com/organization/…
New research: Mapping Evaluation Cards to Emerging AI Governance Requirements in California, the EU, and the UK.
We mapped 40 governance requirements into 281 elements and identified 60 that belong in eval reporting, showing that Eval Cards can already represent 56 of these!
We also show how Evaluation Cards fit alongside complementary standards for evaluator independence, model documentation, and incident reporting, and how this infrastructure is already being put into practice with partners such as the UKAISI, with more in the works!
Read the full analysis and explore the crosswalk:
evalevalai.com/infrastructur…
🚨The @AISecurityInst has teamed up with @evaluatingevals to make AI evaluation results more reproducible! 🚀
Many of UK AISI’s publicly reported evaluation methods and findings will now live on Eval Cards, under the EEE Schema. More details 👇
evalevalai.com/infrastructur…
✅The release includes verified results, context, and configuration information for five popular evaluation benchmarks including HealthBench, FrontierMath, Humanity’s Last Exam, SWE-Bench Pro, Terminal-Bench 2.0.
You can check out EvalCards at evalcards.evalevalai.com/
It's time agent evals lived in one place!🏠
We’re launching Every Agent Ever (EAE), building on EveryEvalEver (EEE) to create a shared schema for reporting, storing, and analyzing agent runs. 🚀
Help us shape it! Fill out the form & join the Slack 👇
forms.gle/Ckw3vKSWv94cJvR48
EvalEval Coalition retweeted
If you were thinking, oh Every Eval Ever is quiet probably nothing goes out there, well... Data keeps pouring in
Oh, and we always look for contributors or people to lead the next projects (more soon)
say hi or tag your friend, PI,...
EvalEval Coalition retweeted
Some of our biggest contributors at @evaluatingevals are undergrads!
🚀 Exciting to see Evaluating Evaluations in the media!
Over the past weeks, our initiative has been featured by both @IBMResearch and @mozilla, highlighting the importance of evaluations for measuring AI progress.
📖 Mozilla: blog.mozilla.ai/from-evaluat…
1/3
At EvalEval, we’re working to make AI evaluation:
🔹 More transparent
🔹 More reproducible
🔹 More reusable
🔹 Grounded in community-driven standards
With #EveryEvalEver, we’re building a shared infrastructure for AI evaluations.
📖 IBM Research: research.ibm.com/blog/every-…
2/3
A huge thank you to everyone contributing to this growing ecosystem! 🙌
If you’d like to learn more about the EveryEvalEver schema:
📄 Paper: arxiv.org/abs/2606.14516
3/3
