This is one of the one the best (if not the best) approach to AI evaluation I've seen. You can't blabla your way to the predictive power they report in section 3.4!
ADeLe, a new evaluation method, explains what AI systems are good at—and where they’re likely to fail. By breaking tasks into ability-based requirements, it has the potential to provide a clearer way to evaluate and predict AI model performance: msft.it/6014SkVGC
These cheeky LLMs make their answers very convincing, and unsurprisingly, this is tricking users into overrelying on them...
1/ New paper @Nature!
Discrepancy between human expectations of task difficulty and LLM errors harms reliability. In 2022, Ilya Sutskever @ilyasut predicted: "perhaps over time that discrepancy will diminish" (youtu.be/W-F7chPE9nU, min 61-64).
We show this is *not* the case!
Wout Schellaert retweeted
1/ New paper @Nature!
Discrepancy between human expectations of task difficulty and LLM errors harms reliability. In 2022, Ilya Sutskever @ilyasut predicted: "perhaps over time that discrepancy will diminish" (youtu.be/W-F7chPE9nU, min 61-64).
We show this is *not* the case!
New and shiny AI systems have superseded the ones we reference (it took a while to publish), but our perspectives and suggestions for evaluating them have only become more relevant. Go have a read! 👽👽
New Article: "Your Prompt is My Command: On Assessing the Human-Centred Generality of Multimodal Models" by Schellaert, Martínez-Plumed, Vold, Burden, Casares, Loe, Reichart, Ó hÉigeartaigh, Korhonen and Hernández-Orallo jair.org/index.php/jair/arti…
with wonderful collaborators @NandoMartinezP, @karinavold, @JohnJBurden, Pablo A.M. Casares, Aiden Loe, @roireichart, @S_OhEigeartaigh, @annalkorhonen and José Hernández-Orallo.
4
182
Happy to be a part of this, among so many I look up to in this field.
Is it time to rethink how we perform system evaluations in AI? In our new @ScienceMagazine paper, we show that over-reliance on aggregate metrics and a lack of transparency in reporting threatens public understanding and hinders progress in the field. 1/8 doi.org/10.1126/science.adf6…
4
207
Wout Schellaert retweeted
Interested in AI robustness and predictability? Come join us in sunny Valencia for an exciting workshop on March 8th! Information here: predictable-ai.org/
This is a goldmine for labs without the compute to run these huge LMs themselves. Great effort!
Replying to @percyliang
From crfm.stanford.edu/helm/, you can explore all the results. Drill down beyond aggregate statistics into individual predictions and exact prompts. It’s fully reproducible. Download it all, perform your own analyses, and let us know what you find!
We're starting an old school mailing list for folks interested in how to evaluate AI (and all questions that come with it).
Open for all to join and post! Come come!
groups.google.com/g/ai-eval
Still 10 full days to submit your papers for the 📐Evaluation Beyond Metrics workshop @IJCAIconf.
Not that you need another excuse to come, with @adinamwilliams and @AmandaMSeed giving a talk!
🔗 sites.google.com/view/ebem20…
📐Our Evaluation Beyond Metrics workshop at IJCAI got accepted... so prepare your cool papers!
💻sites.google.com/view/ebem20…
With @LucyCheke, @DanajaRutar, @JohnJBurden, @DrRyanBurnell, @TomerUllman and twitterless Josh Tenenbaum, José Hernández-Orallo and Fernando Martínez-Plumed
Wout Schellaert retweeted
BREAKING: In a huge victory, the European Parliament has voted 318-278 against #Article13 and #Article11—the disastrous #CensorshipMachine and #LinkTax copyright proposals.
That means we’re close to stopping these terrible proposals—and we’re gaining momentum.
Wout Schellaert retweeted
Ever wanted to query over different types of Linked Data interfaces 🤔?
You can now do this with Comunica 📬.
Try it out now in your browser 🎉! https://xitter.cf/t.co/OK28WywG2N
Announcement: https://xitter.cf/t.co/xe9C7vZS6d
Wout Schellaert retweeted
Free software hosted on @GitHub and elsewhere is threatened by a planned EU law meant to regulate big business infighting. Will the #opensource community discover its political clout in time to stop #UploadFilters? Here’s how you can help: juliareda.eu/2018/04/free-so…
Wout Schellaert retweeted
Have you ever used @github? At your company, your university, your NGO or at home? Then you should be worried about EU #Copyright reform: blog.github.com/2018-03-14-e… #CensorshipMachines #FixCopyright





















