Wout Schellaert (@WoutSchellaert) header
Wout Schellaert (@WoutSchellaert) profile picture

PhD-student at UP Valencia working on AI evaluation. Modelling evaluation as prediction. Also other things.

Valencia
Joined February 2017
Photo by Wout Schellaert (@WoutSchellaert)Photo by Wout Schellaert (@WoutSchellaert)Photo by Wout Schellaert (@WoutSchellaert)
This is one of the one the best (if not the best) approach to AI evaluation I've seen. You can't blabla your way to the predictive power they report in section 3.4!
ADeLe, a new evaluation method, explains what AI systems are good at—and where they’re likely to fail. By breaking tasks into ability-based requirements, it has the potential to provide a clearer way to evaluate and predict AI model performance: msft.it/6014SkVGC
1
7
299
These cheeky LLMs make their answers very convincing, and unsurprisingly, this is tricking users into overrelying on them...
1/ New paper @Nature! Discrepancy between human expectations of task difficulty and LLM errors harms reliability. In 2022, Ilya Sutskever @ilyasut predicted: "perhaps over time that discrepancy will diminish" (youtu.be/W-F7chPE9nU, min 61-64). We show this is *not* the case!
1
1
5
546
Wout Schellaert retweeted
1/ New paper @Nature! Discrepancy between human expectations of task difficulty and LLM errors harms reliability. In 2022, Ilya Sutskever @ilyasut predicted: "perhaps over time that discrepancy will diminish" (youtu.be/W-F7chPE9nU, min 61-64). We show this is *not* the case!
19
288
50
1,038
298,636
New and shiny AI systems have superseded the ones we reference (it took a while to publish), but our perspectives and suggestions for evaluating them have only become more relevant. Go have a read! 👽👽
New Article: "Your Prompt is My Command: On Assessing the Human-Centred Generality of Multimodal Models" by Schellaert, Martínez-Plumed, Vold, Burden, Casares, Loe, Reichart, Ó hÉigeartaigh, Korhonen and Hernández-Orallo jair.org/index.php/jair/arti…
2
3
5
995
with wonderful collaborators @NandoMartinezP, @karinavold, @JohnJBurden, Pablo A.M. Casares, Aiden Loe, @roireichart, @S_OhEigeartaigh, @annalkorhonen and José Hernández-Orallo.
4
182
Happy to be a part of this, among so many I look up to in this field.
Is it time to rethink how we perform system evaluations in AI? In our new @ScienceMagazine paper, we show that over-reliance on aggregate metrics and a lack of transparency in reporting threatens public understanding and hinders progress in the field. 1/8 doi.org/10.1126/science.adf6…
4
207
Wout Schellaert retweeted
Interested in AI robustness and predictability? Come join us in sunny Valencia for an exciting workshop on March 8th! Information here: predictable-ai.org/
3
4
786
This is a goldmine for labs without the compute to run these huge LMs themselves. Great effort!
Replying to @percyliang
From crfm.stanford.edu/helm/, you can explore all the results. Drill down beyond aggregate statistics into individual predictions and exact prompts. It’s fully reproducible. Download it all, perform your own analyses, and let us know what you find!
We're starting an old school mailing list for folks interested in how to evaluate AI (and all questions that come with it). Open for all to join and post! Come come! groups.google.com/g/ai-eval
Still 10 full days to submit your papers for the 📐Evaluation Beyond Metrics workshop @IJCAIconf. Not that you need another excuse to come, with @adinamwilliams and @AmandaMSeed giving a talk! 🔗 sites.google.com/view/ebem20…
1
6
8
📐Our Evaluation Beyond Metrics workshop at IJCAI got accepted... so prepare your cool papers! 💻sites.google.com/view/ebem20… With @LucyCheke, @DanajaRutar, @JohnJBurden, @DrRyanBurnell, @TomerUllman and twitterless Josh Tenenbaum, José Hernández-Orallo and Fernando Martínez-Plumed
2
7
14
Wout Schellaert retweeted
BREAKING: In a huge victory, the European Parliament has voted 318-278 against #Article13 and #Article11—the disastrous #CensorshipMachine and #LinkTax copyright proposals. That means we’re close to stopping these terrible proposals—and we’re gaining momentum.
101
2,908
484
5,657
Ever wanted to query over different types of Linked Data interfaces 🤔? You can now do this with Comunica 📬. Try it out now in your browser 🎉! https://xitter.cf/t.co/OK28WywG2N Announcement: https://xitter.cf/t.co/xe9C7vZS6d
1
16
3
21
Wout Schellaert retweeted
Free software hosted on @GitHub and elsewhere is threatened by a planned EU law meant to regulate big business infighting. Will the #opensource community discover its political clout in time to stop #UploadFilters? Here’s how you can help: juliareda.eu/2018/04/free-so…
7
262
13
183
Wout Schellaert retweeted
Have you ever used @github? At your company, your university, your NGO or at home? Then you should be worried about EU #Copyright reform: blog.github.com/2018-03-14-e… #CensorshipMachines #FixCopyright
24
915
76
640
Wout Schellaert retweeted
Third highest total score at out #hashcode hub @ugent, good job teams!
3
6
Is there a good way to avoid null pointer bugs? Maybe.
17
141
9
381