Vision Language Model evaluation / benchmarking | PhD student @ml_labs_irl @UCDCompSci

Joined May 2019
Small evaluation datasets are a benefit, not a downside. Running evals is expensive.
26
Nils Höhing retweeted
Just noticed that using one of the new codex self-resets also resets the time until you get the next self-reset. I thought the idea was to make usage more plannable, but this just incentivises burning through all resets in one day. Is that intended @thsottiaux?
1
3
133
In my experience not supervising agents yourself wastes a ton of tokens and time. Often you can see when a model goes off into the wrong direction, starts obsessing about minor details or ignores instructions. And human perception is still much better at catching visual bugs.
28
Just noticed that using one of the new codex self-resets also resets the time until you get the next self-reset. I thought the idea was to make usage more plannable, but this just incentivises burning through all resets in one day. Is that intended @thsottiaux?
1
3
133
Nils Höhing retweeted
Excited to share that our paper, TopoBench: Benchmarking LLMs on Hard Topological Reasoning, has been accepted at the #ICLR2026 Workshop on Logical Reasoning of Large Language Models. Nils Hoehing will be presenting it at the workshop. Many real-world tasks require maintaining global spatial constraints through a sequence of local updates: for example in circuit layout, route planning, and molecular structure analysis. We introduce TopoBench, a benchmark for topological reasoning across 6 puzzle families and 3 difficulty levels, designed to study how well LLMs handle global spatial constraints such as connectivity, loop closure, symmetry, and visibility. Our main finding is that even frontier reasoning models solve fewer than 25% of hard instances, and two puzzle families are nearly unsolved. So hard topological reasoning still appears to be a real challenge, even for very strong models. To understand why, we annotated 750 chain-of-thought traces and built an error taxonomy. The main breakdowns looked less like a failure of abstract reasoning alone, and more like failures to extract, maintain, and update the right spatial constraints while solving the puzzle. We also tested mitigation strategies, including cell-aligned representations and tool-based structured constraint checking. These help, but they point to the same conclusion: the main bottleneck seems to lie less in reasoning over constraints once available, and more in reliably deriving them from spatial representations in the first place. Paper: arxiv.org/abs/2603.12133 Website: topobench.github.io Happy to hear others’ thoughts and discuss :) Work with @nilshoehing , @JanakKapuriya , @ArjunK018 , Ellen Rushe, Anthony Ventresque, @oconnorn , and @fergal_reid . We also want to thank Intercom Research @intercom for supporting this work. #ICLR2026
3
4
274
Nils Höhing retweeted
We just released RocketScience, our new Spatial Understanding Benchmark for VLMs. It is contrastive, made of entirely new real-world data and quite diverse in the spatial layouts. It is designed to be super easy for humans but still difficult for many models. 1/3
1
2
5
190
We just released RocketScience, our new Spatial Understanding Benchmark for VLMs. It is contrastive, made of entirely new real-world data and quite diverse in the spatial layouts. It is designed to be super easy for humans but still difficult for many models. 1/3
1
2
5
190
This was joint work with: @mayugmm, Ellen Rushe, @oconnorn and @AnthoVentresque. 3/3
1
30
I am at 77 followers now, which is exactly CLIP's context length.
2
524
"Thanks Bro"-Prompting for Vision and Language Learning (BLIP-2 on @huggingface)
3
387
Nils Höhing retweeted
Today we’re releasing TorchMultimodal, a new PyTorch library containing models, building blocks and end to end examples for training and evaluating SoTA multimodal models. 📖 Learn more about scaling multimodal foundation models in the PyTorch Ecosystem: bit.ly/3Ay3zWe
6
93
8
396
Before moving from "simple" video generation to 3D-video generation, let’s first get the basic understanding of images right! Testing Relational Understanding in Text-Guided Image Generation (arxiv.org/abs/2208.00005):
1
1
“Compositionality requires an understanding of the rules that bind things together.” They evaluate DALLE-2 on physical (“on” etc) and agentic (“helping” etc) relations and find a low match between relational prompts and result images measured by humans.
1
They do however find that CLIPScore is partially sensitive to those relationships. Paper by Colin Conwell and @TomerUllman. The image is from the paper and shows DALL-E’s generation for the prompts.
Nils Höhing retweeted
i've been submitting my work to conferences for more than 10 years, fighting to make each deadline. i have tried many strategies to cope with the anxiety and make sure that i don't have to work insane amounts of hours on the days before the deadline. here's what i learned: 👇🧵
21
86
16
1,104
ClipCap: CLIP Prefix for Image Captioning -Image captioning using the CLIP embedding of the image as a prefix for GPT-2 (CLIP Embedding is first mapped to GPT-2 space) -Allows to train Image Captioning A LOT faster on a single consumer GPU, yielding competitive or sota results
1
2
-Beautiful implementation of Transfer Learning using both a pretrained CLIP and a pretrained GPT-2 -> Hoping to see more work relying on existing models rather than training from scratch! Paper: arxiv.org/pdf/2111.09734.pdf
2
2
Our #CoRL2021 poster session is happening tomr (Nov 9) at 5pm London / 9am Pacific / 12pm Eastern / 2am Tokyo. I'll be in Gathertown Room 3, Booth 4 all week. Happy to chat!
CLIP+VQGAN is amazing at making abstract CLIPArt 🎨, but can we use CLIP to ground abstract concepts in precise spatial actions with a robot 🤖? 🧵Checkout our #CoRL21 paper: CLIPort - What and Where Pathways for Robotic Manipulation Paper & Code: cliport.github.io
2
7