this original technique is used for identifying bad behaviors which come from DPO preference data - it can be extended quite trivially to analyze failures on eval benchmarks (if you have a ground truth demonstration that succeeds). I've found it extremely useful at clustering failure modes, e.g. identifying only one instance in a segmentation problem, or outputting many degenerate boxes in a sort of doom loop. im excited to extend this to filtering SFT data. this was inspired by an interesting convo with @SwannAiden

we started out the summer trying to build the next generation of our VLM family, world class at classical image and video understanding. as we branched into tool calling capability, datasets which skew heavily towards traditional LLM domains like web search, we discovered some incredible cross task generalization: the model can control physical embodiments like drones and perform real world physical search tasks, demonstrating spatial awareness and navigational skills. this model is incredibly powerful, its fast, and you can throw basically any video, image, or real world agentic task at it. give it a try :-)

nah I do it because I hate myself. next question.