devanshu bansal (@devanshuba38471) profile picture

Head of Operations @PatronusAI | Betting on real-world data, human knowledge, and the physical world to unlock the next frontier of AI.

Joined December 2023
As AI continues to advance, extracting meaningful information and signals from the real world will become increasingly difficult. So will finding the right experts who can provide them. One metric I increasingly care about is the information delta which is essentially what does an expert know, observe, or understand that frontier models don't and what is the best way for us to extract that information. I believe that gap will become one of the most valuable things in AI development.
Expert data has a selection bias problem. In an attempt to produce as many tasks as possible, data vendors are chasing the biggest pool of experts they can find. We think this will make models worse at the details of knowledge work. 🧵
1
9
148,960
devanshu bansal retweeted
Expert data has a selection bias problem. In an attempt to produce as many tasks as possible, data vendors are chasing the biggest pool of experts they can find. We think this will make models worse at the details of knowledge work. 🧵
5
6
6
97
166,767
devanshu bansal retweeted
Today we're open-sourcing GlobeBench, 1,632 tasks that test whether LLMs can simulate RL environments well enough to train agents on. The top model, GPT-6 Astra, gets 49.6% right. No other model of the 20 we tested reaches 30%. 🧵
34
49
28
736
120,371