As AI continues to advance, extracting meaningful information and signals from the real world will become increasingly difficult. So will finding the right experts who can provide them.
One metric I increasingly care about is the information delta which is essentially what does an expert know, observe, or understand that frontier models don't and what is the best way for us to extract that information.
I believe that gap will become one of the most valuable things in AI development.
Expert data has a selection bias problem.
In an attempt to produce as many tasks as possible, data vendors are chasing the biggest pool of experts they can find. We think this will make models worse at the details of knowledge work. 🧵