I really love @slatestarcodex 's latest essay "Does Forecasting Have Room At The Top?" For nearly a decade, I've thought about the fundamental limits on forecast accuracy. Scott's post spurred me to write down some of my thinking. Scott's essay examines whether the future will look like scenario A (top human forecast accuracy can't be significantly improved upon) or scenario B (accuracy can be dramatically improved with AI). Someone in the comment section asked what top forecasters think. As a GJP superforecaster, co-creator of Metaculus's Peer score, and a lead member of Futuresearch's forecasting effort I figured I might weigh in. First, I think it is worth drawing an explicit distinction between absolute & relative accuracy. A Brier score in isolation is what I would call an absolute accuracy score. However, as noted in the piece, it is generally not especially interesting because it is a function of question difficulty. What I find more interesting is the relative accuracy between forecasters, along with a confidence interval to assess significance. If we ask 1000 questions where the true probability is 1% then a perfect forecaster will forecast 1%, while a biased forecaster might forecast 2%. Even if we ask enough questions to have a tight confidence interval, the effect size will be quite small (Brier difference of 0.0001 in expectation), even though the biased forecaster is off by a factor of 2x in odds space on every question. So, like absolute accuracy, relative accuracy is also a function of which questions are being asked. And it is really hard to keep question difficulty steady. That said, let's assume we are somehow able to keep the question difficulty fixed over time... At Futuresearch, we've seen our AI forecaster improve ~linearly in absolute Brier score over time. I expected it to start plateauing and I've been wrong so far. But, I still expect its absolute score to start flattening as we approach an irreducible uncertainty limit. Regarding relative accuracy scores (like Metaculus's Peer score), I expect the head-to-head Peer score between human Pros and our AI forecaster to plateau sooner. After all, the only way to get a big relative difference in accuracy is if top humans do not use AI. Assuming top humans use AI and only make small tweaks, the relative accuracy difference must be small. In rough terms, scenario A seems more likely if we're talking about relative/Peer scores. Scenario B seems more representative of absolute scores (although I'd expect the human line to bend upwards in the near term since they'll use AI). Over the medium term (1-3 years), I expect we'll start to see absolute scores plateauing. But, I've been wrong so far & can't point to any convincing evidence to support my forecast. I'm happy to offer a probability if anyone can operationalize a precise question. Thank you Scott for your wonderful essay!

It has been so much fun having our forecaster participate on Metaculus and Kalshi. Now that we've validated its accuracy we're hoping that it can be put to good use for AI Safety and grant making, as well as commercial uses. xitter.cf/dschwarz26/status/2084…

I really wish that I could say this referred to us: "I met an AI superforecaster startup founder who told me his AI had turned $35 into $2 million on Kalshi over seven months." astralcodexten.com/p/the-ai-…