We advance the science of forecasting to improve decision-making on high stakes issues. Co-founded by chief scientist Philip Tetlock.
- Tweets 366
- Following 26
- Followers 4,383
- Likes 81
We're improving the accuracy and relevance of the AI progress forecasts we publish.
Last week, we showed how the median expert, superforecaster, and member of the public in our studies have consistently underestimated AI capabilities progress (particularly on benchmarks) and have a mixed track record on other indicators.
Our findings suggest that median forecasts from these groups have not accurately predicted the trajectory of AI on some questions.
How might we get a better grasp of the trajectory of AI?
We're working on three major changes to the Longitudinal Expert AI Panel (LEAP) to address this question:
1. Sharing forecasts from a subsample of panelists who expect fast AI progress by 2040, for comparison
2. Adding regularly updated LLM forecasts to LEAP findings
3. Highlighting forecasts from the most accurate LEAP participants as soon as we have data to make this judgment rigorously
We’ve run the most comprehensive series of studies on expert AI forecasts over the last four years. Today, we’re sharing an interim update on our findings about the accuracy of these forecasts.
Our major findings are:
1. Experts, including top economists, computer scientists, and biologists, have dramatically underestimated AI capabilities progress each year. Superforecasters have underestimated progress to an even greater extent.
2. Experts have a more mixed forecasting track record on AI diffusion-related measures, with some major underestimates (forecasting AI revenue) and other forecasts on track to be accurate (the share of electricity used for AI).
3. Some notable cases of overestimating AI progress: how much mid-2025 AI models could help amateurs do biorisk-relevant laboratory tasks; the speed of rollout of self-driving cars.
4. It is too soon to say how forecasters have performed at predicting macro-scale impact on outcomes such as GDP growth, major AI harms, and averted deaths from disease.
More 🧵
3
7
42
6,460
Here are two examples of LEAP charts that feature LLM forecasts and a "fast AI progress" sample.
Details of how we selected the "fast AI progress" sample and how we ran the LLM forecasts for these example charts are in the full post linked below.
We've also recruited a new group of LEAP experts from people who work on mitigating catastrophic risk from AI. The forecasts from this group are not included in the charts below as they have only forecast a small number of LEAP questions, but this subsample will be referred to in future LEAP reports.
We'll announce more on this catastrophic risk subsample very soon.
1
4
195
For more information on these changes to LEAP, see the full post:
On Substack: forecastingresearch.substack…
On FRI’s website: forecastingresearch.org/rese…
1
2
181
We’ve run the most comprehensive series of studies on expert AI forecasts over the last four years. Today, we’re sharing an interim update on our findings about the accuracy of these forecasts.
Our major findings are:
1. Experts, including top economists, computer scientists, and biologists, have dramatically underestimated AI capabilities progress each year. Superforecasters have underestimated progress to an even greater extent.
2. Experts have a more mixed forecasting track record on AI diffusion-related measures, with some major underestimates (forecasting AI revenue) and other forecasts on track to be accurate (the share of electricity used for AI).
3. Some notable cases of overestimating AI progress: how much mid-2025 AI models could help amateurs do biorisk-relevant laboratory tasks; the speed of rollout of self-driving cars.
4. It is too soon to say how forecasters have performed at predicting macro-scale impact on outcomes such as GDP growth, major AI harms, and averted deaths from disease.
More 🧵
4
93
348
50,172
Some important caveats on the above findings:
1) We discuss many questions that have resolved early because AI milestones were reached sooner than expected, but this may introduce bias. By definition, we do not have data on late-resolving questions, and this may lead us to identify more areas where forecasters underestimated AI progress than where they have been accurate or overestimated progress.
2) In our analysis, for some unresolved questions we have used projections from LLMs to draw tentative accuracy conclusions. Though LLM forecasts are now comparable to strong human forecasts on many topics (see our work on ForecastBench, a forecasting benchmark), these projections are, by nature, more speculative.
We plan to continue to publish accuracy updates over time, particularly for our Longitudinal Expert AI Panel (LEAP), and will note cases where findings or projections change in a way that alters our previous interim conclusions.
1
8
587
Many more details on the above analysis are available in the full report:
• On Substack: forecastingresearch.substack…
• On FRI’s website: forecastingresearch.org/rese…
Authors of the summary report: @joharosenberg, @mattsreynolds1, @nadja_flechner, and Zack Devlin-Foltz.
This update was built on a large body of prior research with many more coauthors. For full reports and collaborators, see our Research page: forecastingresearch.org/rese…
Coauthors on studies that fed into this analysis included @PTetlock, @Jabaluck, @Afinetheorem, @AmandaCoston, @basilhalperin, @tatsu_hashimoto, @ToddRJones, @hannah_kerner, @random_walker, @2plus2make5, @msalganik, @robseamans, @ysu_nlp, @florian_tramer, @evavivalt, and @lucafrighetti
1
14
652