Predictive performance of international COVID-19 mortality forecasting models

Friedman, Joseph; Liu, Patrick; Troeger, Christopher E.; Carter, Austin; Reiner, Robert C.; Barber, Ryan M.; Collins, James; Lim, Stephen S.; Pigott, David M.; Vos, Theo; Hay, Simon I.; Murray, Christopher J. L.; Gakidou, Emmanuela

Forecasts and alternative scenarios of COVID-19 mortality have been critical inputs for pandemic response efforts, and decision-makers need information about predictive performance. We screen n = 386 public COVID-19 forecasting models, identifying n = 7 that are global in scope and provide public, date-versioned forecasts. We examine their predictive performance for mortality by weeks of extrapolation, world region, and estimation month. We additionally assess prediction of the timing of peak daily mortality. Globally, models released in October show a median absolute percent error (MAPE) of 7 to 13% at six weeks, reflecting surprisingly good performance despite the complexities of modelling human behavioural responses and government interventions. Median absolute error for peak timing increased from 8 days at one week of forecasting to 29 days at eight weeks and is similar for first and subsequent peaks. The framework and public codebase (https://github.com/pyliu47/covidcompare) can be used to compare predictions and evaluate predictive performance going forward.