Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the US

Cramer, E. Y.; Ray, E. L.; Lopez, V. K.; Bracher, J.; Brennen, A.; Castro Rivadeneira, A. J.; Gerding, A.; Gneiting, T.; House, K. H.; Huang, Y.; Jayawardena, D.; Kanji, A. H.; Khandelwal, A.; Le, K.; Muhlemann, A.; Niemi, J.; Shah, A.; Stark, A.; Wang, Y.; Wattanachit, N.; Zorn, M. W.; Gu, Y.; Jain, S.; Bannur, N.; Deva, A.; Kulkarni, M.; Merugu, S.; Raval, A.; Shingi, S.; Tiwari, A.; White, J.; Woody, S.; Dahan, M.; Fox, S.; Gaither, K.; Lachmann, M.; Meyers, L. A.; Scott, J. G.; Tec, M.; Srivastava, A.; George, G. E.; Cegan, J. C.; Dettwiller, I. D.; England, W. P.; Farthing, M. W.; Hunter,

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. In 2020, the COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized hundreds of thousands of specific predictions from more than 50 different academic, industry, and independent research groups. This manuscript systematically evaluates 23 models that regularly submitted forecasts of reported weekly incident COVID-19 mortality counts in the US at the state and national level. One of these models was a multi-model ensemble that combined all available forecasts each week. The performance of individual models showed high variability across time, geospatial units, and forecast horizons. Half of the models evaluated showed better accuracy than a naive baseline model. In combining the forecasts from all teams, the ensemble showed the best overall probabilistic accuracy of any model. Forecast accuracy degraded as models made predictions farther into the future, with probabilistic accuracy at a 20-week horizon more than 5 times worse than when predicting at a 1-week horizon. This project underscores the role that collaboration and active coordination between governmental public health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks. f