Ensemble Forecasts of Coronavirus Disease 2019 (COVID-19) in the U.S.

Ray, E. L.; Wattanachit, N.; Niemi, J.; Kanji, A. H.; House, K.; Cramer, E. Y.; Bracher, J.; Zheng, A.; Yamana, T. K.; Xiong, X.; Woody, S.; Wang, Y.; Wang, L.; Walraven, R. L.; Tomar, V.; Sherratt, K.; Sheldon, D.; Reiner, R. C.; Prakash, B. A.; Osthus, D.; Li, M. L.; Lee, E. C.; Koyluoglu, U.; Keskinocak, P.; Gu, Y.; Gu, Q.; George, G. E.; Espana, G.; Corsetti, S.; Chhatwal, J.; Cavany, S.; Biegel, H.; Ben-Nun, M.; Walker, J.; Slayton, R.; Lopez, V.; Biggerstaff, M.; Johansson, M. A.; Reich, N. G.; Consortium, COVID-19 Forecast Hub

Background The COVID-19 pandemic has driven demand for forecasts to guide policy and planning. Previous research has suggested that combining forecasts from multiple models into a single "ensemble" forecast can increase the robustness of forecasts. Here we evaluate the real-time application of an open, collaborative ensemble to forecast deaths attributable to COVID-19 in the U.S. Methods Beginning on April 13, 2020, we collected and combined one- to four-week ahead forecasts of cumulative deaths for U.S. jurisdictions in standardized, probabilistic formats to generate real-time, publicly available ensemble forecasts. We evaluated the point prediction accuracy and calibration of these forecasts compared to reported deaths. Results Analysis of 2,512 ensemble forecasts made April 27 to July 20 with outcomes observed in the weeks ending May 23 through July 25, 2020 revealed precise short-term forecasts, with accuracy deteriorating at longer prediction horizons of up to four weeks. At all prediction horizons, the prediction intervals were well calibrated with 92-96% of observations falling within the rounded 95% prediction intervals. Conclusions This analysis demonstrates that real-time, publicly available ensemble forecasts issued in April-July 2020 provided robust short-term predictions of reported COVID-19 deaths in the United States. With the ongoing need for forecasts of impacts and resource needs for the COVID-19 response, the results underscore the importance of combining multiple probabilistic models and assessing forecast skill at different prediction horizons. Careful development, assessment, and communication of ensemble forecasts can provide reliable insight to public health decision makers.