Moving beyond the classic difference-in-differences model: A simulation study comparing statistical methods for estimating effectiveness of state-level policies

Griffin, Beth Ann; Schuler, Megan S.; Stuart, Elizabeth A.; Patrick, Stephen; McNeer, Elizabeth; Smart, Rosanna; Powell, David; Stein, Bradley D.; Schell, Terry; Pacula, Rosalie L.

State-level policy evaluations commonly employ a difference-in-differences (DID) study design; yet within this framework, statistical model specification varies notably across studies. Motivated by applied state-level opioid policy evaluations, this simulation study compares statistical performance of multiple variations of two-way fixed effect models traditionally used for DID under a range of simulation conditions. While most linear models resulted in minimal bias, non-linear models and population-weighted versions of classic linear two-way fixed effect and linear GEE models yielded considerable bias (60 to 160%). Further, root mean square error is minimized by linear AR models when examining crude mortality rates and by negative binomial models when examining raw death counts. In the context of frequentist hypothesis testing, many models yielded high Type I error rates and very low rates of correctly rejecting the null hypothesis (<10%), raising concerns of spurious conclusions about policy effectiveness. When considering performance across models, the linear autoregressive models were optimal in terms of directional bias, root mean squared error, Type I error, and correct rejection rates. These findings highlight notable limitations of traditional statistical models commonly used for DID designs, designs widely used in opioid policy studies and in state policy evaluations more broadly.