Over-reliance on p values in Urology: Fragility of findings in the hydronephrosis literature calls for systematic reporting of robustness indicators.

Rickard, Mandy; Lorenzo, Armando J; Hannick, Jessica H; Blais, Anne-Sophie; Koyle, Martin A; Bägli, Darius J

OBJECTIVE To review the robustness of hydronephrosis (HN) literature with the application of fragility index (FI) and fragility quotient (FQ) calculations. METHODS A literature review was conducted using Pubmed, Medline and Ovid for "hydronephrosis" and associated terms and we included all studies with at least 2 groups being compared. FI was calculated by populating study results into a two-by-two contingency table and generating a p-value using Fisher's exact test. Next, events were manually added to the group with the fewest events, while removing a non-event from the same group and Fisher's exact test repeated until the p-value was >0.05. FQ was calculated by dividing FI by the total sample size. RESULTS The 130 included articles were published between 1986-2018 in 32 journals. Median citation count was 14(0-252), 30% were RCTs and most papers originated in the United States(28%), Turkey(10%) and Canada(9%). Median FI was 2(1-112], FQ was0.023 (0.0010-0.55) and 60 papers(46%) had a fragility index of 1, indicating extremely fragile results. There was a significant difference in the FI between observational studies and RCTs(10±17 vs. 4±5; p=0.02); however, there was no difference in FQ(0.032± 0.030 vs. 0.053±0.080; p=0.09) between them. CONCLUSION Nearly half of studies in HN literature reporting significant results are extremely fragile, requiring addition of only a couple of events in one treatment arm to significantly modify the results. As such, objective reporting of robustness of results should include FI and FQ which may help diminish over-reliance on p-values as the main indicator of clinical significance in comparative studies.