Discrepancies in meta-analyses answering the same clinical question were hard to explain: a meta-epidemiological study.

OBJECTIVE: To systematically explore the methodological factors underpinning discrepancies in pooled effect estimates from Cochrane (CR) and non-Cochrane (NCR) systematic reviews answering the same clinical question. STUDY DESIGN AND SETTING: Quantitative and qualitative analysis of concordance in effect estimates between meta-analyses from CR and NCR matched on population, intervention, condition, and outcome. RESULTS: We identified twenty-four matched meta-analyses from 24 CR and 20 NCR reviews (545 randomised controlled trials [RCTs]). Compared to their CR matched-pair, pooled effects from NCR were the same in only one pair, were on average 0.12 log units (13%) higher (P=0.012) and had a greater than two-fold larger effect size in four matched-pairs. Two-thirds of CR (15/24, 70.8%) and 0/20 (0%) NCR were rated to have moderate to high confidence in their results (AMSTAR 2). Differences in pre-defined methods including search strategy, eligibility criteria and performance of dual screening could explain mismatches in included studies. Disagreements in the interpretation of eligibility criteria were identified as reasons underpinning discrepant findings in 14 pairs. 23/24 meta-analyses included at least one study of its match. Only two pairs agreed on the numerical data presented for the same studies. An assessment of 50% of discrepant studies (n=45) showed that reasons for differences in extracted data could be identified in 15 studies. CONCLUSION: On average, meta-analyses from NCR reported higher effect estimates compared with meta-analyses from CR answering the same clinical question. Methodological and author judgements and performance are key aspects underpinning poor overlap of included studies and discrepancies in reported effect estimates. The potential impacts on health care policy and clinical practice are far reaching but as yet remain unknown. Reinforcing awareness and scrutiny of application of reporting guidelines and improvements in protocol registration are needed.