Prediction models for self-harm and suicide: a systematic review and critical appraisal

Background: The number of prediction models for self-harm and suicide has grown substantially in recent years. However, their potential role in improving assessment of suicide risk is debated. In this systematic review, we provide an overview and critical appraisal of the predictive performance and methodological quality of prognostic risk models for self-harm and suicide. Methods: We searched MEDLINE, EMBASE, PsycINFO, CINAHL, and Global Health from inception to 30/11/2021. The search was updated on 25/10/2024 to include new external validations. We included studies describing the development and/or external validation of statistical models for predicting risk of non-fatal self-harm and/or death by suicide. Risk of bias was assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). Results: We included 91 articles describing the development of 167 models and 29 external validations. Most models predicted risk of self-harm (76 models), followed by suicide (51 models), and the composite outcome of suicide or non-fatal self-harm (40 models). Only 8% of developed models (14/167) were externally validated, and 17% (28/167) were presented in a format enabling validation or use by others. The reported C indices ranged from 0.61 to 0.97 (median 0.82) in development studies and from 0.60 to 0.86 (median 0.81) in external validations. Calibration was assessed for 9% of models (15/167) in development studies and 31% of external validations (9/29). Of these, the OxMIS and Simon models showed adequate discrimination and calibration performance in external validation. All model development studies, and all but two external validations, were at high risk of bias. This was mainly driven by inappropriate or incomplete evaluation of predictive performance (180/196, 92%), insufficient sample sizes (151/196, 77%), inappropriate handling of missing data (129/196, 66%), and not adequately accounting for overfitting and optimism during model development (106/167, 63%). Conclusions: Despite skepticism about the feasibility and accuracy of self-harm and suicide risk prediction and assessment, we have identified five models with good predictive performance in external validation. Avoidable sources of research waste include an oversupply of unvalidated prediction models addressing similar research questions, and shortcomings in study design, conduct, and statistical analysis. To address these, new research must prioritise methodological rigour and focus on external validation and updating existing models. Complete, transparent, and accurate reporting is essential, with model presentation in a format that enables independent validation.