The ‘double whammy’ of low prevalence in clinical risk prediction

Background Worldwide, around 800 000 people die each year from suicide,1 which is the leading cause of death in the UK in young adults.2 Prediction modelling studies have attempted to incorporate demographic, clinical and other factors to identify high-risk individuals so that appropriate interventions can be offered.3 4 This approach has a large literature but has not always been judged successful. In one review, all 35 suicide risk prediction studies assessed were classified as having high risk of bias or insufficient diagnostic accuracy, based on targets of 80% sensitivity and 50% specificity.5 Others have written of a performance ‘glass ceiling’ in suicide prediction, and even that ‘risk categorization of individual patients has no role to play in preventing the suicide of psychiatric inpatients’.6 In spite of the mortality data quoted previously, in most populations, risk of death from suicide is low, usually below 1%. This has led to claims about performance of prediction models that appear counterintuitive, such as ‘suicide prediction models produce accurate overall classification models, but their accuracy of predicting a future event is near 0’, while noting that even in high-risk populations, the positive predictive value (PPV) of many prediction rules may be less than 1%.3 We describe two principal reasons why the nature of developing clinical prediction rules in low prevalence scenarios will almost invariably result in concerns about performance, on standards by which this is usually judged. Although we focus primarily on suicide prediction, these points apply more generally to other low prevalence clinical areas, some examples of which are also discussed.