Abstract
Background
The Clinical High Risk (CHR) paradigm has led research into the biological and clinical underpinnings of the risk for psychosis, aiming at predicting and possibly preventing transition to the disorder. Statistical methods like machine learning (ML) and Cox proportional hazard regression have enabled the construction of diagnostic and prognostic models based on different data modalities, e.g., clinical risk factors, neurocognitive performance, or neurobiological data. However, their translation to clinical practice is still hindered by the heterogeneity both of CHR populations and methodologies. One way to tackle this issue is to use a meta-analytic approach to quantitatively investigate models’ performance throughout different outcomes, algorithms and data modalities. The aim of this work was, thus, to investigate the effects of (I) data modality, (II) type of algorithm, and (III) validation paradigms on prognostic and diagnostic models’ performance. We expect our results to facilitate a deeper understanding of the state-of-the-art within the CHR research field and clarify the methodological bottlenecks that impede the clinical translation of diagnostic and prognostic tools.
Methods
We systematically reviewed the literature on diagnostic and prognostic models built on Cox regression and ML. Further, we conducted a meta-analysis on accuracy performances investigating effects of the following moderators: age, sex, data modality, algorithm, presence of cross-validation (CV), being a multisite study and year of publication. For prognostic studies we investigated also follow-up time and prognostic target. All analyses were conducted with R v3.6.0. and results were corrected for False Discovery Rate.
Results
44 articles were included for a total of 3707 individuals for prognostic and 1052 for diagnostic studies (572 CHR and 480 healthy controls, HC). CHR could be classified against HC with 78% sensitivity (95%-CI: 63%-83%) and 77% specificity (95%-CI: 68%-84%). Across prognostic models, sensitivity reached 67% (95%-CI: 63%-70%) and specificity 78% (95%-CI: 73%-82%). Our results point to a higher sensitivity of ML models compared to Cox regression in prognostic studies (p = .009; χ2(2) = 6.96, p = 0.031). This effect was collinear with that of CV, due to the overlap of this factor with algorithm type. Notably, there was a publication bias for prognostic studies (R2 = 0.26, p < .001), yet no significant effects of data modality, CHR or CV type, prognostic target, or any other confounding variable (e.g., age distribution, sex, year of publication or follow-up interval time) on accuracy performance.
Discussion
Our results point to a good models’ performance overall and no effects of data modality or patient population. ML outperformed Cox regression in prognostic studies, these, however, showing a publication bias. These results may be driven by substantial clinical and methodological heterogeneity currently affecting several aspects of the CHR field. A comprehensive change within the current CHR paradigm is required to enable the clinical application of diagnostic and prognostic models for the at-risk state. First, the field requires study design harmonization, which demands, for instance, reliable methodological approaches like cross- or external validation to ensure generalizability. Second, efforts may be made in unifying the CHR definition, both theoretically and practically, and also embrace relevant non-transition outcomes to broaden the prognostic scope. Future studies are needed to investigate whether harmonising procedures within precision psychiatry will lead to more reliable and reproducible translational research in the field.
Process of change: states, transitions, and determinants