CommentaryValidation in prediction research: the waste by data splitting
Section snippets
Large sample validation
Examples with large sample size for development and validation are found in virtually all prediction models coming from the QResearch general practices resulting in Q score algorithms [3]. These can be seen as big data approaches. Here, routinely collected data from hundreds of general practices are used for model development and hundreds for validation. Such a split sample approach is attractive for its simplicity in providing independent and, hence, unbiased assessment of model performance.
Small sample validation
A recent and rather extreme example of data splitting was the evaluation of the prognostic value of single-cell analyses in leukemia [9]. To predict relapse, a data set was available with 54 patients. A model was constructed in 80% of the sample (44 patients) and validated in 20% (10 patients). Discriminative performance was assessed by a standard measure, the C-statistic [4], [10]. The study found that there were three relapses among the 10 patients in the validation cohort, with perfect
Simulation study
A simulation study was designed with three sample sizes and a 30% event rate (as in the leukemia study): extremely small (10 patients, three events), moderate (333 patients, 100 events), and large (1,667 patients, 500 events). We examine the variability of three different prediction models (or “classifiers”) by simulation, assuming that the true C-statistic of the prediction model would be 0.7, 0.8, or 0.9 (Fig. 1). We find that with only three events, a substantial fraction of validations
Implications
From the aforementioned, three implications can be learned for the practice of validation of prediction models:
- 1)
In the absence of sufficient sample size, independent validation is misleading and should be dropped as a model evaluation step [14]. It is preferable to use all data for model development with some form of cross-validation or bootstrap validation for the assessment of the statistical optimism in average predictive performance [15].
- 2)
Basically, we should accept that small size studies on
References (22)
- et al.
External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination
J Clin Epidemiol
(2015) - et al.
Geographic and temporal validity of prediction models: different approaches were useful to examine model performance
J Clin Epidemiol
(2016) - et al.
Poor performance of clinical prediction models: the harm of commonly applied methods
J Clin Epidemiol
(2018) - et al.
Substantial effective sample sizes were required for external validation studies of predictive logistic regression models
J Clin Epidemiol
(2005) - et al.
Prediction models need appropriate internal, internal-external, and external validation
J Clin Epidemiol
(2016) - et al.
Internal validation of predictive models: efficiency of some procedures for logistic regression analysis
J Clin Epidemiol
(2001) - et al.
Assessment of the accuracy of diagnostic tests: the cross-sectional study
J Clin Epidemiol
(2003) - et al.
External validation of multivariable prediction models: a systematic review of methodological conduct and reporting
BMC Med Res Methodol
(2014) - et al.
Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: prospective cohort study
BMJ
(2017) Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis
(2001)
Validation of prediction models: examining temporal and geographic stability of baseline risk and estimated covariate effects
Diagn prognostic Res
Cited by (86)
Merkel cell carcinoma recurrence risk estimation is improved by integrating factors beyond cancer stage: A multivariable model and web-based calculator
2024, Journal of the American Academy of DermatologyMachine learning for decision-making in cardiology: a narrative review to aid navigating the new landscape
2023, Revista Espanola de CardiologiaSystematic review highlights high risk of bias of clinical prediction models for blood transfusion in patients undergoing elective surgery
2023, Journal of Clinical EpidemiologyMethods and procedures of clinical predictive model
2024, Chinese Journal of Evidence-Based Medicine