External validation of the SAPS3-CNIV score to predict hospital mortality following noninvasive ventilation: a retrospective single-centre study

To the Editor:

Prognostication tools are developed to assist clinical decision making and provide valid diagnostic and prognostic outcomes including mortality. Given significant disease and demographic heterogeneity, these tools have to be generally applicable to different patient populations. Therefore, once a model is developed it is internally and externally validated with subsequent clinical impact analyses after which its performance is evaluated and that particular model is then established.

Noninvasive ventilation (NIV) is an invaluable treatment option to reduce endotracheal intubation rates and mortality in selected groups of patients with acute respiratory failure [1, 2]. There are limited prognostication tools available to help predict outcomes and guide management in patients treated with NIV.

The Simplified Acute Physiology Score 3 (SAPS3) has been used as a prognostic model in intensive care units (ICUs) to predict mortality [3]. It takes into account patient demographics, comorbidities, and biochemical and physiological disturbances within the first hour of admission to ICU. The SAPS3-Customized NIV (SAPS3-CNIV) model complements the existing SAPS3 score with additional variables such as haemoglobin, carbon dioxide tension (P_CO2), lactate, do not resuscitate (DNR) orders and aetiology of respiratory failure. It has been suggested to be useful in predicting in-hospital death for patients managed with NIV, regardless of aetiology or comorbidities [4]. However, the SAPS3-CNIV model has not been externally validated.

Our aim was to externally validate SAPS3-CNIV score, with the hypothesis that the SAPS3-CNIV model is more accurate than the SAPS3 score in predicting mortality in patients treated with NIV. Study end-points included in-hospital death or intubation with invasive ventilation during the same admission. We chose to validate SAPS3-CNIV over the SAPS II-CNIV score, developed by the same authors, as it predicts mortality within 1 h of admission to ICU rather than 24 h and hence can guide decisions on patient disposition and location of care (ICU versus general ward).

We performed a retrospective study of consecutive patients managed with NIV between November 2016 and March 2018 in a high dependency unit at St. John of God Midland Public and Private Hospital, which is a general secondary hospital in Perth, Western Australia. Patients younger than 18 years of age, readmissions requiring NIV during the study period and those who were managed with NIV post-invasive ventilation were excluded from the study. Patients were treated with bi-level mode NIV with pressures titrated under direct supervision of a specialist respiratory team. Data were collected from electronic health records and patient outcomes including death or intubation requiring invasive ventilation during the same admission were recorded. Continuous variables were expressed as mean±sd. SAPS3 and SAPS3-CNIV model discriminatory accuracy was assessed using the area under the receiver operating characteristic curve (AUC). Given the alpha- and beta-coefficients of the SAPS3-CNIV logistic model were not published in the original development publication, calibration was assessed by the observed-to-expected mortality ratios across the three previously described risk groups (SAPS3-CNIV score <34, 34–69 and >69). Multivariate logistic regression was used to assess association of SAPS3-CNIV variables with mortality in our cohort. Statistical analysis was performed on SAS University Edition (SAS Studio 3.6, SAS 9.4M4; SAS Institute Inc., Cary, NC, USA). The study protocol was approved by St. John of God Health Care Human Research Ethics Committee (reference: 1214).

A total of 228 patients were managed with NIV during our study period. Mean±sd age was 70.2±14.3 years. There was an almost equal distribution of sex with a female proportion of 50.4%. Prevalent comorbidities included chronic obstructive pulmonary disease (COPD) (64.9%), ischaemic heart disease (30.7%), diabetes (26.3%), metastatic cancer (6.6%) and underlying immunosuppressive state (3.5%). Overall, hypercapnic respiratory failure was present in 75% of our cohort. The most common primary indication for NIV use was exacerbation of COPD (46.5%) followed by acute pulmonary oedema (APO) (17.1%). 27 (11.8%) patients had hypercapnic respiratory failure mainly from community-acquired pneumonia with a comorbid predisposing chronic respiratory disorder. 9.2% had de novo hypoxaemic respiratory failure, which included patients with lower respiratory tract infections and acute respiratory distress syndrome and no known chronic respiratory disorder. Other indications for NIV initiation included sleep disordered breathing and/or central nervous system depressant use (6.6%), exacerbation of asthma (3.5%), interstitial lung disease (2.6%) and neuromuscular diseases (2.6%). The in-hospital mortality rate was 21.5% and 9.7% of the patients underwent intubation with subsequent invasive ventilation.

There was a no significant difference between SAPS3 score (AUC 0.72, 95% CI 0.64–0.80) and SAPS3-CNIV (AUC 0.78, 95% CI 0.70–0.85) to predict in-hospital death (p=0.16) (figure 1a). The results reached near statistical significance (p=0.07) when SAPS3-CNIV was used to predict the combined outcome of intubation and/or death (AUC 0.77, 95% CI 0.71–0.84) (figure 1b). On subpopulation analysis of patients with COPD exacerbation, APO and de novo hypoxaemic respiratory failure, there was no significant difference between discriminatory accuracy of SAPS3 and SAPS3-CNIV to predict mortality (p=0.54, 0.94 and 0.98, respectively). Mortality observed in the three risk groups (SAPS3-CNIV score <34, 34–69 and >69) were 13%, 28% and 65% corresponding to observed-to-expected mortality ratios of 1.24, 0.78 and 1.14, respectively.

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 1

a) Receiver operating characteristic (ROC) curves for comparison of the SAPS 3 and SAPS3-CNIV score to predict mortality and b) the combined outcomes of intubation±mortality. c) Multivariate analysis of SAPS3-CNIV variables for predicting mortality in development and validation cohorts. SJGMPPH: St. John of God Midland Public and Private Hospital; DNR: do not resuscitate; COPD: chronic obstructive pulmonary disease; P_CO2: carbon dioxide tension.

Multivariate analyses of SAPS3-CNIV variables for predicting mortality in our validation cohort were similar to the development cohort with the SAPS3 score, APO and DNR orders being statistically significant risk factors. Presence of COPD and hypercapnic respiratory failure with P_CO2 >55 mmHg were protective with near statistical significance. Lactic acid >2 mg·dL⁻¹ and haemoglobin >10.7 g·dL⁻¹ were not statistically significant (figure 1c).

In our external validation cohort, the SAPS3-CNIV score was not superior to SAPS3 at predicting in-hospital mortality following NIV. This may be due to disease heterogeneity in our population compared with the development cohort with a large proportion of patients with an exacerbation of COPD (46.5% versus 18.3%), lower prevalence of metastatic cancer (6.6% versus 24.5%), a proportionally smaller immunosuppressed population (3.5% versus 56.2%) and less mortality (21.5% versus 32.4%). Furthermore, lactate >2 mg·dL⁻¹ and haemoglobin >10.7 g·dL⁻¹ were not predictive in our population in contrast to the development cohort and therefore may not be repeatable across different settings. This score was originally designed to predict mortality in unselected patients regardless of comorbidities or aetiology of respiratory failure, but this was not reflected in our cohort. The SAPS3-CNIV model performed better when intubation was combined with death as the dependent outcome, but the results did not reach statistical significance in our population.

To our knowledge the study by Martinez-Urbistondo et al. [4] has been the only study which has evaluated SAPS3 with discrimination power and calibration in patients treated with NIV regardless of the aetiology of respiratory failure or comorbidities. A previous study by Metnitz et al. [5] analysed a subgroup of patients from the SAPS3 database and described the patient cohort who were treated with NIV but did not predict the discriminatory accuracy of the score. Another model called the HACOR score [6] takes into account heart rate, acidosis, consciousness, oxygenation and respiratory rate to predict NIV failure in patients with acute hypoxaemic respiratory failure, but the indication for NIV in this scenario is debatable.

Our study has several important limitations. First, it was a retrospective study which resulted in inclusion of patients in whom NIV was prescribed for a variety of indications beyond the evidence-based recommendations. This may potentially have deleterious effects on outcomes and hence these scores need to be applied in the right clinical context. Secondly, it was a single centre study which may limit the external generalisability of the results. Both the SAPS3 and SAPS3-CNIV scores may not be practical in real life given that multiple variables need to be calculated which is time consuming and may not be possible in acute settings. Future research directions would be to develop a simple, accurate score for risk of NIV failure in selected patients where NIV is prescribed according to the evidence-based guidelines. This could guide the location of care as patients with higher scores may need to be monitored in resource-intensive ICU settings.

In conclusion, the SAPS3-CNIV model did not improve mortality prediction in patients over SAPS3 in our cohort. Further development of a simplified and practical score to predict outcomes in patients treated with NIV is required.