INTRODUCTION

Severity-of-illness scores are used in intensive care unit (ICU) settings worldwide for performance evaluation and monitoring, quality improvement and benchmarking.^(1-3) Since their first description in the 1980s, many prognostic models have been developed. However, among them, the Simplified Acute Physiology Score 3 (SAPS 3) is the only one developed from a multinational cohort (16,784 patients from 35 countries).^(4,5) In Brazil, Moralez et al. recently demonstrated that SAPS 3 (standard equation) remains the most accurate prognostic model.⁽⁶⁾ However, the performance of severity-of-illness scores might be different in some institutions due to the case-mix and representativeness of subgroups, such as oncological patients.

Because intensivists are increasingly managing oncological patients,⁽⁷⁾ studies evaluating the performance of these prognostic models in this subgroup of critically ill patients are welcomed. Although several studies have been published concerning this topic,^(8-10) they were published almost ten years ago, and a well-known phenomenon that compromises the performance of these prognostic models is deterioration over time (mainly in calibration) as previously demonstrated.^(8,11) None of these studies compared the model performance in oncological versus nononcological patients.

Therefore, our objectives in the present study were to evaluate the performance of SAPS 3 in patients with cancer admitted to a Brazilian ICU, compare the SAPS 3 performance of patients with and without cancer and to study time trends in SAPS 3 performance.

METHODS

This was a retrospective cohort study of all consecutive patients admitted to our medical-surgical ICU (a 30-bed unit at Hospital Sírio-Libanês, a private tertiary hospital with a dedicated oncology unit, located in São Paulo, Brazil) and was approved by the local ethics committee. The detailed description of our unit was previously published and did not change during the study period.⁽¹²⁾ The exclusion criteria were age younger than 18 years and pregnancy. If the patients had more than one admission during the inclusion period, only the first admission was included. Some of the patients included in this study were also included in a previous analysis of our group regarding ICU readmission (1,702 patients).⁽¹²⁾

Our analysis used administrative data that were prospectively collected in a cloud-based software database (Sistema Epimed™) by trained nurses.⁽¹³⁾ The study period was from January 1st, 2012 to July 31st, 2016. The oncological condition was defined as any patient admitted with an active solid cancer (current curative or palliative chemotherapy, radiotherapy, immunotherapy or surgery) in the last 12 months. Hematological patients were excluded because they usually have distinct characteristics compared with patients with solid tumors (e.g., a higher burden of active disease; required oncological treatment during ICU stay; prolonged duration of neutropenia; a higher intensity of immunosuppression; concomitant bone marrow transplant; and a higher incidence of specific complications, such as invasive mold fungal infections and cytomegalovirus infection).

The data recorded included age, sex, date of ICU admission, SAPS 3,^(4,5) referring facility, admission diagnosis, surgical procedures before admission, Charlson index for comorbidities,⁽¹⁴⁾ resource utilization during ICU stay (mechanical ventilation, vasoactive drugs or renal replacement therapy), oncological status (locoregional or metastatic) and hospital mortality. The SAPS 3 was calculated using data from the ICU admission. As recommended, missing values were coded as “normal” for each variable.⁽⁶⁾

RESULTS

During the study period, 8,345 ICU admissions occurred. After excluding 846 readmissions and 945 hematological patients, 7,390 patients remained. However, 136 patients (1.8%) did not have a hospital discharge status in our database. Therefore, our final study cohort comprised 7,254 patients (41.5% had cancer). Oncological patients were younger, had a lower SAPS 3, were admitted more frequently after elective surgery, and had higher hospital mortality than nononcological patients (14.1% versus 10.6%, respectively; p < 0.001; Table 1).

Table 1 - General characteristics of the included patients

Using the standard SAPS 3 predictive equation, the overall SMR was 0.93 (95%CI 0.88 - 0.98), with a Brier score of 0.088 and an AUROC of 0.82 (95%CI 0.81-0.83). Calibration belt analysis showed that SAPS 3 tended to overestimate mortality in our entire cohort of patients (Table 2).

Table 2 - Comparison of SAPS 3 performance in all patients and subgroups of oncological and nononcological patients

Comparison of the SAPS 3 performance in oncological patients and nononcological patients

The discrimination of SAPS 3 was higher for oncological patients than for nononcological patients (p < 0.001; Table 2). Calibration belt analysis demonstrated that, in oncological patients, no miscalibration was observed. However, in nononcological patients, SAPS 3 overestimated mortality in those with low-middle predicted risk (Figure 1).

Figure 1 - Calibration belt for Simplified Acute Physiology Score 3 in oncological (A) and nononcological patients (B), described as bisector deviation intervals. The predicted mortality intervals at which the calibration belt significantly deviates from the bisector and the 80% and 95% confidence levels are described in the lower right region of the plots. SAPS 3 - Simplified Acute Physiology Score 3.

Download

Figure 1 - Calibration belt for Simplified Acute Physiology Score 3 in oncological (A) and nononcological patients (B), described as bisector deviation intervals. The predicted mortality intervals at which the calibration belt significantly deviates from the bisector and the 80% and 95% confidence levels are described in the lower right region of the plots. SAPS 3 - Simplified Acute Physiology Score 3.

Time trend evaluation

The frequency of oncological patients did not change between the two time periods evaluated (1,498 of 3,542 patients [42.3%] versus 1,510 of 3,712 patients [40.7%], respectively; p = 0.16). The discrimination of SAPS 3 was not affected within subgroups by time period (Figure 2). Calibration belt analysis showed no miscalibration in the oncological subgroup of patients within either period. However, in the nononcological group, undercalibration was observed in the first period, and overcalibration was observed in the second period (Figure 3 andTable 3).

Figure 2 - Area under the Receiver Operating Characteristic curves for Simplified Acute Physiology Score 3 in oncological (upper part) and nononcological patients (lower part). Subgroups were divided by the time period of intensive care unit admission (first and second half).

Download

Figure 2 - Area under the Receiver Operating Characteristic curves for Simplified Acute Physiology Score 3 in oncological (upper part) and nononcological patients (lower part). Subgroups were divided by the time period of intensive care unit admission (first and second half).

Figure 3 - Calibration belt for Simplified Acute Physiology Score 3 in oncological (A and C) and nononcological patients (B and D), described as bisector deviation intervals. The predicted mortality intervals at which the calibration belt significantly deviates from the bisector and the 80% and 95% confidence levels are described in the lower right region of the plots. Subgroups were divided by the time period of intensive care unit admission (first period A and B; second period C and D). SAPS 3 - Simplified Acute Physiology Score 3.

Download

Figure 3 - Calibration belt for Simplified Acute Physiology Score 3 in oncological (A and C) and nononcological patients (B and D), described as bisector deviation intervals. The predicted mortality intervals at which the calibration belt significantly deviates from the bisector and the 80% and 95% confidence levels are described in the lower right region of the plots. Subgroups were divided by the time period of intensive care unit admission (first period A and B; second period C and D). SAPS 3 - Simplified Acute Physiology Score 3.

Table 3 - SAPS 3 performance for oncological versus nononcological patients in the first and second periods

	SMR (95%CI)	Discrimination AUROC (95%CI)	Calibration*		Precision
			Over the bisector 95%CI	Under the bisector 95%CI	Brier score
First period (2012 - 2014)
Oncological patients	1.27 (1.15 - 1.40)	0.85 (0.83 - 0.87)	Never	Never	0.10
Nononcological patients	1.08 (0.95 - 1.20)	0.79 (0.77 - 0.80)	0.00 - 0.04	Never	0.09
Second period (2014 - 2016)
Oncological patients	1.01 (0.91 - 1.12)	0.85 (0.83 - 0.87)	Never	Never	0.095
Nononcological patients	0.71 (0.62 - 0.81)	0.81 (0.79 - 0.83)	Never	0.02 - 0.74	0.075

SMR - standardized mortality rate; 95%CI - 95% confidence interval; AUROC - area under the Receiver Operating Characteristic curve.

* Calibration described as the bisector deviation intervals using the calibration belt method.

Table 3 - SAPS 3 performance for oncological versus nononcological patients in the first and second periods

DISCUSSION

In this retrospective Brazilian cohort, we evaluated SAPS 3 performance in both oncological and nononcological critically ill patients. We found that (1) the SAPS 3 discrimination and calibration were accurate in our cohort for oncological patients; (2) the discrimination was greater and calibration was more accurate for oncological patients than for nononcological patients; and (3) the calibration was affected by the time period only for nononcological patients.

Recently, Moralez et al. published the largest validation study of severity-of-illness scores in Brazil using contemporary data from a multicenter cohort.⁽⁶⁾ They showed that the SAPS 3 standard equation was accurate in predicting outcomes in our country, supporting the national initiative from the Associação de Medicina Intensiva Brasileira (AMIB) for benchmarking units.⁽¹⁸⁾ However, the case-mix variation between units may lead to performance deterioration.⁽¹⁹⁾ A particular subgroup of interest is critically ill oncological patients because 15% of the patients admitted to European ICUs have cancer.⁽²⁰⁾

Some previous publications evaluated SAPS 3 performance in oncological patients.^(8-10) Overall, these studies suggest that the measure is accurate for both discrimination and calibration, as demonstrated by our data. This finding is reassuring because oncological patients are usually poorly represented in development cohorts (8% in the original SAPS 3 cohort).^(4,5) Nevertheless, the hospital mortality of critically ill patients with cancer depends on acuity rather than on the presence and characteristics of the malignant disease itself.^(20,21) Therefore, the general severity of illness scores might capture most of the short-term prognosis in this population.

A novel approach of the present study was the comparison of SAPS 3 performance in oncological versus nononcological patients. We observed that this score was superior in oncological patients admitted to the ICU in terms of discrimination and calibration. To the best of our knowledge, this is the first study to compare these subgroups in the same cohort. The reasons for this observation are unclear but highlight the importance of case-mix differences between units and how this might affect model comparisons.

Another relevant observation was the effect of time on performance components. Calibration is particularly susceptible to time trends. Zimmerman et al. showed that the discrimination of Acute Physiology And Chronic Health Evaluation (APACHE) IV was robust and changed little throughout the evaluated time period but that calibration deteriorated in the general ICU population.⁽¹¹⁾ Soares et al. also demonstrated this deterioration in their temporal analysis of SAPS 3 calibration in oncological patients.⁽⁸⁾ Again, we observed different effects of the time trend analysis in oncological compared with nononcological patients. No miscalibration was observed in oncological patients in either period. However, not only did miscalibration occur in nononcological patients but it also moved from underestimation to overestimation from the first to the latest period evaluated. We speculated that clinical practice changes affected mainly nononcological patients, but this suggestion may be oversimplistic. Selection bias at ICU preadmission and end-of-life practice may also affect calibration.

Our study has some strengths, such as a large sample size, which allowed the performance of hypothesis-generating subgroup analyses (oncological versus nononcological patients) not previously performed. Our study also evaluated a score validated in the same settings as in previous studies in oncological patients as well as in one of the largest external validation cohorts.⁽⁶⁾ However, this study also has limitations. First, it is a single-center cohort; our results reflect local practice and our case mix (which limits the generalizability of our study). Nevertheless, some of our results agree with previous study findings. Second, the study had a long period of data collection. Although this factor might affect the overall performance evaluation, it was required for the time period analysis. Finally, end-of-life decisions were not systematically annotated in our database.

CONCLUSION

We observed that SAPS 3 discrimination was better for oncological than for nononcological patients and that SAPS3 showed no relevant deviations from optimal calibration in oncological patients. SAPS 3 performance (mainly calibration) varied over time differently according to the oncological status in our single-center cohort.