Free On-line Access

SPCI - Sociedade Portuguesa de Cuidados Intensivos

Revista Brasileira de Terapia Intensiva

AMIB - Associação de Medicina Intensiva Brasileira

OFFICIAL JOURNAL OF THE ASSOCIAÇÃO BRASILEIRA DE MEDICINA INTENSIVA AND THE SOCIEDADE PORTUGUESA DE CUIDADOS INTENSIVOS

ISSN: 0103-507X
Online ISSN: 1982-4335

Ícone Fechar

How to Cite


 

Taniguchi LU, Siqueira EMP. Comparação do desempenho do SAPS 3 em pacientes com e sem tumor sólido admitidos a uma unidade de terapia intensiva no Brasil: um estudo de coorte retrospectiva. Rev Bras Ter Intensiva. 2020;32(4):521-527

 

 

2020;32(4):521-527
ORIGINAL ARTICLE

10.5935/0103-507X.20200089

Comparison of SAPS 3 performance in patients with and without solid tumor admitted to an intensive care unit in Brazil: a retrospective cohort study

Comparação do desempenho do SAPS 3 em pacientes com e sem tumor sólido admitidos a uma unidade de terapia intensiva no Brasil: um estudo de coorte retrospectiva

Leandro Utino Taniguchi1,2,3, Ellen Maria Pires Siqueira1

1 Hospital Sírio-Libanês - São Paulo (SP), Brazil.
2 Emergency Medicine Discipline, Hospital das Clínicas, Faculdade de Medicina, Universidade de São Paulo - São Paulo (SP), Brazil.
3 Brazilian Research in Intensive Care Network (BRICNet) - São Paulo (SP), Brazil.

Conflicts of interest: None.

Disclaimer: Dr. Taniguchi, Section Editor for Revista Brasileira de Terapia Intensiva, was not involved in the evaluation or decision to publish this article. Dr. Taniguchi has received financial fees for lectures from Nestlé.

Responsible editor: Márcio Soares

AUTHORS’ CONTRIBUTIONS

L.U. Taniguchi was responsible for the study conception, data interpretation and drafting of the manuscript. E.M.P. Siqueira acquired the data. L.U. Taniguchi and E.M.P. Siqueira analyzed the data. All the authors read and approved the final manuscript.

Submitted on January 30, 2020
Accepted on April 05, 2020

Corresponding author: Leandro Utino Taniguchi, Unidade de Terapia Intensiva, Hospital Sírio-Libanês, Rua Daher Cutait, 69 - Bela Vista, Zip code: 01308-060 - São Paulo (SP), Brazil. E-mail: [email protected]

 

Abstract

OBJECTIVE: To compare the performance of the Simplified Acute Physiology Score 3 (SAPS 3) in patients with and without solid cancer who were admitted to the intensive care unit of a comprehensive oncological hospital in Brazil.
METHODS: We performed a retrospective cohort analysis of our administrative database of the first admission of adult patients to the intensive care unit from 2012 to 2016. The patients were categorized according to the presence of solid cancer. We evaluated discrimination using the area under the Receiver Operating Characteristic curve (AUROC) and calibration using the calibration belt approach.
RESULTS: We included 7,254 patients (41.5% had cancer, and 12.1% died during hospitalization). Oncological patients had higher hospital mortality than nononcological patients (14.1% versus 10.6%, respectively; p < 0.001). SAPS 3 discrimination was better for oncological patients (AUROC = 0.85) than for nononcological patients (AUROC = 0.79) (p < 0.001). After we applied the calibration belt in oncological patients, the SAPS 3 matched the average observed rates with a confidence level of 95%. In nononcological patients, the SAPS 3 overestimated mortality in those with a low-middle risk. Calibration was affected by the time period only for nononcological patients.
CONCLUSION: SAPS 3 performed differently between oncological and nononcological patients in our single-center cohort, and variation over time (mainly calibration) was observed. This finding should be taken into account when evaluating severity-of-illness score performance.

Keywords: Simplified acute physiology score; Prognostic; Neoplasms; Intensive care units.

 

INTRODUCTION

Severity-of-illness scores are used in intensive care unit (ICU) settings worldwide for performance evaluation and monitoring, quality improvement and benchmarking.(1-3) Since their first description in the 1980s, many prognostic models have been developed. However, among them, the Simplified Acute Physiology Score 3 (SAPS 3) is the only one developed from a multinational cohort (16,784 patients from 35 countries).(4,5) In Brazil, Moralez et al. recently demonstrated that SAPS 3 (standard equation) remains the most accurate prognostic model.(6) However, the performance of severity-of-illness scores might be different in some institutions due to the case-mix and representativeness of subgroups, such as oncological patients.

Because intensivists are increasingly managing oncological patients,(7) studies evaluating the performance of these prognostic models in this subgroup of critically ill patients are welcomed. Although several studies have been published concerning this topic,(8-10) they were published almost ten years ago, and a well-known phenomenon that compromises the performance of these prognostic models is deterioration over time (mainly in calibration) as previously demonstrated.(8,11) None of these studies compared the model performance in oncological versus nononcological patients.

Therefore, our objectives in the present study were to evaluate the performance of SAPS 3 in patients with cancer admitted to a Brazilian ICU, compare the SAPS 3 performance of patients with and without cancer and to study time trends in SAPS 3 performance.

METHODS

This was a retrospective cohort study of all consecutive patients admitted to our medical-surgical ICU (a 30-bed unit at Hospital Sírio-Libanês, a private tertiary hospital with a dedicated oncology unit, located in São Paulo, Brazil) and was approved by the local ethics committee. The detailed description of our unit was previously published and did not change during the study period.(12) The exclusion criteria were age younger than 18 years and pregnancy. If the patients had more than one admission during the inclusion period, only the first admission was included. Some of the patients included in this study were also included in a previous analysis of our group regarding ICU readmission (1,702 patients).(12)

Our analysis used administrative data that were prospectively collected in a cloud-based software database (Sistema Epimed™) by trained nurses.(13) The study period was from January 1st, 2012 to July 31st, 2016. The oncological condition was defined as any patient admitted with an active solid cancer (current curative or palliative chemotherapy, radiotherapy, immunotherapy or surgery) in the last 12 months. Hematological patients were excluded because they usually have distinct characteristics compared with patients with solid tumors (e.g., a higher burden of active disease; required oncological treatment during ICU stay; prolonged duration of neutropenia; a higher intensity of immunosuppression; concomitant bone marrow transplant; and a higher incidence of specific complications, such as invasive mold fungal infections and cytomegalovirus infection).

The data recorded included age, sex, date of ICU admission, SAPS 3,(4,5) referring facility, admission diagnosis, surgical procedures before admission, Charlson index for comorbidities,(14) resource utilization during ICU stay (mechanical ventilation, vasoactive drugs or renal replacement therapy), oncological status (locoregional or metastatic) and hospital mortality. The SAPS 3 was calculated using data from the ICU admission. As recommended, missing values were coded as “normal” for each variable.(6)

Data analysis

The primary outcome was hospital mortality. Quantitative parametric data were presented as means ± standard deviation (SD), nonparametric data were presented as medians (25% - 75% interquartile range - IQR), and categorical variables were presented as percentages. Categorical variables were compared using chi-squared test.

The primary outcome was hospital mortality. The estimated mortality rate was calculated using the standard equation for the SAPS 3. SAPS 3 discrimination was evaluated using the area under the Receiver Operating Characteristic curve (AUROC). Comparisons between AUROCs were performed using the Delong method.(15) Calibration was assessed using the calibration belt method as described by the GiViTI group.(16) This method applies a generalized polynomial logistic function between the outcome and logit transformation of the estimated predicted probability, with the respective 95% and 80% confidence interval (CI) boundaries. A statistically significant deviation from the bisector (the line of perfect calibration) occurs when the 95%CI boundaries of the calibration belt do not include the bisector.(16) Standardized mortality rates (SMRs) with 95%CIs were calculated by dividing the observed by the predicted mortality rates. The Brier score is an overall performance measure that was calculated using a standard formula.(17) To study time trends in the SAPS 3 performance, we split the cohort into two subgroups by the ICU admission date (October 1st, 2014 to create two subgroups with similar sizes).

The data were analyzed using IBM SPSS Statistics, Version 21 and R (http://www.r-project.org). All the statistics were two-tailed, and a p value < 0.05 was considered statistically significant.

RESULTS

During the study period, 8,345 ICU admissions occurred. After excluding 846 readmissions and 945 hematological patients, 7,390 patients remained. However, 136 patients (1.8%) did not have a hospital discharge status in our database. Therefore, our final study cohort comprised 7,254 patients (41.5% had cancer). Oncological patients were younger, had a lower SAPS 3, were admitted more frequently after elective surgery, and had higher hospital mortality than nononcological patients (14.1% versus 10.6%, respectively; p < 0.001; Table 1).

Table 1 - General characteristics of the included patients
  All patients Oncological Nononcological p value
n 7.254 3.008 4.246  
Age (years) 66.2 ± 18.4 62.6 ± 16.6 68.7 ± 19.2 < 0.001
Male 3911 (53.9) 1711 (56.9) 2200 (51.8) < 0.001
SAPS 3 41 (32 - 51) 38 (30 - 51) 42 (33 - 51) < 0.001
Admission type       < 0.001
    Medical 3.744 (51.6) 1.014 (33.7)  2.730 (64.3)  
    Emergency surgery 640 (8.8) 176 (5.9) 464 (10.9)  
    Elective surgery 2.867 (39.5) 1.817 (60.4) 1.050 (24.7)  
Length of hospital stay before ICU admission (days) 1 (0 - 2) 1 (0 - 3) 1 (0 - 2) < 0.001
Charlson comorbidity index 2 (0 - 3) 2 (2 - 6) 0 (0 - 1) < 0.001
Performance status*       < 0.001
    Independent 3.698 (80.3) 1.652 (89.2) 2.046 (74.3)  
    Assistance required 547 (11.9) 128 (6.9) 419 (15.2)  
    Restricted/bedridden 362 (7.9) 73 (3.9) 289 (10.5)  
Most common primary tumor site        
    Colon --- 538 (17.9) ---  
    Central nervous system --- 471 (15.7) ---  
    Pancreas --- 265 (8.8) ---  
    Lung --- 260 (8.6) ---  
Type of cancer        
    Locoregional --- 2.147 (71.4) ---  
    Metastatic --- 861 (28.6) ---  
Main ICU admission diagnosis        
    Gastrointestinal 1.575 (21.7) 975 (32.4) 520 (12.2)  
    Neurological 1.385 (19.1) 662 (22.0) 723 (17.0)  
    Infection 1.368 (18.9) 393 (13.1) 975 (23.0)  
Mechanical ventilation during ICU stay 1.398 (19.3) 573 (19) 825 (19.5) 0.67
Vasopressors during ICU stay 2.054 (28.3) 874 (29.1) 1.180 (27.8) 0.25
Renal replacement therapy during ICU stay 368 (5.1) 97 (3.2) 271 (6.4) < 0.001
Hospital mortality 876 (12.1) 424 (14.1) 452 (10.6) < 0.001

SAPS 3 - Simplified Acute Physiology Score 3; ICU - intensive care unit.

* Among 4,607 patients for whom data could be ascertained. Results expressed as means ± standard deviation, n (%) or medians (25% - 75% interquartile range).

Table 1 - General characteristics of the included patients

Using the standard SAPS 3 predictive equation, the overall SMR was 0.93 (95%CI 0.88 - 0.98), with a Brier score of 0.088 and an AUROC of 0.82 (95%CI 0.81-0.83). Calibration belt analysis showed that SAPS 3 tended to overestimate mortality in our entire cohort of patients (Table 2).

Table 2 - Comparison of SAPS 3 performance in all patients and subgroups of oncological and nononcological patients
  SMR
(95%CI)
Discrimination AUROC (95%CI) Calibration* Precision
Over the bisector 95%CI Under the bisector 95%CI Brier score
All patients 0.93 (0.88 - 0.98) 0.82 (0.81 - 0.83) Never 0.18 - 0.80 0.088
Oncological patients 1.09 (1.01 - 1.17) 0.85 (0.83 - 0.86) Never Never 0.096
Nononcological patients 0.82 (0.75 - 0.89) 0.79 (0.78 - 0.81) Never 0.08 - 0.43 0.082

SMR - standardized mortality rate; 95%CI - 95% confidence interval; AUROC - area under the Receiver Operating Characteristic curve.

* Calibration described as the bisector deviation intervals using the calibration belt method.

Table 2 - Comparison of SAPS 3 performance in all patients and subgroups of oncological and nononcological patients

Comparison of the SAPS 3 performance in oncological patients and nononcological patients

The discrimination of SAPS 3 was higher for oncological patients than for nononcological patients (p < 0.001; Table 2). Calibration belt analysis demonstrated that, in oncological patients, no miscalibration was observed. However, in nononcological patients, SAPS 3 overestimated mortality in those with low-middle predicted risk (Figure 1).

Figure 1 - Calibration belt for Simplified Acute Physiology Score 3 in oncological (A) and nononcological patients (B), described as bisector deviation intervals. The predicted mortality intervals at which the calibration belt significantly deviates from the bisector and the 80% and 95% confidence levels are described in the lower right region of the plots. SAPS 3 - Simplified Acute Physiology Score 3.

Time trend evaluation

The frequency of oncological patients did not change between the two time periods evaluated (1,498 of 3,542 patients [42.3%] versus 1,510 of 3,712 patients [40.7%], respectively; p = 0.16). The discrimination of SAPS 3 was not affected within subgroups by time period (Figure 2). Calibration belt analysis showed no miscalibration in the oncological subgroup of patients within either period. However, in the nononcological group, undercalibration was observed in the first period, and overcalibration was observed in the second period (Figure 3 andTable 3).

Figure 2 - Area under the Receiver Operating Characteristic curves for Simplified Acute Physiology Score 3 in oncological (upper part) and nononcological patients (lower part). Subgroups were divided by the time period of intensive care unit admission (first and second half).

Figure 3 - Calibration belt for Simplified Acute Physiology Score 3 in oncological (A and C) and nononcological patients (B and D), described as bisector deviation intervals. The predicted mortality intervals at which the calibration belt significantly deviates from the bisector and the 80% and 95% confidence levels are described in the lower right region of the plots. Subgroups were divided by the time period of intensive care unit admission (first period A and B; second period C and D). SAPS 3 - Simplified Acute Physiology Score 3.

Table 3 - SAPS 3 performance for oncological versus nononcological patients in the first and second periods
  SMR
(95%CI)
Discrimination
AUROC (95%CI)
Calibration* Precision
Over the bisector
95%CI
Under the bisector
95%CI
Brier score
First period (2012 - 2014)          
    Oncological patients 1.27 (1.15 - 1.40) 0.85 (0.83 - 0.87) Never Never 0.10
    Nononcological patients 1.08 (0.95 - 1.20) 0.79 (0.77 - 0.80) 0.00 - 0.04 Never 0.09
Second period (2014 - 2016)          
    Oncological patients 1.01 (0.91 - 1.12) 0.85 (0.83 - 0.87) Never Never 0.095
    Nononcological patients 0.71 (0.62 - 0.81) 0.81 (0.79 - 0.83) Never 0.02 - 0.74 0.075

SMR - standardized mortality rate; 95%CI - 95% confidence interval; AUROC - area under the Receiver Operating Characteristic curve.

* Calibration described as the bisector deviation intervals using the calibration belt method.

Table 3 - SAPS 3 performance for oncological versus nononcological patients in the first and second periods

DISCUSSION

In this retrospective Brazilian cohort, we evaluated SAPS 3 performance in both oncological and nononcological critically ill patients. We found that (1) the SAPS 3 discrimination and calibration were accurate in our cohort for oncological patients; (2) the discrimination was greater and calibration was more accurate for oncological patients than for nononcological patients; and (3) the calibration was affected by the time period only for nononcological patients.

Recently, Moralez et al. published the largest validation study of severity-of-illness scores in Brazil using contemporary data from a multicenter cohort.(6) They showed that the SAPS 3 standard equation was accurate in predicting outcomes in our country, supporting the national initiative from the Associação de Medicina Intensiva Brasileira (AMIB) for benchmarking units.(18) However, the case-mix variation between units may lead to performance deterioration.(19) A particular subgroup of interest is critically ill oncological patients because 15% of the patients admitted to European ICUs have cancer.(20)

Some previous publications evaluated SAPS 3 performance in oncological patients.(8-10) Overall, these studies suggest that the measure is accurate for both discrimination and calibration, as demonstrated by our data. This finding is reassuring because oncological patients are usually poorly represented in development cohorts (8% in the original SAPS 3 cohort).(4,5) Nevertheless, the hospital mortality of critically ill patients with cancer depends on acuity rather than on the presence and characteristics of the malignant disease itself.(20,21) Therefore, the general severity of illness scores might capture most of the short-term prognosis in this population.

A novel approach of the present study was the comparison of SAPS 3 performance in oncological versus nononcological patients. We observed that this score was superior in oncological patients admitted to the ICU in terms of discrimination and calibration. To the best of our knowledge, this is the first study to compare these subgroups in the same cohort. The reasons for this observation are unclear but highlight the importance of case-mix differences between units and how this might affect model comparisons.

Another relevant observation was the effect of time on performance components. Calibration is particularly susceptible to time trends. Zimmerman et al. showed that the discrimination of Acute Physiology And Chronic Health Evaluation (APACHE) IV was robust and changed little throughout the evaluated time period but that calibration deteriorated in the general ICU population.(11) Soares et al. also demonstrated this deterioration in their temporal analysis of SAPS 3 calibration in oncological patients.(8) Again, we observed different effects of the time trend analysis in oncological compared with nononcological patients. No miscalibration was observed in oncological patients in either period. However, not only did miscalibration occur in nononcological patients but it also moved from underestimation to overestimation from the first to the latest period evaluated. We speculated that clinical practice changes affected mainly nononcological patients, but this suggestion may be oversimplistic. Selection bias at ICU preadmission and end-of-life practice may also affect calibration.

Our study has some strengths, such as a large sample size, which allowed the performance of hypothesis-generating subgroup analyses (oncological versus nononcological patients) not previously performed. Our study also evaluated a score validated in the same settings as in previous studies in oncological patients as well as in one of the largest external validation cohorts.(6) However, this study also has limitations. First, it is a single-center cohort; our results reflect local practice and our case mix (which limits the generalizability of our study). Nevertheless, some of our results agree with previous study findings. Second, the study had a long period of data collection. Although this factor might affect the overall performance evaluation, it was required for the time period analysis. Finally, end-of-life decisions were not systematically annotated in our database.

CONCLUSION

We observed that SAPS 3 discrimination was better for oncological than for nononcological patients and that SAPS3 showed no relevant deviations from optimal calibration in oncological patients. SAPS 3 performance (mainly calibration) varied over time differently according to the oncological status in our single-center cohort.

FUNDING

This work was supported by a local grant from Hospital Sírio-Libanês, São Paulo (Brazil).

REFERENCES

Salluh JI, Soares M. ICU severity of illness scores: APACHE, SAPS and MPM. Curr Opin Crit Care. 2014;20(5):557-65. Link DOI
Le Gall JR. The use of severity scores in the intensive care unit. Intensive Care Med. 2005;31(12):1618-23. Link DOI
Keegan MT, Soares M. What every intensivist should know about prognostic scoring systems and risk-adjusted mortality. Rev Bras Ter Intensiva. 2016;28(3):264-9. Link DOI
Metnitz PG, Moreno RP, Almeida E, Jordan B, Bauer P, Campos RA, Iapichino G, Edbrooke D, Capuzzo M, Le Gall JR; SAPS 3 Investigators. SAPS 3--From evaluation of the patient to evaluation of the intensive care unit. Part 1: Objectives, methods and cohort description. Intensive Care Med. 2005;31(10):1336-44. Link DOI
Moreno RP, Metnitz PG, Almeida E, Jordan B, Bauer P, Campos RA, Iapichino G, Edbrooke D, Capuzzo M, Le Gall JR; SAPS 3 Investigators. SAPS 3--From evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med. 2005;31(10):1345-55.
Moralez GM, Rabello LS, Lisboa TC, Lima MD, Hatum RM, De Marco FV, Alves A, Pinto JE, de Araújo HB, Ramos GV, Silva AR, Fernandes GC, Faria GB, Mendes CL, Ramos Filho RA, de Souza VP, do Brasil PE, Bozza FA, Salluh JI, Soares M; ORCHESTRA Study Investigators. External validation of SAPS 3 and MPM0-III scores in 48,816 patients from 72 Brazilian ICUs. Ann Intensive Care. 2017;7(1):53. Link DOI
Azoulay E, Soares M, Darmon M, Benoit D, Pastores S, Afessa B. Intensive care of the cancer patient: recent achievements and remaining challenges. Ann Intensive Care. 2011;1(1):5.
Soares M, Salluh JI. Validation of the SAPS 3 admission prognostic model in patients with cancer in need of intensive care. Intensive Care Med. 2006;32(11):1839-44. Link DOI
Soares M, Silva UV, Teles JM, Silva E, Caruso P, Lobo SM, et al. Validation of four prognostic scores in patients with cancer admitted to Brazilian intensive care units: results from a prospective multicenter study. Intensive Care Med. 2010;36(7):1188-95. Link DOI
Xing X, Gao Y, Wang H, Huang C, Qu S, Zhang H, et al. Performance of three prognostic models in patients with cancer in need of intensive care in a medical center in China. PLoS One. 2015;10(6):e0131329.
Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients. Crit Care Med. 2006;34(5):1297-310. Link DOI
Rodrigues CM, Pires EM, Feliciano JP, Vieira JM Jr, Taniguchi LU. Admission factors associated with intensive care unit readmission in critically ill oncohematological patients: a retrospective cohort study. Rev Bras Ter Intensiva. 2016;28(1):33-9. Link DOI
Zampieri FG, Soares M, Borges LP, Salluh JI, Ranzani OT. The Epimed Monitor ICU Database(r): a cloud-based national registry for adult intensive care unit patients in Brazil. Rev Bras Ter Intensiva. 2017;29(4):418-26. Link DOI
Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373-83. Link DOI
DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837-45. Link DOI
Finazzi S, Poole D, Luciani D, Cogo PE, Bertolini G. Calibration belt for quality-of-care assessment based on dichotomous outcomes. PLoS One. 2011;6(2):e16110.
Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128-38. Link DOI
Associação de Medicina Intensiva Brasileira (AMIB). UTIs brasileiras. Registro Nacional de Terapia Intensiva [internet]. [citado 2019 out 01]. Disponível em http://www.utisbrasileiras.com.br/
Keegan MT, Gajic O, Afessa B. Severity of illness scoring systems in the intensive care unit. Crit Care Med. 2011;39(1):163-9. Link DOI
Taccone FS, Artigas AA, Sprung CL, Moreno R, Sakr Y, Vincent JL. Characteristics and outcomes of cancer patients in European ICUs. Crit Care. 2009;13(1):R15. Link DOI
Schellongowski P, Sperr WR, Wohlfarth P, Knoebl P, Rabitsch W, Watzke HH, et al. Critically ill patients with cancer: chances and limitations of intensive care medicine-a narrative review. ESMO Open. 2016;1(5):e000018. Link DOI

Submission On-line

Indexed in

Scopus

SciELO

LILACS

Associação de Medicina Intensiva Brasileira - AMIB

Rua Arminda nº 93 - 7º andar - Vila Olímpia - São Paulo, SP, Brasil - Tel./Fax: (55 11) 5089-2642 | e-mail: [email protected]

Cookie Policy

GN1 - Systems and Publications