Application of the Pediatric Index of Mortality 2 in pediatric patients with complex chronic conditions

Abstract

Objective

To evaluate the performance of the Pediatric Index of Mortality 2 (PIM2) in a pediatric intensive care unit (PICU) with a high prevalence of patients with complex chronic conditions (CCCs), and compare the performance between patients with and without CCCs.

Methods

A prospective cohort study was conducted in a PICU in Brazil, with patients admitted between 2009 and 2011. The performance was evaluated through discrimination and calibration. Discrimination was assessed by calculating the area under the ROC curve, and calibration was determined using the Hosmer-Lemeshow goodness-of-fit test.

Results

A total of 677 patients were included in the study, of which 83.9% had a CCC. Overall mortality was 9.7%, with a trend of higher mortality among patients with CCCs when compared to patients without CCCs (10.3% vs. 6.4%, p = 0.27), but with no difference in the mean probability of death estimated by PIM2 (5.9% vs. 5.6%, p = 0.5). Discrimination was considered adequate in the general population (0.840) and in patients with and without CCCs (0.826 and 0.944). Calibration was considered inadequate in the general population and in patients with CCCs (p < 0.0001 and p < 0.0001), but it was considered adequate in patients without CCCs (p = 0.527).

Conclusions

PIM2 showed poor performance in patients with CCCs and in the general population. This result may be secondary to differences in the characteristics between the study samples (high prevalence of patients with CCCs); the performance of the PIM2 should not be ruled out.

Resumo

Objetivo

Avaliar o desempenho do Pediatric Index of Mortality 2 (PIM2) em Unidade de Terapia Intensiva Pediátrica (UTIP) com alta prevalência de pacientes com condições crônicas complexas (CCC), e comparar o desempenho entre pacientes com e sem CCC.

Métodos

Estudo de coorte prospectivo, realizado em UTIP no Brasil, com pacientes admitidos entre 2009 e 2011. O desempenho foi avaliado através da discriminação e calibração. A discriminação foi avaliada através do cálculo da área sob a curva ROC e a calibração através do teste de ajuste de Hosmer-Lemeshow.

Resultados

Foram incluídos no estudo 677 pacientes, com 83,9% deles apresentando uma CCC. A mortalidade geral foi 9,7%, com tendência de maior mortalidade entre pacientes com CCC quando comparados com pacientes sem CCC (10,3% vs. 6,4%; p = 0,27), porém sem diferença na média de probabilidade de morte estimada pelo PIM2 (5,9% vs. 5,6%; p = 0,5). A discriminação foi considerada adequada na população geral (0,840) e nos pacientes com e sem CCC (0,826 e 0,944). A calibração foi considerada inadequada na população geral e nos pacientes com CCC (p < 0,0001 e p < 0,0001), porém foi considerada adequada nos pacientes sem CCC (p = 0,527).

Conclusões

O PIM2 apresentou desempenho inadequado nos pacientes com CCC e na população geral. O desempenho inadequado pode ser secundário à diferença das características entre as amostras do estudo (alta prevalência de pacientes com CCC), e o desenvolvimento do escore não pode ser descartado.

Introduction

Children with complex chronic conditions (CCCs) are characterized by the presence of any medical condition in which the pathology duration is expected to last at least 12 months (except when the patient dies), affecting any body system or organ severely enough to require care from a pediatric specialty, and probably requiring hospitalization in a tertiary hospital.¹

There is an increased prevalence of children with CCCs admitted to the pediatric intensive care unit (PICU), with the observation that patients with CCCs have higher risk of mortality and increased length of stay in the PICU when compared to patients without CCCs.²

Outcome prediction scores are tools that quantify the patient's clinical condition severity and predict mortality, and are considered important components to measure and improve the quality of care offered in the PICU.³ The two most commonly used outcome prediction scores in the pediatric population, the Pediatric Index of Mortality 2 (PIM2)⁴ and Pediatric Risk of Mortality III (PRISM III),⁵ include fewer chronic conditions as prediction variables in their models.

Based on the concept that outcome prediction scores might not work properly in a setting where patient characteristics or diagnoses are substantially different from those of patients used in the development of the score,⁴ and the observation of the increased prevalence of children admitted to the PICU with CCCs in recent years, who had higher mortality rates than patients without CCCs,² it was considered important to evaluate the performance of outcome prediction scores in a setting that would reflect these changes.

The aim of this study was to evaluate the performance of the PIM2 score for predicting outcomes in a PICU with high prevalence of patients with CCCs, and to compare the score performance between patients with and without CCCs.

Methods

This study was approved by the Research Ethics Committee of Universidade Federal de Minas Gerais, Belo Horizonte, Brazil, and an informed consent was obtained for all participants.

A prospective cohort study was conductions in the PICU of the Hospital das Clínicas. The PICU has ten beds, with approximately 370 admissions per year, both clinical and surgical, including cardiac surgery.

All patients admitted to the PICU between February 1, 2009 and January 31, 2011 were included in the study. Patients younger than 30 days or older than 18 years, patients who died within the first two hours of admission, patients with suspected brain death at admission that was later confirmed, patients considered as having no chance of curative treatment (NCCT), and patients whose parents/guardians did not sign the consent form were excluded from the study.

The study variables were those used to characterize the patients, death probability calculated by PIM2, and outcome variables. The data from the analyzed variables were collected by the researchers from the medical and nursing records.

For patient characterization, the following data were evaluated: age, gender, presence of CCC, type of CCC, type of admission (medical or surgical), type of clinical pathology, type of surgical pathology, elective or non-elective admission condition, and use of invasive mechanical ventilation.

The presence of CCC was recorded when the patient had any medical condition characterized by pathology duration of at least 12 months (except when the patient died) that affected any body system or organ severely enough to require care from a pediatric specialty, and probably requiring hospitalization in a tertiary hospital.¹ The patient's CCC was classified according to the classification developed by Feudtner et al.:¹ neuromuscular malformations; cardiovascular malformations; respiratory, renal, gastrointestinal, hematological or immunological, and metabolic; other genetic or congenital defects; and neoplasms.

The patient was classified as having a clinical pathology in the presence of cardiovascular, respiratory, renal, hepatic, neurological, post-cardiorespiratory arrest, or oncological/hematological diseases. The patient was classified as having a surgical pathology in the presence of cardiovascular, neurological, general surgery, orthopedic, transplants, or cardiac catheterization procedures. The admission was considered elective when the PICU admission or surgery could be postponed for more than six hours without adverse effects upon the patient. The probability of death for each patient was calculated using the instructions provided by the authors of the PIM2 score.⁴ The analyzed outcome variables were PICU discharge event (discharge or death) and length of hospitalization.

Descriptive statistics were used to characterize the patients, and the PIM2 performance was assessed by measures of score discrimination and calibration. Discrimination was assessed by calculating the area under the receiver operator characteristic (ROC) curve,⁶ and was considered adequate when the area was > 0.7.⁷ Calibration was assessed using the Hosmer-Lemeshow goodness-of-fit test,⁶ and was considered appropriate when the p-value of the test was > 0.05.⁷ In the Hosmer-Lemeshow test, patients were divided into ten groups with increasing risk probability and a similar number of cases in each group.

The comparison between the mortality rate in the study population with the mortality predicted by PIM2 score was performed by dividing the observed mortality rate by the expected mortality rate (as calculated by the score), termed the standardized mortality rate (SMR).⁸ The SMR was shown with its 95% confidence interval (95% CI). If the 95% CI of the SMR included 1, its performance was considered medium; if the 95% CI of the SMR had an upper limit < 1, its performance was considered good; and if the 95% CI of the SMR had a lower limit > 1, its performance was considered poor.⁹ All analyses were performed using SPSS Statistics for Windows release 12 for Windows (SPSS, Chicago, Illinois, United States).

Results

During the study period, 756 patients were admitted to the PICU, of whom 79 were excluded: 63 patients were younger than 30 days old, ten patients were considered NCCT, four patients died within the first two hours after admission to the unit, and two patients had a suspected diagnosis of brain death on admission, which was later confirmed. A total of 677 patients were included in the study, of whom 568 (83.9%) had a CCC.

The main clinical characteristics of the general population and the subgroups of patients with and without CCC are shown in Table 1. The median length of stay, mortality rates, and mean probability of death estimated by PIM2 and SMR of the general population and subgroups are shown in Table 1. When comparing patients with and without CCC, no statistically significant difference was observed in the median length of stay in PICU (3 days vs. 3 days; p = 0.84), in mortality (10.3% vs. 6.4%, p = 0.27), and in the mean probability of death estimated by PIM2 (5.9% vs. 5.6%, p = 0.5).

Table 1.

Characteristics of the general population and subgroups evaluated.

Characteristics (%)	General population (677 patients)	Patients with CCC (568 patients)	Patients without CCC (109 patients)
Age (months)	59 (13-122)^a	64 (17-123)^a	18 (5-82)^a
Male gender	349 (51.5%)	287 (50.5%)	62 (56.8%)
Complex chronic condition	568 (83.9%)
Clinical admission	320 (47.3%)	248 (43.6%)	72 (66%)
Surgical admission	357 (52.7%)	320 (56.3%)	37 (33.9%)
Invasive mechanical ventilation	370 (54.6%)	322 (53.8%)	48 (44%)
Non-elective admission	320 (47.2%)	243 (42.8%)	77 (70.6%)
Types of CCC
Neoplasias		171 (30.1%)
Cardiovascular malformations		154 (27.1%)
Neuromuscular malformations		64 (11.2%)
Gastrointestinal		55 (9.6%)
Renal		49 (8.6%)
Respiratory		27 (4.7%)
Other genetic or congenital defects		20 (3.5%)
Hematological or immunological		19 (3.3%)
Metabolic		9 (1.5%)
Reason for clinical admission
Respiratory	134 (41.8%)	95 (38.3%)	39 (54.1%)
Circulatory	76 (23.7%)	69 (27.8%)	7 (9.7%)
Neurological	32 (10%)	22 (8.8%)	10 (13.8%)
Others	25 (7.8%)	21 (8.4%)	4 (5.5%)
Hepatic	19 (5.9%)	14 (5.6%)	5 (6.9%)
Post-CRA	18 (5.6%)	14 (5.6%)	4 (5.5%)
Onco-hematologic	9 (2.8%)	9 (3.6%)	0 (0%)
Renal	7 (2.1%)	4 (1.6%)	3 (4.1%)
Type of surgery performed
Pediatric general surgery	105 (29.4%)	86 (26.8%)	19 (51.3%)
Neurosurgery	91 (25.4%)	76 (23.7%)	15 (40.5%)
Cardiovascular surgery	84 (23.5%)	83 (25.9%)	1 (2.7%)
Hemodynamics	39 (10.9%)	39 (12.1%)	0 (0%)
Transplant	17 (4.7%)	17 (5.3%)	0 (0%)
Others	12 (3.3%)	11 (3.4%)	1 (2.7%)
Orthopedic	9 (2.5%)	8 (2.5%)	1 (2.7%)
Length of stay	4 days^b	3 days^b	3 days^b
Estimated mortality by PIM2	5.9%	5.9%	5.6%
Mortality	9.7%	10.3%	6.4%
SMR	1.65 (1.26-2.04)^a	1.75 (1.31-2.19)^a	1.14 (0.29-1.99)^a

CCC, complex chronic condition; CRA, cardiac arrest; PIM2, pediatric index of mortality 2; SMR, standardized mortality rate.

a

95% confidence interval.

b

median.

The score discrimination assessed by the area under the ROC curve was considered adequate both in the general population and in the subgroups of patients with and without CCC (Table 2). The score calibration assessed by the Hosmer-Lemeshow goodness-of-fit test was considered inadequate in the general population and in the subgroup of patients with CCC, but it was considered appropriate in the subgroup of patients without CCC (Table 2).

Table 2.

Performance of PIM2 in the general population and subgroups.

Categories	Area under ROC curve	HL test X^2	HL test p
General	0.840 (0.810-0.867)	43.405	< 0.0001
Complex chronic condition	0.826 (0.792-0.856)	36.38	< 0.0001
No complex chronic condition	0.944 (0.883-0.979)	7.089	0.527

ROC, receiver operator characteristic; X², chi-squared; HL, Hosmer-Lemeshow.

Discussion

The PIM2 score was chosen for evaluation, as it is considered to be user-friendly and efficient, and it is public domain.¹⁰ The PIM2 is the updated version of the Pediatric Index of Mortality that was published in 1997, and has been used extensively since then.^11,12

Since its publication in 2003, several studies have been published evaluating the score performance in populations and scenarios that were different from those used in the score development study, with most studies showing adequate score performance.¹⁰

In the present study, 83.9% of patients admitted to the PICU had one CCC, higher values than those found in a recent study that found a prevalence of 53% in a cohort that included patients from 54 PICUs in the United States.² The high proportion of patients with CCCs that was observed in the present study and in the recent literature may be related to advances in medical care in recent years, especially in pediatric and neonatal intensive care, which have resulted in improvement in survival rates of patients with previously unmanageable diseases, leading to an increase of patients with CCCs, who have higher risk of hospitalization in the PICU than the general population.^13,14

The overall mortality of the study population was 9.7%, consistent with the currently observed rates in the PICUs, which range from 5-10%.¹⁵ There was a trend toward higher mortality in patients with CCCs, when compared with patients without CCCs, but without statistical significance.

The observation that patients with chronic diseases have higher mortality rates when admitted to the PICU has been reported in recent literature. Wölfler et al., in a study evaluating the PIM2, observed a high mortality in patients with chronic diseases compared with the general population (15.6% vs. 5.2%).¹⁶ Odetola et al. observed a significantly higher mortality in patients with comorbidities when compared to patients without comorbidities admitted to the PICUs in the United States in 1997 (12.5% vs. 8.6%) and in 2006 (10.8% vs. 7.8%).¹⁷

Edwards et al. also observed a higher mortality rate among patients with CCCs, when compared to patients without CCCs admitted to PICUs in the United States in 2008 (3.9% vs. 2.2%).²

However, this trend of increased mortality in patients with CCCs in the present study was not verified by the probability of death estimated by PIM2, which was similar in patients with and without CCCs (5.9% vs. 5.6%, p = 0.5) and may indicate poor score performance. This observation was also reported by Edwards et al., who demonstrated that patients with CCCs had significantly higher risk of mortality in the PICU than that predicted by the PIM2.²

Therefore, it can be hypothesized that the differences in characteristics of patients with CCCs in the present study in relation to the population used for the score development could explain the poor performance of the PIM2, as reported by the authors who affirm that the differences in characteristics and diagnoses of a population, when compared with the population used in the score development, can result in poor performance of the score when evaluated in a new scenario.⁴ It is important to emphasize that there are no data on the prevalence of CCCs in the population used for the development of the PIM2, but some chronic diseases are outcome variables, such as severe combined immunodeficiency and neurodegenerative disease.⁴

When evaluating the PIM2 performance, the score was considered inadequate for both the total sample and the subgroup of patients with CCCs as a result of inefficient calibration measured by the Hosmer-Lemeshow goodness-of-fit test, although the score showed adequate performance in patients without CCCs.

The interpretation of the Hosmer-Lemeshow test results when scores are applied to new populations should be carried out carefully, due to the fact that possible external factors, such as sample size, can influence the test's p-value.¹⁸

Marcin et al. reported that when the calibration is evaluated in a sample independent from the score development sample, inadequate calibration may result from an inadequate score, but it can also be an indicator of differences in quality of care offered between the study samples and those used for the score development.³

In situations in which the evaluation of the score conducted in an independent sample shows inadequate performance, some questions should be answered:¹⁹ Is the difference in performance related to the staff responsible for patient care? Do the diagnostic and therapeutic options have similar accessibility and use? Is the health system, of which the PICU is part, efficient? Was data collection adequate? Is the score not appropriate for a different mix of cases?

The study design does not allow for the answering of the first three questions, which are related to the quality of health care offered in the studied PICU. However, it may be noted that the unit is part of a public health system in a developing country, facing difficulties related to funding, structure, personnel, and operational organization, which can negatively affect the quality of health care offered to patients.

The fourth question can be answered through the methodology used to collect data from the study, which followed the guidelines proposed by the authors of the PIM2.⁴ The fifth question, which asks whether PIM2 may not be appropriate for a different mix of cases, is the hypothesis raised in the present study. The poor performance of the score in patients with CCCs and in the total sample, which had 83.9% of patients with CCCs, may be secondary to differences in the characteristics between the study samples and those used in the score development.

Murphy-Filkins et al. reported that changes in the demographic characteristics of patients and in the prevalence of diseases alter the mix of cases, which may influence score performance.²⁰ Perhaps, with the recently observed increase in the prevalence of CCCs among patients admitted to the PICU and the observation that these patients have a higher mortality rate than the general population, it will be necessary that future scores for outcome prediction use this condition when their models are being constructed.

The SMR of 1.65 (95% CI: 1.26 to 2.04) for the general population and of 1.75 (95% CI: 1.31 to 2.19) for the subgroup of patients with CCCs indicated that the quality of health care offered at the PICU for these patients during the study period was worse than the quality of health care offered by the PICUs that participated in the PIM2 development study in the period between 1997 and 1999. However, the SMR of 1.14 (0.29 to 1.99) for patients without CCCs indicated that the quality of health care offered at the PICU for this subgroup in the study period was similar to the quality of health care offered by the PICUs that participated in PIM2 development study between 1997 and 1999.

The SMR is the main indicator of health care quality in the PICU, and is part of the first set of quality indicators that were submitted to the Joint Commission on Accreditation of Healthcare Organizations, published in 2005.²¹ However, the accuracy of this indicator depends on the capacity of the score in predicting mortality in the studied population;⁸ thus, in addition to the hypothesis of inadequate quality as the cause of the high SMR in the overall study population and in the subgroup of patients with CCCs, the hypothesis that the PIM2 may not be adequate for the studied population cannot be ruled out.

This study has some limitations. The principal limitation is the fact that the study was conducted in a single PICU, making it difficult to prove the hypothesis and subsequently, the generalizability of the findings. Operating conditions (equipment, medications, and staff) in the studied PICU may not be similar to those of PICUs in developed countries, which may have influenced the results. Finally, the statistical analysis may have been influenced by the relatively small sample size.

The PIM2 showed poor performance in the subgroup of patients with CCCs and the overall study population, which had 83.9% of patients with CCCs. Although the poor performance of the score may be secondary to the quality of health care offered by the PICU, the hypothesis that the difference in characteristics between the study sample and the sample used for the score development is responsible for inadequate performance of PIM2 cannot be ruled out. With the increasing prevalence of CCC patients admitted to PICUs observed in recent decades, it may be necessary that future outcome prediction scores consider this condition when creating their models.

Conflicts of interest

The authors declare no conflicts of interest.