Validity of the computerized version of the pediatric triage system CLARIPED for emergency care

Abstract

Objective

To evaluate the validity of the computerized version of the pediatric triage system CLARIPED.

Methods

Prospective, observational study in a tertiary emergency department (ED) from Jan-2018 to Jan-2019. A convenience sample of patients aged 0-18 years who had computerized triage and outcome variables registered. Construct validity was assessed through the association between urgency levels and patient outcomes. Sensitivity, specificity, positive and negative predictive values (PPV and NPV), undertriage, and overtriage rates were assessed.

Results

19,122 of 38,321 visits were analyzed. The urgency levels were: RED (emergency) 0.02%, ORANGE (high urgency) 3.21%, YELLOW (urgency) 35.69%, GREEN (low urgency) 58.46%, and BLUE (no urgency) 2.62%. The following outcomes increased according to the increase in the level of urgency: hospital admission (0.4%, 0.6%, 3.1%, 11.9% and 25%), stay in the ED observation room (2.8%, 4.7%, 15.9%, 40.4%, 50%), ≥ 2 diagnostic or therapeutic resources (7.8%, 16.5%, 33.7%, 60.6%, 75%), and ED length of stay in minutes (18, 24, 67, 120, 260). The odds of using ≥ 2 resources or being hospitalized were significantly greater in the most urgent patients (Red, Orange, and Yellow) compared to the least urgent (Green and Blue): OR 7.88 (95%CI: 5.35-11.6) and OR 2.85 (95%CI: 2.63-3.09), respectively. The sensitivity to identify urgency was 0.82 (95%CI: 0.77-0.85); specificity, 0.62 (95%CI: 0.61-0.6; NPV, 0.99 (95%CI: 0.99-1.00); overtriage rate, 4.28% and undertriage, 18.41%.

Conclusion

The computerized version of CLARIPED is a valid and safe pediatric triage system, with a significant correlation with clinical outcomes, good sensitivity, and low undertriage rate.

Introduction

Emergency services are subject to periods of excessive patient demand, which exceed their capacity to respond at an appropriate time. In the pediatric population, especially in Low and Middle-Income Countries (LMIC), an early assessment and approach reduce morbidity and mortality due to the high prevalence of time-sensitive medical emergencies.¹ Over the years, several triage systems have been developed and improved to reduce long waiting queues for assistance, optimize flows, improve the quality of care for customers and the satisfaction of the multidisciplinary team.² Currently, some validated pediatric triage systems are originating in different countries, such as the Manchester Triage System (MTS, United Kingdom), the Canadian Triage and Acuity Scale (CTAS, Canada), the Emergency Severity Index (ESI, United States of America) and the Classification of Risk in Pediatrics (CLARIPED, Brazil).^3,4 However, the validity of these systems varies according to the patient load, the severity of illness case mix, available infrastructure, and available resources.⁵ The greater the socioeconomic and epidemiological diversity, the greater the difference in the performance of these tools. Although the most commonly used systems have shown good validity in the pediatric population in their countries of origin, evidence shows that this does not occur in LMIC without an adequate cross-cultural adaptation.^1,2,4,5

The validity studies carried out in LMIC are limited and of low to moderate quality and include heterogeneous instruments, such as the Pediatric Early Warning Score – PEWS,^1,6 the Emergency Triage Assessment and Treatment – ETAT,^1,7-9 a three-level system advocated by WHO for low-income countries, and the South African Pediatric Screening Scale (pSATS), a four-level system validated in South Africa.¹⁰ Thus, there is a lack of evidence to support the use of the more traditional triage systems in the pediatric population in this group of countries.

The CLARIPED (Classification of Risk in Pediatrics), until now the only pediatric triage system developed in Brazil, was validated in its first version, for manual use, in a sample of 1416 patients from 0 to 15 years old, in the emergency department (ED) of a tertiary hospital in Rio de Janeiro, Brazil, in 2013. It proved to be a valid, reliable, and very safe tool, with high sensitivity (89%; 95% CI: 78%-95%) to identify very urgent patients, a low percentage of undertriage (7.4%), a consistent association between urgency categories and ED outcomes (hospitalization, number of diagnostic or therapeutic resources used, and length of stay in ED) and substantial interobserver agreement [kw² = 0.75 (95% CI: 0.74-0.79)].¹¹ Since then, a computerized version of CLARIPED has been developed, adding more operational efficiency to the triage process, being adopted in public and private pediatric services, of secondary and tertiary complexity, in southeastern Brazil. Also, some modifications were included based on the demands arising from the use of the system. Thus, this study aimed to evaluate the validity of the computerized version of the CLARIPED pediatric triage system.

Materials and methods

Study design, patient selection, and setting

This was a prospective, observational study, conducted in the pediatric ED of a private tertiary hospital in Rio de Janeiro, Brazil. The pediatric ED is physically separated from the adult ED and is one of the most important high-complexity services for pediatric emergency care in the city, attending about 40,000 patients per year. Although it is not a reference center for trauma, organ transplants, congenital heart disease, or cancer patients, the hospital and the ED have physical and human resources to attend to patients from almost all specialties. The study population was a convenience sample of patients aged 0 to 18 years, who had a complete medical record with the study variables, from January 1, 2018, to January 31, 2019. The study was approved by the Research Ethics Committee of the D'Or Institute for Research and Education (IDOR) under No. 2,665,936.

Data collection, processing, and statical analysis

The main differences between the computerized version and the old manual version of CLARIPED (and the reasons for the changes) are presented in a table in the Supplementary File. The risk classification variables, obtained from the CLARIPED computerized system, and the outcome variables, obtained from a specific form routinely filled out after emergency medical care, were later associated and treated anonymously. To determine the sample's representativeness, some variables on the total population assisted in the same period were obtained through a computerized hospital registration system: sex, age group, time, and day of the week. In the absence of a gold standard, the construct validity was assessed by the association between the urgency levels assigned by the CLARIPED system and clinical outcomes, proxies of urgency, in the emergency department (patient's destination, number of diagnostic or therapeutic resources used, and length of stay assessed from the beginning of medical consultation up to discharge from the emergency room). The statistical significance of the variation in outcome frequencies at each level of urgency was analyzed using the chi-square test for trends. Logistic regressions were performed to estimate the odds ratios (OR) for hospitalization and use of two or more resources (adjusted for age, admission date, and day of the week for assistance) comparing the most urgency levels (RED, ORANGE, and YELLOW) with less urgent levels (BLUE and GREEN). CLARIPED sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) to discriminate more urgent patients were estimated based on the outcome “hospital admission”. The rates of undertriage and overtriage were calculated based on the following definitions, respectively: patients classified as less urgent (BLUE and GREEN) who were hospitalized and patients classified as most urgent (YELLOW, ORANGE and RED) who were discharged home after medical consultation, with no admission to the observation room or hospitalization. A significance level of 5% was adopted for all analyses, and 95% confidence intervals were computed when applicable. The statistical software R 4.0.3 was used for data analysis.

Results

The authors of the present study included 19,122 patients out of the 38,321 patients seen in the pediatric ED in the study period. The most frequent age groups were 1 to 4 years (42.8%) and 5 to 11 years (34.2%); periods with the highest number of visits were 12 to 18h (34.2%) and 18 to 0h (32.8%); the percentage of visits on each day of the week ranged from 13.5% (Friday) to 15.8% (Sunday); respiratory diseases accounted for 42% of the reasons for consultation, followed by general and nonspecific causes (40.7%). The distribution of the urgency categories was: 2.6% BLUE (no urgency), 58.5% GREEN (low urgency), 35.7% YELLOW (urgency), 3.2% ORANGE (high urgency), and 0.02% RED (emergency). The hospitalization rate was 1.8%. There was no relevant difference between the total population's characteristics and the studied population (Table 1).

Table 1.

Characteristics of the study population and of the total population attended in the pediatric emergency department in the study period (January-2018 to January-2019).

Characteristics	Study population	%	Total population	%
Total – n (%)	19122	(100)	38186	(100)
Sex
Male	7253	(52.4)^a	20118	(52.7)^a
Female	6584	(47.6)^a	17944	(47.0)^a
Not available	5285		124
Age range
< 1 year	1950	(10.2)	1391	(3.6)
1 - 4 years	8182	(42.8)	17546	(46.0)
5 -11 years	6549	(34.2)	13960	(36.6)
12 - 17 years	2296	(12.0)	5283	(13.8)
Not available	145	(0.8)	6	(0.0)
Visit periods – n (%)
0h - 6h	895	(4.7)	2082	(5.5)
6h - 12h	5379	(28.2)	10336	(27.1)
12h - 18h	6528	(34.2)	13192	(34.5)
18h as 0h	6259	(32.8)	12576	(32.9)
Days of the week – n (%)
Sunday	3023	(15.8)	5368	(14.1)
Monday	2862	(15.0)	6209	(16.3)
Tuesday	2690	(14.1)	5760	(15.1)
Wednesday	2575	(13.5)	5506	(14.4)
Thursday	2640	(13.8)	5490	(14.4)
Friday	2582	(13.5)	4923	(12.9)
Saturday	2750	(14.4)	4930	(12.9)
Main diagnoses – n (%)
Upper respiratory disease	6596	34,5
Lower respiratory disease	2017	10,5
Gastrointestinal	2853	14,9
Ear and eye diseases	1764	9,2
Trauma and external causes	1622	8,5
Other infectious diseases	1478	7,7
Skin	1131	5,9
Genitourinary	552	2,9
Osteoarticular	197	1,0
Neurologic and behavioral	132	0,7
Others^b	188	1,0
Unspecified symptoms^c	595	3,1
NI	20	0,1
Risk Classification – n (%)
Blue	501	(2.62)
Green	11178	(58.46)
Yellow	6825	(35.69)
Orange	614	(3.21)
Red	4	(0.02)
Hospitalization – n (%)	353	(1.8)	762	(1.9)

a

Percentage of female and male sex, not including the not available data in the denominator.

b

Others: cardiovascular (n = 73); hematologic/neoplasia (n = 15); neonatal diseases (n = 15); endocrine/metabolic diseases (n = 9); congenital malformations (n = 5); medical guidance (n = 71).

c

Unspecified isolated symptoms: fever (n = 480); headache (n = 56); malaise/fatigue (n = 3); unspecific pain (n = 15); adenomegaly (n = 22); syncope (n = 6); edema (n = 2); fluid/food intake disorders (n = 3); unspecific symptoms (n = 8).

The frequency of emergency marker outcomes, admission to the emergency observation room, hospitalization, and use of two or more resources significantly increased according to the gradually greater level of urgency assigned by the CLARIPED (p < 0.001). On the other hand, the frequency of outcomes such as being discharged home after medical consultation and the use of fewer than two resources decreased significantly with the increase in urgency level (p < 0.001) (Table 2). The median length of stay in the emergency department was 35 minutes (IQR 25;75), with a significant increase according to the increase in urgency level (p-value < 0.001).

Table 2.

Distribution of the frequency of destination outcomes and number of resources used according to the urgency level assigned by CLARIPED.

Risk Classification	Destination				Resources		Total
	Discharge from hospital^a	Admission Observation room^b	Hospitalization	NI	< 2 resources	≥ 2 resources
Blue	43186 %	142.8 %	20.4 %	5410.8 %	46292.2 %	397.8 %	501100 %
Green	956785.6 %	5284.7 %	630.6 %	10209.1 %	933983.5 %	183916.5 %	11178100 %
Yellow	497372.9 %	108215.9 %	2143.1 %	5568.1 %	452266.3 %	230333.7 %	6825100 %
Orange	24239,4 %	24840,4 %	7311,9 %	518,3 %	24239,4 %	37260,6 %	614100 %
Red	125 %	250 %	125 %	00 %	125 %	375 %	4100 %
p-value^c	< 0.001	< 0.001	< 0.001	0.009	< 0.001	< 0.001
Total	1521479.6 %	18749.8 %	3531.8 %	16818.8 %	1456676.2 %	455623.8 %	19122100 %

NI, not informed.

a

Discharge from hospital (without observation or hospitalization).

b

Admission to the emergency observation room (without hospitalization).

c

p-value < 0,001 (chi-square for trend).

The odds ratio (OR) for the use of two or more resources, from the most urgent levels (YELLOW, ORANGE, and RED) to the least urgent (BLUE and GREEN, the reference), adjusted for age, assistance time, and day of the week, was 2.85 (95% CI: 2.63-3.06); for hospitalization, it was 7.88 (95% CI: 5.35 11.6) (Table 3). The sensitivity and specificity of CLARIPED to identify the most urgent patients, based on the outcome “hospitalization” were, respectively, 0.82 (95% CI: 0.77-0.85) and 0.62 (95% CI: 0.61 0.62). The negative predictive value (NPV) was 0.99 (95% CI: 0.99-1.00), the overtriage rate was 4.28%, and the undertriage was 18.41% (Table 4).

Table 3.

Odds ratio for resource utilization and hospitalization in the study population using CLARIPED.

CLARIPED (Urgency Level)	OR - simple (CI 95%)	OR - adjusted^a (CI 95%)
≥ 2 resources
Red, Orange and Yellow	2.93 (2.74-3.14)	2.85 (2.63-3.09)
Green and Blue	1.0	1.0
Hospitalization
Red, Orange and Yellow	7.13 (5.44-9.35)	7.88 (5.35-11.60)
Green and Blue	1.0	1.0

CI, confidence interval; OR, Odds ratio.

a

Adjusted for age, time and day of the week.

Table 4.

Sensitivity, specificity, PPV and NPV of CLARIPED to discriminate high urgency and rates of overtriage and undertriage.

	Hospitalized	Non hospitalized	Sensitivity (CI 95%)	Specificity (CI95%)	PPV (CI 95%)	NPV (CI 95%)	Overtriage (CI 95%)	Undertriage (CI 95%)
Most Urgent LevelsRED, ORANGE, and YELLOW	288	6548	0.82(0.77-0.85)	0.62(0.61-0.62)	0.04(0.04-0.04)	0.99(0.99-1.00)	18.41(14.72-22.79)	34.28(33.53-35.04)
Less Urgent LevelsGREEN, BLUE	65	10540

CI, confidence interval.

Sensitivity - number of most urgent and hospitalized patients (red, orange, and yellow)/total number of hospitalized patients.

Specificity - number of less urgent and not admitted patients (green and blue)/ total number of not admitted patients.

PPV (positive predictive value) - number of hospitalized patients / numbers of most urgent patients (red, orange, and yellow).

NPV (negative predictive value) - number of outpatients / numbers of less urgent patients.

Undertriage - number of less urgent patients (blue and green) admitted / total number of hospitalized patients x 100.

Overtriage - number of most urgent patients (red, orange, and yellow) discharged home / total number of patients discharged after prescription x 100.

Discussion

The computerized CLARIPED proved to be a safe triage system, with good sensitivity to identify very urgent patients, a low rate of undertriage, and a strong association with clinical outcomes. The present study results are consistent with those found in a previous study of the first manual version of CLARIPED.¹¹ These findings corroborate the tool's validity and support its implementation in pediatric emergency care with similar populations.

The distribution of levels of urgency in the study population is similar to that found in other validity studies on different triage systems, conducted in tertiary hospitals of developed countries, using consecutive samples, which describe 70 to 90% of the visits in the levels 3 or 4.^12-17

The strong association between increased urgency levels and increased frequency of severity outcomes (use of two or more resources, admission to the emergency observation room, length of stay in the emergency department, and hospitalization) demonstrated the convergent construct validity of the CLARIPED system. These results are like those observed in other triage systems and superior in some respects. This study noted a decreasing and discriminative gradient in the hospitalization percentage (25% to 0.4%), from the highest to the lowest urgency level. Studies on PaedCTAS and ESI in their countries of origin have also shown discrimination gradients. Still, the percentages are higher at all levels of urgency, ranging from 100 to 60% at the most urgent patients to 2 to 1% at the least urgent.^12-17 Even higher percentages of hospitalization were observed in Israel with PaedCTAS, reaching 12.32% at the least urgent level¹⁸). In Iran, ESI showed even higher percentages at levels 1 and 2 (the most urgent patients) but with low discrimination capacity among 3, 4, and 5 (less urgent patients).¹⁹ A 3.5% hospitalization percentage was found among less urgent patients in a study on MTS safety,²⁰ while the SATS non-urgent category had a hospital admission rate of 4.7%.¹⁰ In all these studies, the percentages of hospitalization at low urgency levels are not negligible and may impair these instruments’ safety. It is noteworthy that this study found only 0.4% to 0.6% of hospitalizations at the less urgent levels (BLUE and GREEN, respectively). However, populations with more severe profiles, as suggested by the highest rates of general hospitalization, 8 to 8.6% in Canadian studies and 24.4% in Israel, compared to the 1.8% rate in the present study, may explain, in part, these differences in results. Different hospitalization policies, different application of urgency criteria in populations with varying degrees of severity, level of team understanding, and expertise in using the triage tool are also possible explanations. Considering all these aspects, the comparative analysis of the hospitalization percentages at different levels of urgency between the present study and similar studies from other systems shows that CLARIPED has a good discriminative capacity and a better safety profile. It also showed a very high negative predictive value and a percentage of hospitalization of less urgent patients (BLUE and GREEN) significantly below that found in other systems validity studies.

In this study, the odds of hospitalization in the most urgent categories (RED, ORANGE, and YELLOW) were almost eight times higher than in the less urgent categories (BLUE and GREEN). Studies evaluating CTAS and MTS^13,20 found odds ratios of 4.94 and 2.5 to 3.5 (depending on the subgroup), respectively, comparing the high urgency levels (level 1 and 2) to level 3 (urgent), considered as the reference. In this study, the YELLOW category (urgent) showed admission rates to the emergency observation room and hospitalization rates considerably higher than the GREEN and BLUE categories (less urgent and non-urgent). The YELLOW and GREEN categories (levels 3 and 4) correspond to the two largest care groups in most pediatric emergency services globally, accounting for about 70 to 90% of the visits. In this study, the percentages of visits YELLOW and GREEN amounted to 94.15%. Thus, although the main focus of triage systems is given to the rapid identification of high-urgency patients (RED and ORANGE, levels 1 and 2), the safety of the triage process needs to have safe discrimination between urgent patients (YELLOW) and less urgent patients (GREEN), which determined the use of this cutoff for the OR, sensitivity, specificity, PPV, VPN, overtriage and undertriage rates in this study.

The ED length of stay showed a discriminative gradient between the highest and lowest levels of urgency, ranging from 260 to 18 minutes. In this calculation, the authors did not consider the time between the patient's arrival and the start of medical care, as the waiting time after triage is inversely proportional to the urgency and could unbalance the calculation of this outcome. Studies on PedCTAS^12,13 and ESI v.4¹⁵ measured the time from arrival to discharge from ED and found less discriminatory distributions.

There is no recommendation for safe limits of sensitivity, undertriage, and overtriage. However, as a priority, a triage tool should present good sensitivity and a low undertriage rate to ensure its safety.¹¹ A high rate of overtriage means that many patients with low urgency were incorrectly classified as high urgency, which reduces the effectiveness of the triage tool, increasing the contingent of patients to be admitted and the waiting time for the high urgency patients.⁴ A high undertriage rate showed that many high-urgency patients were incorrectly classified as low-urgency, reducing the process's safety. In the present study, the CLARIPED tool showed high sensitivity to discriminate between most urgent patients (YELLOW, ORANGE, and RED) from less urgent patients (GREEN and BLUE), based on the hospitalization outcome. The undertriage and overtriage rates were moderate, and the VPN was high. Using the same cutoff point, the CLARIPED previous validity study showed more moderate values (sensitivity 0.74, specificity 0.62, undertriage rate 7.4%, and overtriage rate 59.1%), showing the progress of the current computerized version. The sensitivity to identify high urgency (levels 1 and 2) of other triage systems, using the outcome “admission to the intensive care unit” were: 0.67 to 0.93 with PaedCTAS,⁴ 0.71 with MTS,⁴ and 0.91 with SATs.¹⁰

The present study has some limitations. Although the convenience sample was huge (n = 19122) and higher than most validity studies in pediatrics (median = 1496 visits),⁴ it comprised around 50% of the visits during the study period. It could not be representative of the general population. However, the comparative analysis of the study population with the general population, in the same period, did not show relevant differences regarding age, sex, day of the week, and service hours and also did not identify a systematic pattern of losses (hour or day of the week of service).

On the other hand, more important than the sample's representativeness would be ensuring enough visits at all levels of urgency. Few validity retrospective studies used sampling strategies to ensure a pre-determined number of patients at each level of urgency.^2,15 Like most studies on pediatric emergency triage, the present study had very few visits at the RED level because this level is not frequent in most general pediatric EDs worldwide. This could be pointed out as another limitation. Nevertheless, the authors identified that almost 3% of the patients in this ED were seen directly by the medical team without going through the triage. They were immediately identified at the emergency reception with signs of high urgency (they would be RED and ORANGE levels in CLARIPED). The most frequent diagnoses in these patients were seizures, head trauma, burns, polytrauma, bruising wounds with active bleeding, acute laryngitis, anaphylaxis, and dehydration associated with constant vomiting. Based on this scenario, the authors believe that the present study had a lower proportion of visits at the ORANGE and RED levels than the real proportion at this ED. This loss would more likely attenuate the CLARIPED outcome gradients and sensitivity to detect more urgent patients.

Finally, one could argue that not blinding the emergency physicians to the assigned triage level could influence the outcomes. Blinding could only be achieved with a retrospective design. In this pragmatic prospective observational study, the ED professionals were unaware of the study, and the researchers did not interfere with the team's care routine. The interest of the study was to investigate what happens in real life when the authors believe that the physician's knowledge of the urgency level previously assigned does not influence his decision-making.

Another limitation of this and all other studies on the validity of triage systems would be the absence of a gold standard, which would allow an assessment of criterion validity, with greater methodological homogeneity and the possibility of a better comparison between the performance of the different tools. Some studies on MTS^2,21,22 used a reference standard developed by specialists defining the “real” urgency, based on the combination of several outcomes, offering more objective and significant criteria for assessing validity. In contrast, there is no evidence of the adequacy and validity of this reference standard. Although the first CLARIPED validity study (manual version) also used an adapted reference standard, like the MTS studies, in the present study, it was decided to evaluate the construct validity through associations with each of the outcomes separately to make the results comparable to most other studies.

The results of this study emphasize the validity of CLARIPED, characterizing it as a Brazilian tool that performs well in pediatric emergency services, with an excellent discriminative capacity between the levels of urgency. It is the only pediatric risk classification system with validity evidence in the Brazilian pediatric population, supporting its implementation in pediatric emergency services in this country and others with similar characteristics.