To verify the performance of the Net Promoter Score (NPS) as a tool to assess parental satisfaction in pediatric intensive care units (PICUs).
MethodsThe authors conducted an observational cross-sectional multicenter study in the PICUs of 5 hospitals in Brazil. Eligible participants were all parents or legal guardians of PICU-admitted children, aged 18 years or over. The NPS was administered together with the EMpowerment of PArents in THe Intensive Care (EMPATHIC-30), used as the gold standard, and a sociodemographic questionnaire. For analysis, the results were dichotomized into values greater than or equal to the median of the tests. The associations between the 2 tools were evaluated and the distribution of their results was compared.
ResultsThe parents or legal guardians of 78 PICU-admitted children were interviewed. Of the respondents, 85% were women and 62% were in a private hospital. The median NPS was 10 (IQR, 10-10), and the median EMPATHIC-30 score was 5.7 (IQR, 5.4-5.9). Compared with the gold standard, the NPS had a sensitivity of 100% at all cutoff points, except at cutoff 10, where the sensitivity was slightly lower (97.5%). As for specificity, NPS performance was poorer, with values ranging from 0% (NPS ≥ 5) to 47.4% (NPS = 10).
ConclusionsNPS proved to be a sensitive tool to assess parental satisfaction, but with poor ability to identify dissatisfied users in the sample.
The growing concern of health organizations with users’ well-being has increased the accountability for the quality of care provided by institutions, where the patient or family plays a more active role in the assessment of services. As a result, there has been an increasing demand for tools to assess users’ satisfaction and expectations.1,2
By raising awareness of the importance of parents in promoting child health and well-being,1 institutions have encouraged the continued stay of family members during the child's hospitalization.3,4 In pediatric intensive care units (PICUs), parents’ experiences are often related to the admission of a critically ill child, being perceived as a moment of crisis, in which the needs and stressors involved in the process become apparent.3,5 Therefore, it is important that PICU staff are aware of parents’ expectations and experiences so that the care directed to them can be optimized to meet their needs and increase satisfaction with the care provided.4,6,7
The Net Promoter Score (NPS), a method developed by Reichheld,8 is an internationally recognized tool for measuring the level of customer satisfaction and loyalty in different industries.9 Increasingly, NPS is gaining ground in healthcare facilities and being used as a research tool in several specialties.9,10 The test aims to assess user experience using only one question: “How likely are you to recommend our service to a friend or family?”, which is scored on a scale from 0 to 10.8 According to the author, respondents scoring a 9 or a 10 are “promoters” of this service. Individuals answering 7 or 8 are considered indifferent (“passives”), whereas those answering 0 to 6 are probably dissatisfied with the service, being defined as “detractors”.8,11
Due to its simplicity, reliability, and flexibility, NPS is widely used and has revolutionized the way user satisfaction is assessed.9,12 Initially, it was introduced as an alternative to traditional survey methods, which are time-consuming, complicated, and difficult to analyze and understand in practice.13 Therefore, the present study's objective was to further investigate the value of NPS in the assessment of parental satisfaction and experiences in PICUs.
MethodsThe authors conducted a cross-sectional multicenter study of parents or legal guardians of children admitted to 5 PICUs in Brazil, in the states of Rio Grande do Sul and Sao Paulo. These PICUs admit patients from hospital wards, emergency departments, and operating rooms, as well as those transferred from other hospitals and health care units. The PICUs have between 10 and 19 beds and all have physicians on duty, nurses, psychologists, nutritionists, physiotherapists, and social workers. The hospitals participating in this study are centers that provide care through the publicly funded Brazilian Unified Health System, private health insurance, or out-of-pocket.
The authors used a sample by interest. Eligible participants were all parents or legal guardians of a child admitted to the PICU for at least 24 hours who accompanied their child during PICU stay full-time or part-time. Parents whose son/daughter died in the PICU were not eligible for the study. Parents of a child readmitted to the same PICU more than once were interviewed only once.
Instrument administrationA sociodemographic questionnaire was administered, followed by the NPS. Subsequently, as the gold standard, the EMpowerment of PArents in THe Intensive Care (EMPATHIC-30) was applied by trained researchers. The satisfaction survey was administered within 72 hours of patient discharge from the PICU when they were transferred to a ward.
The Pediatric Index of Mortality 2 (PIM2) was used to assess severity on admission.14 Patients with a PIM2 greater than or equal to the 75th percentile of the interquartile range of the sample were considered more severe.
For the NPS, the participants were asked to rate on a scale from 0 to 10 how likely they would be to recommend the service. Subsequently, the participants were asked the second question proposed by the test,11 with the purpose of understanding the reason or reasons for their score: “What is the primary reason for the score you just gave us?”, which is an open-ended question to be answered in the participant's own words. The NPS can be found in the supplementary material.
The authors used the EMPATHIC-30, developed to measure parental satisfaction with the quality of care in PICUs,15 which is currently considered a gold standard in the field in several countries around the world.16,17 At present, this is the only questionnaire validated and translated for use in PICUs in the Brazilian population.18 It is a self-report questionnaire consisting of 30 items divided into 5 domains: information, care and treatment, organization, parent participation, and professional attitude.15 Respondents were asked to rate each question on a scale from 1 to 6. For “not applicable” questions, the participants were instructed to assign a score of 0. The questionnaire can be found in the supplementary material.
Data analysisIn order to assess the predictive criterion validity of the NPS, and because there is no cutoff point to define satisfaction/dissatisfaction using the gold standard, participants with an EMPATHIC-30 score greater than or equal to the median of the test in the sample were defined as satisfied. The EMPATHIC-30 total score was calculated by dividing the sum of the scores given to each of the questions by the number of test items. Domain scores were calculated by dividing the sum of the scores given to each of the statements in that domain by the number of items that comprise the domain. The sensitivity and specificity of each NPS cutoff point were calculated based on this classification. A receiver operating characteristic (ROC) curve was used to determine the NPS cutoff point with the best performance, in terms of sensitivity and specificity, in identifying individuals satisfied with the care provided. NPS accuracy was obtained by calculating the area under the ROC curve. Positive and negative likelihood ratios were calculated for each NPS cutoff point. Stata 12.0 was used for all calculations.
Content analysisThe content analysis technique proposed by Laurence Bardin19 was used for the treatment of NPS qualitative data. The questions and scope of each of the 5 domains of the EMPATHIC-30 were used as a guiding concept to adapt the material to the purpose of the study. The processing of qualitative data is described in the supplementary material.
Ethical approvalThe study was approved by the Research Committees of all 5 participating centers. All respondents provided written informed consent before answering the questionnaires.
ResultsThe study included 78 PICU-admitted children (Table 1), with a median age of 3.7 (IQR, 0.9-10) years, of whom 53% were male. Children admitted via private health insurance accounted for 68% of the sample, and a large part of the patients (n = 49) were on their first admission to a PICU. Most children (n = 41) had chronic diseases, accounting for 53% of the sample. A total of 26% of patients with a PIM2 greater than 1.25% were classified as “more severe.” The median PICU length of stay was 4 days, with most patients (70%) staying in the PICU for less than 7 days.
Characteristics of children, according to independent variables.
n, number of patients; PIM2, Pediatric Index of Mortality 2; %, percentage; PICU, pediatric intensive care unit.
Regarding the characteristics of respondents (Table 2), the majority (n = 66) were mothers of the patients, with a median age of 38 years; 56% had completed higher education and more than one-third (36%) had a monthly household income greater than one thousand seven hundred and seventy dollars (average income of Brazilians is around 429 dollars). Most respondents (25%) were women who reported being stay-at-home parents; the remaining respondents reported different occupations, and only 4 refused to report their occupation. Almost all respondents (n = 71) accompanied their child daily in the PICU, staying with the child full-time (24 hours) during hospitalization (78%).
Characteristics of respondents, according to independent variables.
n, number of patients; %, percentage; PICU, pediatric intensive care unit.
The NPS assessment of parental satisfaction showed that 85% of respondents would recommend the study PICU. Only 76 respondents answered the second NPS question. The responses were grouped according to their common characteristics. Subsequently, the concepts of each of the 5 domains of the EMPATHIC-30 were used as final categories. The domains were divided into positive and negative to be in accordance with the intention of the participant's response. Content analysis showed that most participants (n = 50) reported the quality of care and treatment (n = 32) and professional attitude (n = 18) as positive factors. Organization (as a negative factor) (n = 6) was the most commonly mentioned issue when respondents reported the problems that they had encountered in the PICU (n = 10).
The mean NPS score was 9.6 (SD, 0.9), and the median NPS score was 10 (IQR, 10-10). No participant scored less than 5 or equal to 7, and the vast majority (76%) scored a 10 (maximum score) when answering the first NPS question (Table 3). The EMPATHIC-30 total score ranged from 3.2 to 6, with a mean of 5.5 (SD, 0.9) and a median of 5.7 (IQR, 5.4-5.9). The prevalence of individuals completely satisfied with PICU services, i.e., with a maximum EMPATHIC-30 score, was 13% (95% CI, 5.5%-20.5%). All EMPATHIC-30 domains had a mean of 5.0 points or more (Table 3). The median values (6 points) of the ‘information’ and ‘professional attitude’ domains indicate that half or more of the respondents were very satisfied with the service in that category.
Measures of central tendency and dispersion of NPS and EMPATHIC-30 total scores and by domains.
n, number of patients; NPS, Net Promoter Score; EMPATHIC-30, EMpowerment of PArents in THe Intensive Care.
Table 4 shows the sensitivity and specificity of each NPS cutoff point, compared with the EMPATHIC-30 total score dichotomized according to the median value as ≥ 5.7 points (n = 40 satisfied respondents) or < 5.7 points (n = 38 dissatisfied respondents). The sensitivity of scores 5-9 to identify respondents satisfied with the care provided in the PICU was 100% (respondents who gave these scores were able to correctly identify all those who scored ≥ 5.7 on the EMPATHIC-30). However, specificity (i.e., the ability to correctly identify those who were dissatisfied) was 0% at cutoff ≥ 5, increasing slowly up to cutoff ≥ 8, then reaching 23.7% and 47.4%, respectively, at cutoffs ≥ 9 and 10. NPS had an accuracy of 72.7% (95% CI, 61.8%-0.82%). At all cutoff points, both positive and negative likelihood ratios were very close to 1, indicating poor test performance in correctly distinguishing between satisfied and dissatisfied service users.
NPS sensitivity and specificity compared with EMPATHIC-30 total score.
| NPS* | N | Sensitivity | Specificity | LR+ | LR- | |
|---|---|---|---|---|---|---|
| ≥ 5 | 78 | 100 | 0 | 1.0 | 0 | |
| ≥ 6 | 77 | 100 | 2.6 | 1.03 | 0 | |
| ≥ 8 | 76 | 100 | 5.3 | 1.06 | 0 | |
| ≥ 9 | 69 | 100 | 23.7 | 1.31 | 0.05 | |
| 10 | 59 | 97.5 | 47.4 | 1.85 | 1.00 |
NPS had similar performance across all 5 domains, with accuracy ranging from 62% to 77%. At all cutoff points, the sensitivity of the NPS to identify satisfied individuals in each of the 5 domains of the EMPATHIC-30 ranged from 90% to 100%. However, specificity was low overall, except at cutoff 10, where it was approximately 40% in the ‘information,’ ‘care and treatment,’ ‘organization,’ and ‘parent participation’ domains, and 60.9% in the ‘professional attitude’ domain. Both positive and negative likelihood ratios were very low, indicating that NPS had a low predictive ability to correctly identify individuals satisfied and dissatisfied, respectively, with the care associated with each EMPATHIC-30 domain in this sample.
DiscussionThis is the first study to test the use of NPS as a tool to assess parental satisfaction in PICUs and general pediatrics. No other study was found in the literature that evaluated the sensitivity and specificity of this instrument. In the present analysis, compared with the gold standard (EMPATHIC-30), NPS was highly sensitive at all cutoff points analyzed, but with low specificity. This finding may be due to the fact that all participants scored greater than or equal to 5 on the test, probably reflecting the small sample size. Specificity was slightly higher only at NPS cutoff 10 for the ‘professional attitude’ domain, reaching 60.9%.
Several studies9,10,12,20 using NPS to assess user satisfaction with the provided health services found a satisfaction rate ranging from 60% to 96%. In the present survey, the NPS overall score was 85%, which indicates high user satisfaction with the services provided in the PICU.
Given the increasing use of NPS to assess patient satisfaction with health services and the lack of data in the literature on its validity in this field, some studies have applied it together with other tools in order to allow a comparison of the approaches. In the Netherlands, based on data from patients from 6 hospitals, the NPS was shown to be moderate to strongly correlated with 3 other constructs in the ‘promoters’ and ‘detractors’ categories.13 Likewise, Raassens and Haans21 also found corresponding scores between promoters and online word-of-mouth (eWOM) behavior. In an outpatient clinic in California, NPS was able to correctly identify the users who were satisfied with the service.9 The present findings are consistent with these results, as the NPS could sensitively detect the parents or legal guardians who were satisfied with the care provided in the PICUs.
Since its creation, the NPS has been questioned for its straightforward way of assessing customer satisfaction, especially when used in the health field. However, some authors9,10,20 consider that this metric provides a broad but valuable view of users’ perceptions, especially when associated with the second descriptive question proposed by the tool. A survey conducted in 18 private hospitals in Brazil,22 using the NPS, found the quality of care and staff as the main factor for customer satisfaction. In the present study, the content analysis showed that ‘care and treatment’ was the category most often cited as a positive factor for parental satisfaction, followed by ‘professional attitude.’ This is in line with the results of a study conducted in England,10 which identified that the relationship between care and treatment is an important factor associated with patient satisfaction, being already consolidated as a core feature of most current therapeutic models. Regarding dissatisfaction, ‘organization’ was the category most cited by NPS respondents in the present study. Likewise, on the EMPATHIC-30, the ‘organization’ domain obtained the lowest score compared with the other domains.
In the present study's survey, the authors obtained few critical responses (13%), occasionally with the NPS score being different from the descriptive response. This also occurred in a study of 188 participants in India,20 which suggested changing the second question of the tool to: “What could be done to improve the service?”. In doing so, the authors believe that users will feel more comfortable in providing feedback, as the question would be asked in a less personal way, thus possibly increasing the rate of responses with negative content as well as suggestions for service improvement.20
Initially, some studies questioned whether the NPS question “would you recommend...” would be suitable for the health field.12 In the authors’ experience of administering the test, the authors noticed that there were no uncertainties or lack of understanding in relation to answering the question, as the participants always answered it easily. This was also observed in previous surveys conducted in the United States and Europe.9,10 In addition, when applied to functional illiterate populations, no problems were found in relation to the applicability and acceptability of the tool.20 These authors have even reported that this was the most answered question in their satisfaction survey, allowing it to reach a larger number of respondents.
For the most part (62% of the sample), data were collected in private hospitals, which already use the NPS as a form of an overall assessment of the institution. The hospitals often send the survey by email days after hospital discharge, which results in a low response rate, especially in pediatric units. This also hinders an accurate assessment of parental satisfaction with care. These data are consistent with those of a study conducted in the United States,12 which obtained a response rate of 100% on the NPS when applied at the time of patient discharge, against a response rate of 27% when applied via email or phone call. Therefore, it is important to collect the data from patients while in the hospital, before they are discharged home.
In the present study, most respondents (n = 44) had completed higher education and had a monthly household income greater than or equal to 9 minimum monthly salaries (45%). These factors are associated with a high level of requirement due to high expectations of the service provided.23 In the present study, the 2 lowest NPS scores (5 and 6), as well as the 3 lowest EMPATHIC-30 scores (3.8, 3.2, and 3.9), were given by respondents within this profile. However, in general, the parents were satisfied with the care provided in the PICUs. This may be related to the fact that most patients in the present sample were considered less severe and did not require intubation (n = 62). The fragile health status of a patient or family member is associated with a lower level of satisfaction and willingness to recommend services.10
According to a survey linked to the Brazilian National Association of Private Hospitals,22 customers classified as promoters are willing to spend more time going to the hospital of their choice, with a return rate 3 times higher than that of detractors. The survey results also highlighted the importance of recommendations from satisfied users, as 60% of patients reported that the primary reason for choosing the hospital was the recommendation from a physician, family member, or friend.
This study has some limitations, such as the small sample size, which resulted in a partial assessment of NPS performance. Also, this study is subject to a type of information bias, known as courtesy bias. In an attempt to minimize this bias, the interviews were conducted outside the PICU premises. Since the authors didn't interview parents whose children died in the PICU, the authors can consider it as a selection bias.
Compared with EMPATHIC-30, NPS showed high sensitivity to correctly detect users satisfied with PICU care, but with low specificity in identifying dissatisfied users in this analysis. However, both tests obtained similar results regarding the main factors associated with parental satisfaction and dissatisfaction. The authors consider NPS a promising test because it is sensitive in assessing parental satisfaction in the PICU. Nevertheless, the authors believe that studies with larger sample sizes are needed to further assess the value of using NPS in PICUs.
FundingThis study was funded in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – Brasil (CAPES) – Código de Financiamento 001.
