To develop and validate entrustable professional activities (EPAs) for the training of pediatric residents on topics that interface with pediatric surgical areas in the Brazilian context.
MethodsThe study was conducted in two phases. In the first phase, experts were oriented and contextualized, and they were responsible for developing the initial list of EPAs. In the second phase, the Delphi technique was applied in three rounds: the first for consensus, the second for selection according to relevance and agreement, and the third for validation and detailing of the EPAs.
ResultsIn the first phase of the study, 9 experts listed 88 EPAs, which were applied in the Delphi method. In the first round of Delphi, the consensus of these experts defined 31 EPAs, with CVI ≥ 0.80, and ICC of 0.893 (95 % CI 0.823–0.945). In the second round, 25 coordinators of Medical Residency Programs selected 17 EPAs by agreement and relevance (CVI ≥ 0.80, and ICC of 0.851–95 % CI 0.753 to 0.924). In the third round, 50 preceptors from all over Brazil validated 14 EPAs with CVI ≥ 0.965 and ICC 0.866 (95 % CI 0.804–0.915), which were organized and detailed into 7 final EPAs.
ConclusionSeven pediatric surgery EPAs were developed, consensualized, selected, and validated by experts for the work of pediatricians in Brazil through the Delphi method. The great participation and interest of medical residency preceptors with a wide geographical coverage in Brazil were strong points of this study, and these EPAs can be applied, reviewed, and updated.
The 21st-century medical resident is immersed in a socio-cultural reality vastly different from that of the early 20th century when Osler and Halsted founded residency programs. In this context, the competency-based medical education (CBME) model emerged in the early 1990s as a framework for undergraduate and postgraduate medical education [1,2]. This model suggests that professionals should develop specific and comprehensive competencies, including knowledge, skills, and attitudes, that are durable, teachable, measurable, related to professional activities, and connected to other individual competencies desirable for excellence in their training [1–3]. CBME defines objectives to be achieved at the end of training and has been widely adopted in medical schools and residency programs worldwide [2,3]. However, CBME has been criticized for being overly theoretical, difficult to verify in practice, and having limited applicability in the individual assessment of trainees [3].
To address these challenges and provide a more practical teaching tool, Dutch physician and educator Theodorus Jan (Olle) Ten Cate introduced the concept of Entrustable Professional Activities (EPAs) in 2005 [4]. EPAs are essential tasks that professionals must be able to perform independently by the end of training, serving as a bridge between competencies and clinical practice [4,5]. When implemented effectively, EPAs allow students, instructors, and even patients to have a clear understanding of the physician’s current scope of practice [5,6].
According to Ten Cate, EPAs do not replace competencies but rather translate them into clinical practice, allowing the training physician progress toward specialization, to independently care for the patient [5,7]. When this model is conducted clearly, both students and instructors, as well as patients themselves, can better understand what the physician is or is not qualified to do [8–9].
Pediatrics is one of the many medical fields where most professionals are trained through residency programs, which are facing significant challenges in ensuring high-quality training [10,11]. In this scenario, competency matrices with their respective EPAs have been developed and validated internationally, both for general pediatrics and pediatric subspecialty residency programs [11–14]. However, few well-designed studies have been conducted on the subject, as the concepts of EPAs are still, in their essence, poorly understood [7,14].
In Brazil, interest in EPAs has been growing among medical educators, especially at the undergraduate level, and is an increasingly present topic in forums, courses, congresses, and publications [15–16]. However, its application in the training of resident physicians is still in its early stages [16]. In Pediatrics, the authors can praise the pioneering work of Costa and collaborators, who recently validated EPAs for pediatric intensivists in Brazil [17].
In this context, the competency matrix for pediatric residency programs, published by the National Medical Residency Commission in 2016 and implemented in 2019 [18], determines that the resident physician must acquire knowledge, skills, and attitudes necessary for managing the most prevalent pediatric surgical diseases [18]. This directive underscores the need to enhance resident training through the development of a curricular model that incorporates EPAs related to pediatric surgery, tailored to general pediatricians – a gap that has yet to be addressed.
Therefore, this unprecedented study in Brazil proposes, through the Delphi method, to develop and validate EPAs for the training of pediatric residents in topics that interface with pediatric surgical areas, while still aligned with the Brazilian competency matrix. This pioneering work seeks to fill a gap in national medical education, contributing to the improvement of the quality of teaching and care in Pediatrics.
MethodsDesign, setting, and participantsThis study was conducted in two phases to establish Pediatric Surgery EPAs for general pediatric residency programs in Brazil. To assemble the study's expert panel, the primary criterion was experience in Pediatric Surgery and in the training of general pediatric residents, aiming for the broadest possible geographic representation in Brazil. The first phase involved orientation and contextualization of the work group, followed by the development of a list of all professional activities that interface with Pediatric Surgery that a General Pediatrician should perform in their career. In the second phase, the Delphi technique was applied in three rounds: the first for consensus, the second for selection based on relevance and agreement, and the third for validation and detailing of the EPAs.
This study was submitted to the Ethics Committee and approved under the CAAE number 55420822.0.0000.0144.
Data collection and analysisDuring the second phase, to apply the Delphi method with three rounds among experts for consensus, selection, and validation of Pediatric Surgery EPAs for general pediatric residents in Brazil, structured questionnaires were used to collect and record data through the REDCap® platform. The questionnaire was distributed to experts by WhatsApp®, a free and secure, cross-platform instant messaging app. Data analysis was performed using MS Excel (Microsoft Office Professional Plus, 2013) and IBM SPSS (Statistical Package for the Social Sciences v. 23.0, 2015).
First phase of the study: expert panelFor this phase, professionals in the position of supervisors/coordinators of Pediatric Surgery Residency Programs, all with experience in training general pediatric residents, and specialist members of the CIPE (Brazilian Pediatric Surgery Association) were contacted and invited via telephone. Initially, orientations were provided on the EPA curricular model and the Delphi method, including the forwarding of material on the subjects for a better understanding of the study.
Subsequently, the experts were individually instructed to evaluate and propose potential Pediatric Surgery EPAs for training general pediatric residents. The completion time for this phase was six weeks, during which clarifications about the subject and working method were provided by the study coordinators.
At the end of the first phase, duplicate or highly similar entries were excluded, to ensure that the proposed EPAs met the established criteria: essential activities are understandable and assessable.
Second phase: delphi methodRound 1 – consensusIn this round, a questionnaire was developed to facilitate content consensus analysis. The Entrustable Professional Activities (EPAs) identified by experts in the first phase were compiled and sent to all experts for peer review and evaluation, covering all 88 EPAs. This round lasted three weeks, and experts assessed the importance of each EPA using a five-point rating scale: “not important,” “mildly important,” “important,” “very important,” and “indispensable.”
Round 2: selection by relevance and agreementIn this stage, three weeks after the previous round, the experts were instructed to determine the representativeness and relevance of the content of each EPA. The scale used was the same as in round 1, ranging from 1 to 5, from not important to indispensable. A four-week deadline was set for the completion of this phase. After completion, the results of round 2 were compared with those of round 1 to further assess relevance and agreement.
The statistical analysis of the first two rounds of the Delphi method was divided into descriptive analysis, content validity, and reliability analysis. Qualitative variables were described using frequencies and percentages. The Content Validity Index (CVI) in this study was calculated by the mean value. The answers “very important” and “indispensable” were considered adequate, and the value 1 was assigned to them in each situation. The CVI corresponds to the average of the item values. The intraclass correlation coefficient (ICC) evaluates the agreement between more than two data sets or more than two raters. Currently, it is also a widely used value to determine the validity of an instrument through the agreement between judges (experts). An ICC close to 1 indicates high agreement among the values of the same group, and a low ICC close to zero expresses low agreement among the values. To calculate the ICC, the Likert scale initially used at five levels (0 to 4) was considered.
Round 3: validation and detailingThis stage of the Delphi study lasted two months and aimed to validate the Pediatric Surgery EPAs that are fundamental to the training of general pediatric residents. For this stage of the study, Brazilian Pediatric Surgery professionals who act as preceptors for resident physicians in both surgical and clinical areas of child care were invited.
The ideal sample size was determined based on the ICC of the previous phase. For a minimum ICC of 0.80 (p) and an acceptable error of 0.20 (wp) with a 95 % confidence interval, using the formula: 1+ [8(1,96)2(1 – p)2(1 + p)2/(2wp)2], the authors have n = 50 preceptors of Pediatric Residency programs. To assess the level of agreement and validation, a five-item rating scale was developed, ranging from strongly disagree to strongly agree. The questionnaire included demographic data, level of specialization, professional experience, and experience as a residency program preceptor.
The statistical analysis of this phase of the study was also divided into descriptive analysis, content validity, and reliability analysis. For the content validity of round 3, the answers “agree” and “strongly agree” were considered adequate, and the value 1 was assigned to them in each situation, where the CVI corresponds to the average of the item values. To calculate the ICC of this stage of the Delphi study, the Likert scale used was evaluated at five levels (1 to 5) of agreement.
Following the completion of the validation process, the Pediatric Surgery EPAs were organized, grouped, and detailed to align with the scope of general pediatric practice. This structuring enables comparative analysis and supports its implementation as a curricular model, preserving both the breadth and specificity required for effective pediatric care.
ResultsFor the first phase of the study, 12 residency program supervisors/coordinators with specialist titles were invited from across Brazil’s five geographic regions to ensure coverage of the diverse realities in Brazilian pediatric training. Of these, nine professionals agreed to participate, three from the Southeast region, two from the Northeast, two from the North, one from the Midwest, and one from the South region. These specialists initially developed a total of 88 Pediatric Surgery EPAs for the training of pediatric residents in Brazil (Figure 1).
In the first round of the Delphi method (consensus stage) of the second phase, the questionnaire was applied through the REDCap® platform to the initial nine specialists regarding the 88 activities listed in the first phase of the study. The responses were evaluated and analyzed, obtaining consensus on 31 EPAs, with CVI ≥ 0.80, for the categories “very important” and “indispensable”, and ICC of 0.893 (95 % CI 0.823 – 0.945) - Figure 1.
For the second round of the Delphi method (to define relevance and agreement), invitations were sent to all supervisors/coordinators of 54 Brazilian Pediatric Surgery residency programs. At this stage, 25 Pediatric Surgery preceptors participate in the training of general pediatric residents (9 specialists from phase 1 and 16 new preceptors), with a wide geographical distribution throughout the country, as shown in Table 1. From the 31 EPAs of the first round of the Delphi method, 17 EPAs were selected, for the answers “very important” and “indispensable” with CVI ≥ 0.80, and ICC of 0.851 (95 % CI 0.753 – 0.924) - Figure 1.
Descriptive analysis of the professional data of supervisors and coordinators of Pediatric Surgery Residency Programs who participated in the Delphi method – rounds 1 and 2.
The third round of the Delphi method was for the validation of the EPAs selected so far, and in this part of the work, from a total universe of approximately 450 preceptor physicians, 56 professionals (12.4 %) in Pediatric Surgery returned, of which two did not sign the informed consent form, three were duplicates, and one was incompletely filled out. Thus, 50 responses were considered valid and included in this study (Table 2).
Descriptive analysis of professional data from preceptors, supervisors and coordinators of Pediatric Surgery Medical Residency programs who participated in the Delphi - Round 3.
The questionnaire in this round assessed the agreement and validation of the 17 EPAs identified in the previous two Delphi rounds. Considering responses of “agree” and “strongly agree” as adequate, the calculated CVI was 0.965. However, three of the 17 previously selected EPAs had a CVI below 0.96, despite having been considered relevant in the earlier phase (Figure 1). The ICC for this round was 0.866 (95 % CI: 0.804–0.915).
In the detailing phase, to enhance clarity and applicability, the 14 EPAs validated in the third Delphi round were organized and condensed into seven broader EPAs aligned with general pediatric practice. For example, two EPAs related to outpatient surgical assessment were merged into a single EPA focused on the clinical evaluation and referral of children with suspected surgical conditions. Likewise, EPAs on inguinal and genital assessment were integrated into one comprehensive EPA. Neonatal surgical diagnoses, emergency care scenarios, and postoperative management were also grouped according to clinical context and overlapping required competencies. This reorganization preserved the essential content of each original EPA while improving coherence and feasibility for implementation. The final seven EPAs are presented in Table 3.
Content Validity Index (CVI) and Detailed Analysis – Round 3 of the Delphi Method.
The present study, conducted with professionals in Pediatric Surgery, preceptors of Medical Residency Programs (MRPs), from all five geographic regions of Brazil, allowed for the validation of seven EPAs in surgical areas for the practice of Pediatricians in the country. As this work addresses an innovative topic in medical education, the development, validation, and implementation of a curriculum based on EPAs are challenging [3,19–20].
In 2021, the American Board of Pediatrics (ABP) published on its website guidelines for the development of EPAs for the training of general pediatricians in a North American setting [21]. Even with the endorsement of one of the largest pediatric societies in the world, these guidelines have been criticized, mainly regarding their practical implementation and the ability to assess trainees [22].
One of the major obstacles observed in the development and validation of a curricular model with EPAs is the establishment of clear, measurable, and universally accepted criteria [3,6,7,20]. These competencies, when coupled with the demands of modern society´s health needs, which are very dynamic, and the local realities that influence these needs, [14,22] require excellent training of professionals in the medical area [1,3,5]. Thus, regarding the curricular model with EPAs, different institutions and regions may have varied expectations and particularities, which can complicate the development of a standardized curriculum [14,22].
To minimize these difficulties and achieve the goal of developing, reaching consensus, on selecting, and validating EPAs, the Delphi method has been adopted as one of the most used approaches to date [23–25]. The Delphi technique is considered effective for determining expert consensus, especially where there is little or no empirical evidence on a given subject [26,27]. In addition, its application allows for the balancing of divergent aspects and the solution of vague or unanswered questions about a specific area of knowledge [27,28].
In phase 1 of the study, prior to the application of the Delphi method, it was essential to have experienced professionals in the training of medical residents [17,23,25]. Therefore, the authors performed an active selection of specialists with experience not only in Pediatric Surgery but also in general pediatrics training, an aspect considered positive by the authors of this study. All the specialists who participated in this phase held supervisory/coordination positions in MRPs and have a specialist title from CIPE, in addition to participating in the training of medical residents in Pediatric Surgery and/or General Pediatrics. This selection facilitated the compilation of an initial list of professional activities for the pediatrician with an interface in Pediatric Surgery that more closely reflects the Brazilian reality.
In a country of continental dimensions, with diverse realities, the application of the Delphi method for the second phase of the study was considered appropriate to establish a more significant and comprehensive assessment [23,26]. A very relevant aspect of the present study is the geographical representativeness of the specialists who answered the questionnaires in the three rounds of the Delphi, especially in the third. Studies by Jesus et al. and Bistorff-Silva et al. (2023) on the distribution of pediatric surgeons in Brazil have shown a concentration in the Southeast and South regions, followed by the Northeast, Midwest, and North, [29,30] a pattern similar to that observed in the present data.
Another highlight during the application of the method was the participation of a significant number of qualified professionals with proven experience in resident training. According to 2023 data, there are 1414 pediatric surgeons in Brazil, [30] of which more than half are involved in teaching, including preceptorship in medical residency [29]. Fifty questionnaires were answered appropriately in the validation phase of the study, respecting the minimum determined by sample size calculation. These results demonstrate a significant interest of the specialists in the topic of EPAs in the training of medical residents, and this participation is expressive when compared to other similar studies [23,25].
However, the authors emphasize that there are limitations that should be pointed out. Studies using the Delphi method are inherently subject to criticism, particularly if not well designed and controlled [26,28]. In addition, they may present with response bias (participants more interested in the topic may be more likely to respond), where the influence of dissident dominant opinions can influence the real training environment [26]. To minimize these issues, the authors made an effort to organize the conduct of the study according to previously established criteria.
And, despite the significant involvement of professionals in the teaching of Pediatric Surgery from all over Brazil, the little experience with still very innovative topics (EPAs and Delphi methodology) can be considered another limitation of the results of the present study [7,28]. A possible reflection of this inexperience of the participants was the fact that 3 of the EPAs considered relevant by the initial panel of professionals and specialists in the first two rounds of the Delphi method were not validated by the national group of preceptors. A comparable study involving medical education experts familiar with EPAs, or pediatric specialists, could be undertaken to verify the relevance of the activities identified in this work.
To enhance clarity and support broader dissemination of the results, the authors incorporated an explanatory component on the EPAs into the third round of the Delphi method, alongside the validation stage [17,18,28]. This process led to organizing and condensing the 14 validated EPAs into 7 final EPAs, without compromising the overall findings. All 7 final EPAs exhibited the same validity index and a statistically significant content confirmation index.
The authors conclude that this pioneering study developed and validated 7 EPAs in Pediatric Surgery for the practice of General Pediatricians in Brazil, thereby filling a gap in medical education. The implementation of these EPAs holds the potential to standardize and enhance the training of pediatric residents, significantly contributing to both general pediatric and surgical care in the country.
FundingThere weren’t any financial funds used in this study.
The authors declare no conflicts of interest.
