To present, the process of development and evaluation of an educational software on the Child Health Handbook proposed for the continuing education of primary care nurses and physicians.
MethodsQuantitative study of methodological development. For software development, the following steps were followed: definition of objectives; determination of the target audience; choice of pedagogical and theoretical reference for content; content selection and structuring; software development and evaluation by experts (five nurses and four physicians). All responded to an instrument that included four domains: pedagogical; content; functionality; system presentation and usability. The evaluation criteria were arranged on a Likert-type scale. The percentage of agreement and Content Validity Index were used for the quantitative analysis of the degree of agreement, considering a Content Validity Index cutoff point equal to 0.80.
ResultsThe overall agreement index, calculated by the arithmetic mean of the Contents Validity Index of the evaluated domains, was 0.96, with scores ranging from 0.90 to 1.00. The average percentage of agreement of the experts per domain was 92.86%, with lower agreement in the content (80.95%), presentation, and usability (90.48%) domains. 100% of percentage of agreement was observed in the pedagogical and functionality domains among the evaluated specialists.
ConclusionThe percentage of agreement, Content Validity Index and overall agreement index of the Child Health Handbook educational software in the context of primary care disclosed the software adequacy as an educational resource for continuing education of primary care nurses and physicians. Considering the assessed dimensions, it can also be used by other health professionals and undergraduate students.
Continuing education is based on significant learning and the possibility of transforming professional practices having people's health, sector management and social control in health needs as reference.1,2 Additionally, it advocates the (re)significance of the training process based on the needs of professional practice and situations experienced in the service.1,3
In 2007, the new guidelines and strategies for the implementation of the National Policy on Permanent Health Education (PNEPS, Política Nacional de Educação Permanente em Saúde) considered regional specificities, the overcoming of regional inequalities, specific training needs and the development for health work, and the already installed capacity of the institutional offer of formal health education.1
In the context of continuing education, learning through technological resources, especially in vocational training and development, makes it easier to access knowledge and information, overcomes distance problems and access to bibliographies, fosters the circulation of information and the development of debates on health topics.2
The Child Health Handbook (CHH) was proposed aiming at being the main instrument for child health surveillance, seeking to stimulate the child’s full growth and development, and should be filled out by all professionals who work with childcare.4 Thus, the CHH is filled out particularly in maternity hospitals and primary care services, where the main information about the child is generated.5
However, the underuse of the CHH in the follow-up of children by primary care professionals has been observed, justified by factors such as lack of training to use the instrument, coupled with a lack of awareness of its importance. Studies carried out in public health units indicate that a large number of professionals responsible for childcare stated that they did not receive specific training for child health care actions, which may affect the quantity and quality of care provided.6,7
Considering the importance and challenges for the use of the CHH as a key instrument for monitoring child health in primary care, this study aimed to develop and evaluate an educational software for continuing education of primary care professionals on the CHH.
MethodsThe development of the educational software Caderneta de Saúde da Criança (Child Health Handbook, CHH) in the context of primary care consisted of a methodological development study.8 Therefore, the steps proposed by Botti et al.9 for the development of the educational instrument were followed: (1) definition of objectives; (2) establishment of the target audience’s characteristics; (3) choice of pedagogical reference; (4) choice of theoretical reference in the topic; (5) content selection and structuring; and (6) software development and evaluation.
Aiming to contributing to a critical, reflexive and technically competent performance in the development of actions in the field of primary care related to the follow-up of child growth and development, primary care nurses and physicians were selected as the target audience. These are referred to as the professionals responsible for monitoring child growth and development during childcare consultations.4 However, the software can also be useful for CHH use training of other health professionals and undergraduate students.
The proposed educational software based its pedagogical principles on: Continuing Health Education (CHE),1,3 in problematization10; Significant Learning (SL)11 and distance education.12
The PNEPS is based on SL, which is an underlying concept of the constructivist learning theories.11 In the context of the produced software, SL was used to promote the transformation of care practices on their current actions in the health service network.3
The content selection was guided through a previous educational diagnosis aiming at identifying the perceptions of seventeen primary care nurses and physicians about the CHH, which indicated difficulties to act in topics such as: follow-up of children with Down syndrome; children's rights; oral, vision, and hearing health counseling, among others.
The educational software developed utilizing the distance education model used a web platform and included features such as: (i) student registration; (ii) “Student’s Area”, which presents the necessary guidelines for navigation and follow-up of the instrument’s didactic path; (iii) a 'Course Presentation' section describing the strategy, content and assessment; (iv) ‘Student Guide’ with guidelines for using the software; (v) a presentation section of the Child Health Handbook describing its proposal and structure; and; (vi) the 'Clinical Situations' section, which depicts the several scenarios proposed for problematization.
Each proposed clinical situation comprises a summary of the clinical history of a child patient and tabs that represent the management of the CHH in the follow-up of the neuropsychomotor development, growth, vaccination schedule, and other directions for the nurses’ and physicians’ practice during childcare consultations (Fig. 1). This fill-out sequence was organized according to the presentation of the CHH contents, but the students can develop their own sequence, alternating the topics according to their preference and need.
The main subjects addressed in each 'Clinical Situation' were:
- 1)
Clinical situation 1 (Routine consultation): to develop comprehensive child health care; to encourage the use of the CHH in periodic childcare consultations.
- 2)
Clinical situation 2 (Preterm child): to develop comprehensive health care for the preterm child; to mobilize knowledge regarding the classification of newborns at risk and corrected gestational age; to perform child development surveillance, considering the prematurity; to stimulate the use of specific curves to monitor the preterm newborn’s growth.
- 3)
Clinical situation 3 (Childcare and Down syndrome): to develop comprehensive health care for children with Down syndrome; to perform surveillance of the development and growth of children with Down syndrome through specific curves and tables.
- 4)
Clinical situation 4 (Follow-up of the child with congenital Zika virus syndrome). To perform child development surveillance; to perform growth surveillance (assessment, identification of risk factors, decision-making guidance and recording of anthropometric measurements); to give advice on oral, vision and hearing health.
Nine experts, five nurses and four physicians, participated in an online interview at the educational software evaluation stage. Pasquali13 recommends six to twenty subjects, requiring at least three individuals in each group of selected professionals.
The experts were selected by snowball sampling, a non-probabilistic type of sampling that uses reference chains. As inclusion criteria, the parameters of the studies by Freitas et al.14 were adapted, considering the knowledge and experience of the experts, who needed to obtain a minimum score of five points.
The questionnaire for the educational software pedagogical evaluation was made available through Google Docs and created based on Behar's adapted protocol,15 consisting of 36 variables that covered four domains: pedagogical; content; functionality; system presentation and usability. The experts assigned a concept arranged on a Likert-like scale for each domain variable: (1) I Totally Disagree (TD); (2) I Partially Disagree (PD); (3) I Neither Agree nor Disagree (NAND); (4) I Partially Agree (PA) and (5) I Totally Agree (TA). If they felt they had no expertise to comment on a particular item, the evaluators were instructed to select the NAND option. At the end of the evaluation of each domain, a space was provided for the evaluators to comment about the items considered as having discrepancies or errors identified in the software, aiming to obtain information about adjustments and improvements of the instrument.
For the quantitative analysis of the degree of agreement among the experts during the evaluation process, the Content Validity Index (CVI) and the Percentage of Agreement (PA) were used among the specialists. The first measures the proportion or percentage of experts who agree on certain aspects of the instrument and its items. It initially allows analyzing each item individually, and then the instrument as a whole. The CVI cutoff point was considered as equal to 80% (0.80) according to the adopted reference.8
The PA was the method used to evaluate the degree of interobserver agreement. It consists in calculating the number of times the evaluators agree and dividing by the total number of evaluations (ranging from 0 to 100%). When using this method, we considered a minimum of 90% of agreement between the committee members.16
This study was approved by the Human Research Ethics Committee, under Opinion N. 1.214.970.
ResultsThree PhD’s, two individuals with a Master’s degree and four experts participated in the validation phase, of which four were physicians and five were nurses. The mean age of these experts was 36.4 years, ranging from 30 to 57 years, with a median of 34 years. Most were females (n=7). Regarding their professional experience, eight of them worked in child health care, primary care or pediatrics, and four had teaching experience in the areas of child health care or pediatrics. Of the latter, two mentioned software-mediated teaching experience. The time of care practice ranged from two to 34 years, with a mean of 11 years and a median of 11 years. In teaching, the mean was 6.2 years, ranging from two to 11 years and a median of 5 years.
Among the specialists, eight worked with childcare consultations, and five had over 10 years’ experience. All stated (n=9) that they always used the CHH as a guiding instrument in the consultations.
The educational software was evaluated by these specialists who analyzed the representativeness of the items in relation to the following domains: pedagogical; content; system functionality, presentation and usability.
When the CVI was analyzed by domain, it is noteworthy that the CVI of the pedagogical domain was evaluated with a maximum degree of 1.00 (indicating high agreement among the experts). This domain includes elements that underlie the decision-making regarding the adequacy and the software mode of use as an educational tool (Table 1).
Content Validity Index regarding the Pedagogical domain. Recife, 2018.
| Pedagogical | TA | PA | NAND | PD | TD | CVI | |
|---|---|---|---|---|---|---|---|
| The software allows a reflection on the use of the CHH in childcare care practice. | 8 | 1 | 1.00 | ||||
| The strategy based on clinical situations depicts the care practice of CHH use | 7 | 2 | 1.00 | ||||
| The software clearly shows the CHH filling out process in childcare practice. | 6 | 3 | 1.00 | ||||
| The software allows the development of professional skills in CHH management (development, growth, vaccination schedule and guidance). | 7 | 2 | 1.00 | ||||
| Resources of feedback, suggestions, notes, tips, etc. as presented in the system, contribute to the reflection on the use of CHH in childcare care practice. | 7 | 2 | 1.00 | ||||
| The software encourages learner's autonomy. | 6 | 3 | 1.00 | ||||
| The supporting material is consistent with the possible needs of practitioners regarding CHH guidance. | 5 | 4 | 1.00 | ||||
| The software can contribute to changes in the professional practice of primary care physicians and nurses. | 5 | 4 | 1.00 |
CHH, Child Health Handbook; TA, I totally agree, PA, I partially agree; NAND, I neither agree nor disagree; PD, I partially disagree; TD, I totally disagree.
The overall CVI of content domain was 0.90. For all items evaluated in this domain, the CVI value was >0.80 (minimum value indicated for six or more experts). When the CVI was evaluated by item, considering whether the presented clinical situations are written in a structured, objective manner consistent with what is available in the CHH, it can observed that the clinical situations 3 (Childcare and Down Syndrome) and 4 (Follow-up of children with Zika virus syndrome) achieved the lowest scores (0.78) (Table 2).
Content Validity Index for the Content domain. Recife, 2018.
| Content | TA | PA | NAND | PD | TD | CVI |
|---|---|---|---|---|---|---|
| Is the content used in the software based on reliable references? | 7 | 2 | 1.00 | |||
| The content is appropriate for the pedagogical proposal. | 5 | 4 | 1.00 | |||
| The software is appropriate for the socio-cultural profile of the target audience. | 5 | 4 | 1.00 | |||
| The presented clinical situations - Routine consultation, Preterm child, Childcare and Down Syndrome, Follow-up of children with ZIKA Syndrome - are appropriate to work with the topic of using / filling out the CHH in childcare. | 5 | 3 | 1 | 0.89 | ||
| The sequence of the presented clinical situations allows the learning of significant topics for the use of the CHH. | 8 | 1 | 1.00 | |||
| The presented clinical situations are written in a structured, objective manner. | 1.00 | |||||
| Clinical Situation 1 - Routine Consultation | 8 | 1 | 0.89 | |||
| Clinical Situation 2 - Premature Child | 8 | 1 | 0.78 | |||
| Clinical Situation 3 - Childcare and Down Syndrome | 6 | 1 | 2 | 0.78 | ||
| Clinical Situation 4 - Zika virus Syndrome | 6 | 1 | 2 | 1.00 | ||
| The content presented on child development is consistent with what is available in the CHH. | 1.00 | |||||
| Clinical Situation 1 - Routine Consultation | 7 | 2 | 0.78 | |||
| Clinical Situation 2 - Premature Child | 6 | 3 | 0.78 | |||
| Clinical Situation 3 - Childcare and Down Syndrome | 4 | 3 | 2 | 1.00 | ||
| Clinical Situation 4 - Zika virus Syndrome | 4 | 3 | 2 | 0.89 | ||
| The content presented on child growth is consistent with what is available in the CHH. | 0.78 | |||||
| Clinical Situation 1 - Routine Consultation | 6 | 3 | 0.78 | |||
| Clinical Situation 2 - Premature Child | 7 | 1 | 1 | 1.00 | ||
| Clinical Situation 3 - Childcare and Down Syndrome | 4 | 3 | 1 | 1 | 0.78 | |
| Clinical Situation 4 - Zika virus Syndrome | 4 | 3 | 1 | 1 | 0.78 | |
| The content presented on vaccination is consistent with what is available in the CHH. | 0.78 | |||||
| Clinical Situation 1 - Routine Consultation | 6 | 3 | 1.00 | |||
| Clinical Situation 2 - Premature Child | 6 | 1 | 1 | 1 | 0.89 | |
| Clinical Situation 3 - Childcare and Down Syndrome | 4 | 3 | 1 | 1 | 0.78 | |
| Clinical Situation 4 - Zika virus Syndrome | 5 | 2 | 1 | 1 | 0.78 | |
| The content presented on guidelines is coherent. | 1.00 | |||||
| Clinical Situation 1 - Routine Consultation | 8 | 1 | 1.00 | |||
| Clinical Situation 2 - Premature Child | 8 | 1 | 1.00 | |||
| Clinical Situation 3 - Childcare and Down Syndrome | 4 | 3 | 1 | 1 | 1.00 | |
| Clinical Situation 4 - Zika virus Syndrome | 5 | 2 | 1 | 1 | 1.00 | |
| The contents of the Student's Area module are clearly and concisely described. | 7 | 2 | 1.00 | |||
| The contents of the Course Presentation module are clearly and concisely described. | 7 | 2 | 0.89 | |||
| The contents of the Student's Guide module are described clearly and concisely. | 7 | 2 | 1.00 |
CHH, Child Health Handbook; TA, I totally agree, PA, I partially agree; NAND, I neither agree nor disagree; PD, I partially disagree; TD, I totally disagree.
As for the system functionality domain, the CVI was equal to 1.00, same value of the overall CVI. Among the system functionalities, the instrument offers: user registration, certification, evaluation, password change, good practice tips, learning feedback, clinical situation monitoring, content references, among other resources (Table 3).
Content Validity Index regarding the domains of Functionality; Presentation and Usability. Recife, 2018.
| Functionality | TA | PA | NAND | PD | TD | CVI |
|---|---|---|---|---|---|---|
| The system has the required functionalities for a distance education course (registration, certification, evaluation, password change, clinical case follow-up, etc.). | 6 | 3 | 1.00 | |||
| Presentation and usability | TA | PA | NAND | PD | TD | CVI |
| The software is self-explanatory. | 7 | 2 | 1.00 | |||
| The program is easy to learn and use. | 6 | 3 | 1.00 | |||
| The program is interesting for the user. | 7 | 2 | 1.00 | |||
| The program provides motivating stimuli. | 6 | 3 | 1.00 | |||
| The overall presentation of the program at screen level is adequate. | 6 | 2 | 1 | 0.89 | ||
| The software layout is attractive to the target audience and visually pleasing. | 5 | 3 | 1 | 0.89 | ||
| The program is efficient in user-system communication regarding the amount of information per screen. | 5 | 3 | 1 | 0.89 | ||
| The program can be used without difficulty by childcare nurses and physicians. | 7 | 2 | 1.00 | |||
| The software provides feedback to the learner upon completion of each clinical situation. | 7 | 2 | 1.00 | |||
| The pop-ups facilitate the learner's usability (Pop-up is a window that opens in the browser when visiting a web page or accessing a specific link). | 7 | 2 | 1.00 | |||
| The messages (suggestion, notes, tips) sent by the program are clear. | 6 | 2 | 1 | 0.89 | ||
| The hit-or-miss checking module provides reflection on the error. | 5 | 3 | 1 | 0.89 | ||
| The FAQ module provides relevant questions for the student's clarification. | 5 | 3 | 1 | 0.89 | ||
| The program is efficient in the user-system communication regarding the memorization of commands. | 4 | 5 | 1.00 |
TA, I totally agree, PA, I partially agree; NAND, I neither agree nor disagree; PD, I partially disagree; TD, I totally disagree.
The CVI value of the presentation and usability domains in all evaluated items was 0.89 or higher, and the overall CVI related to these domains was 0.95. When analyzed individually, the items: General presentation of the program at screen level; The program is efficient in the user-system communication regarding the amount of information per screen; The hit-or-miss checking module provides reflection on the error; The software provides feedback to the learner upon completion of each clinical situation, were evaluated by an expert with the option “Totally Disagree” (Table 3).
The overall agreement index of the educational software was 0.96. This value was calculated by the arithmetic mean of the CVI of the domains with scores ranging from 0.90 to 1.00.
When the percentage of agreement among the experts by assessed domain was calculated, it was observed that the domain with the lowest agreement among the experts was related to the content domain (80.95%), followed by presentation and usability (90.48%). In the pedagogical and functionality domains, 100% agreement was verified among the evaluated experts. In general, the average percentage of agreement was 92.86%.
DiscussionSoftware programs can be disseminated in primary care for several purposes, from continuing education resources to work process management.17–19 Regarding educational software programs, the evaluation by judges aims to investigate the following domains: pedagogical; content; functionality; system presentation and usability.15
Regarding the educational software ‘Child Health Handbook’, in the context of primary care, the judgment of pedagogical attributes showed high agreement (1.00) among the experts, considering the objectives, relevance, scope of pedagogical assumptions and supporting material. The use of the pedagogical reference of SL has special importance for software developed for CHE, since it recommends the use of technology connected to the work process, linking content with action. The maximum evaluation obtained for this domain suggested that the software has significant potential to promote changes in the professional practice of primary care nurses and physicians, reaffirming that the didactic strategies for presenting information and cognitive tasks were in accordance with the educational objective and characteristics of the target audience.
Other studies have proposed the use of health education software with the cognitive approach postulated by Vygotsky.9,17,18 These studies presented software for undergraduate education, aiming at the significance of academic contents, therefore presenting a different proposal of an instrument for professional approach, adhering to the PNEPS requirements.
When assessing the CVI of the content domain, we sought to identify the content completeness and correctness, considering the area of knowledge being explored by the software.20 The content validity assessment by specialists thoroughly analyzes whether it is appropriate for what it has been proposed, and suggestions, corrections and additions to the suggested content can be collected. This validation process ensures the reliability and consistency of the information intended to be conveyed through the software.8
The agreement index of 0.90 suggested the review of specific items conceptualized as ‘NAND’ or ‘TD’. In this dimension, the structure of clinical situations and language regarding their clarity and conciseness was evaluated, among other aspects. The items that received the lowest score (0.77) were Clinical Situations 3 (Childcare and Down syndrome) and 4 (Follow-up of children with Zika virus syndrome). Both depicted the follow-up of childcare of children with peculiar growth and development patterns, not contemplated in the current CHH, a fact that may have influenced the lower score given by the experts. This study suggests the inclusion of such follow-up parameters for children with disabilities in the new 'Child Health Handbook - Passport for Citizenship’,21 which will be launched by the MoH, with an intersectoral component that incorporates health, social care and education information.
The validation of content with a high degree of agreement was a relevant aspect of this study. The CHE advises that the training proposal be based on the population’s real care needs. In this sense, it is justified to adopt the proposed themes such as Down syndrome and congenital Zika virus syndrome, seeking to update primary care professionals regarding the demands of the service.
As for the system functionality domain, the CVI obtained 100% agreement among the experts. Adopted features such as best practice tips, learning feedback, and user-friendly content references while using the CHH considered the target audience characteristics, styles, and types of learning that are meant to significantly facilitate learning by making the best use of interactivity and user control over what is learned and how it is learned.15,22
The presentation and usability domain of the software was positively evaluated, reaching 95% of agreement. The use of computers for teaching becomes more efficient when it has an appropriate interface (layout), favoring the interaction between the learner and the machine. The development of an interface compatible with the user language23 facilitates the occurrence of SL. In other words, the covered concepts will be more easily assimilated if they are presented in a language that makes sense for them.23 For this purpose, the images of the CHH were reproduced faithfully, showing the learner how to handle the real instrument through the software.
The term usability refers to the quality of a product, that is, one that satisfies and meets the user needs.24 Squires and Preece25 emphasize that the evaluation of digital didactic material should consider both the learning and usability aspects, seeking the integration between them.
In this study, the educational software was developed considering the usability measures: easy learning; efficiency of use; easy memorization and low error rate.26 Moreover, the following were defined as objectives: the possibility of intuitive, free navigation in a nonlinear and flexible structure,27 and the ability to report and retrieve errors. Therefore, these aspects were included in the CHH software according to the judges' perception.
The evaluation phase is considered essential for the software evolution and, consequently, for the quality assurance of the final product.28 Therefore, most suggestions for software improvement were accepted, especially those related to grammatical corrections related to textual agreement and necessary changes to the proposed content, considering the conciseness and clarity of digital texts, which are decisive elements for a pleasant and meaningful learning, resulting in health practice transformations.
The educational instrument for the CHH can also be applied in undergraduate teaching in Medical and Nursing schools, allowing students to make a practical association in childcare with the addressed theoretical content, as well as allowing their contact with work processes in primary care assistance. Such practice is in line with the principles of pedagogical innovation, considering new ways of teaching and learning and new relations between theory and practice.29
The evaluation of the expert judges regarding the ‘Child Health Handbook’ educational software in the context of primary care, considered it appropriate for its use as a resource in continuing education about the CHH of primary care nurses and physicians, considering the analyzed dimensions and criteria. The educational software was developed to promote autonomy in the process of training linked to work, that is, the construction of knowledge fostering the teaching-service integration (theory and practice fields) allowing changes to be made in the care of children in primary care based on SL. In continuing health education, the workspace is a privileged learning site.3
Aiming to modify the professional practice based on education,30 it is suggested that an assessment be carried out with the target audience (primary care nurses and physicians).9 The feedback from users allows the identification of instructional deficiencies and gaps that will need to be adjusted or adapted in a new version, aiming to further expand the reach of the software in childcare and the effective implementation in their care processes.
Conflicts of interestThe authors declare no conflicts of interest.
Please cite this article as: Barbosa CP, Belian RB, de Araújo CM. Continuing education in the child health handbook: an educational software for primary care. J Pediatr (Rio J). 2020;97:80–7.
Study conducted at Universidade Federal de Pernambuco (UFPE), Recife, PE, Brazil.
