Journal Information
Share
Share
Download PDF
More article options
Visits
...
Original article
Open Access
Available online 16 September 2022
Predictive validity and cut-off point of the Turkish version of the Infant Colic Scale in the diagnosis of colic
Visits
...
Renginar Öztürk Dönmeza,
Corresponding author
, Ayla Bayik Temela, Feyza Koçb
a Ege University Faculty of Nursing, Department of Public Health Nursing, Izmir, Turkey
b Ege University Faculty of Medicine, Department of Social Pediatrics, Izmir, Turkey
Received 22 April 2022. Accepted 11 July 2022
Article information
Abstract
Full Text
Bibliography
Download PDF
Statistics
Abstract
Objective

To investigate the predictive validity and cut-off point of the Turkish version of the Infant Colic Scale (ICS) in the diagnosis of colic.

Methods

This methodological study was carried out in a pediatric outpatient clinic of a university hospital in Turkey with infants aged 6–16 weeks (n = 133). The data were collected using the Mother-Infant Description Form, the ICS, and the Rome IV criteria form. The scale is a 6-point Likert-type scale consisting of 19 items in total. A low total mean score obtained from it indicates that the probability of colic increases, while a high mean score indicates that the probability of colic decreases. The Rome IV criteria were used as the gold standard.

Results

The mean score obtained from the ICS was 59.4 ± 13.7. According to the Rome IV criteria, 26.3% of the infants had colic. The area under the ROC curve was 87.4% (95% CI = 0.815–0.934, SE = 0.30, p = 0.001), and the cut-off point for the best sensitivity value (88.6%) and the best specificity value (70.5%) of the ICS was determined to be 60.5. According to the cut-off point, the positive predictive value was 51%, and the negative predictive value was 94%.

Conclusion

The predictive validity of the Turkish version of the ICS was found to be at a good level with high sensitivity and acceptable specificity for a cut-off point of 60.5. Healthcare professionals working in the child field can use the ICS to exclude colic in infants.

Keywords:
Infant Colic Scale
Predictive validity
Cut-off value
Full Text
Introduction

Crying is an innate and important behavioral ability of neonates to express their emotions. With crying behavior, infants express their needs, e.g., being hungry, feeling cold, and the desire to be taken on their lap.1 However, it is observed that some infants cry for longer than three hours a day, it is difficult to calm them down, and the intensity, severity, duration, and frequency of crying differ from normal crying. This condition is defined as infantile colic.2 The incidence of colic peaks at six-eight weeks after birth and decreases after 12 weeks.2,3 Although it is stated in the literature that infantile colic is a common health problem affecting 5–40% of healthy infants, there are no definite data on the incidence of colic since its etiology cannot be explained completely.4,5 Due to the crying behavior in early infancy, about half of parents consult a physician, and therefore, the costs of the provided care are very high.6 Crying that cannot be calmed may result in 'shaken baby syndrome' as an indicator of child neglect.7,8 5.6% of parents have engaged in this dangerous behavior at least once by the age of six months to stop their infants from crying.8 Thus, the diagnosis of colic constitutes a cost-effective approach both by reducing unnecessary care and diagnosis expenditures and by improving the quality of life of infants and their families.6,9,10

Wessel et al. described infantile colic as unexplained and uncontrollable crying episodes that occur in the first weeks after birth, are usually observed in the first three months, last for more than three hours a day, occur more than three days a week, and continue for at least three weeks, and usually occur in the afternoon and evening.2,11 Then, the Rome I criteria were developed in 1992, Rome II criteria were developed in 1999, and Rome III criteria developed in 2006 were used for a long time.12 Due to the very strict rules of the criteria in the diagnosis of infantile colic, an infant's restlessness was accepted as the criterion for diagnosis in the Rome IV criteria in 2016. According to the Rome IV criteria, in the physical examination or phone call to parents, uncontrollable crying or restlessness observed in infants younger than five months without clear causes and any cause such as illness or fever, lasting for at least three hours a day in the last week and occurring at least three days a week was defined as infantile colic.7

In a study conducted in Indonesia, it was determined that 57.3% of pediatricians used Rome IV for the diagnosis of infantile colic. However, in this study, 70% of these physicians had average knowledge of diagnosing infantile colic with Rome IV. Only 14.5% knew Wessel's rule of threes (crying for ≥ 3 h per day, ≥ 3 days per week, for ≥ 3 weeks).13 In a study in Turkey, 78.7% of physicians confuse the diagnosis of infantile colic with infantile spasm.14 Failed diagnosis and treatment may exacerbate the parents’ emotional stress.6,13

The Infant Colic Scale (ICS) developed by Ellett et al. can be used to determine risk factors for colic and also to diagnose colic.15 The validity and reliability study of the Turkish version of the ICS was conducted by Cetinkaya and Başbakkal.16 While using the scale, the decrease in the total score obtained from the scale is interpreted that the probability of colic in an infant increases. However, a definitive diagnosis of colic cannot be made in this way. Furthermore, if the scale will be used in clinical practice for diagnostic purposes, it is stated that the predictive validity of the tool should be taken into consideration.17,18 It was found that the predictive validity of the ICS was not determined in any study. When the predictive value or the cut-off point for this scale is determined, healthcare professionals will be able to diagnose infantile colic in infants and identify the associated causes.

This study aimed to determine the predictive validity and cut-off point of the ICS in the diagnosis of colic. In this study, the answers to the following research questions were sought:

  • 1.

    What is the predictive validity value of the ICS?

  • 2.

    What is the cut-off point of the ICS to diagnose colic?

Material and methodsDesign and participants

This methodological study was conducted in the pediatric outpatient clinic of a university hospital in Turkey between February 2017 and May 2018. For routine follow-up, the number of infants admitted to the pediatric outpatient clinic every month was 500, and 150 of them were 6–16 weeks old. The sample size was determined by Power and Sample size calculation (PS) (version 3.0.43).19 The data in the study conducted by Canivet et al. were used for power analysis.20 Accordingly, the sample size was determined to be 124 infants. This study was completed with 133 infants who were 6–16 weeks old.

Inclusion criteria for the study

  • -

    The infant being born after the 37th week of gestation

  • -

    The infant birth weight of 2500 g and above

  • -

    The infant being 6–16 weeks old

  • -

    The mother being literate

  • -

    The mother volunteering to participate in the study

Exclusion criteria for the study

  • -

    Infants will be excluded from the scope of the study if their parents do not want their information to be used in the study and if they withdraw from the study.

The Standards for Reporting of Diagnostic Accuracy (STARD) checklist was used in the study design and drafting of the manuscript.21

Assessment

In the study, the data were collected using the Mother-Infant Description Form (MIDF), Rome IV criteria form, and the ICS.

The Mother-Infant Description Form consisted of a total of 17 questions describing the characteristics of the mother and the infant.4,12,16,22 In the characteristics of the mother section, there were 9 questions to determine the socio-demographics of the mothers, drinking tea or coffee, and smoking status. In the characteristics of the infant section, there were a total of eight questions about the sex, birth method, feeding style, birth order, birth weight, and birth week of the infants.

The Rome IV criteria form was prepared based on the Rome IV Diagnostic Criteria for Functional Gastrointestinal Disorders.7,12 In the form, the three following criteria were questioned in infants younger than 6 months:

  • -

    Crying or restlessness lasting for at least 3 days a week in the last week and for at least 3 hours a day (yes/no),

  • -

    An inability to control crying or restlessness (yes/no),

  • -

    In addition to the mentioned criteria in the form, to exclude an organic disease in the infant, a history of vomiting, respiratory problems, and atopy was questioned (yes/no).

The Infant Colic Scale consists of a total of 19 items and the subscales of “cow's milk/soy protein allergy/intolerance,” “immature digestive system,” “immature central nervous system,” “problematic infant,” and “parent-infant interaction + problematic infant.” The items of the ICS were evaluated on a 6-point Likert-type scale. The six response options were “strongly disagree,” “moderately disagree,” “slightly disagree,” “slightly agree,” “moderately agree,” and “strongly agree”. Negative questions are coded reversely to ensure consistency in the interpretation of scores. A minimum score of 19 and a maximum score of 114 can be obtained from the ICS. A low mean total score obtained from the scale indicates that the probability of colic increases, while a high mean score indicates that the probability of colic decreases. The ICS was developed by Ellet et al., and its validity and reliability studies in the Turkish population were carried out by Cetinkaya and Başbakkal.15,16 They found the Cronbach's alpha coefficient of the scale to be 0.70.16 In this study, Cronbach's alpha coefficient of the ICS was determined to be 76.9.

Procedure

The Rome IV criteria were used as the gold standard. While infants were in the routine outpatient follow-up, they were evaluated by the physician-researcher according to the Rome IV criteria for colic/non-colic. The data collection forms presented in the mother-infant description form and the ICS were filled out by the nurse researcher while infants were waiting in the same outpatient clinic. The ICS score was determined for each infant. The evaluations of the physician-researcher and the nurse researcher were compared according to the protocol number.

Ethical Considerations

The permission from the authors to use the scale, the Ege University Scientific Research Ethics Committee approval (Date = 14/02/2017, Number = 17–1.1/8), and the permission from participants were obtained.

Statistical analysis

After a sufficient sample size was reached, the data were analyzed using SPSS 16.0 software for number, percentage, mean, logistic regression analysis was used for the cut-off point of the ICS, and the McNemar test was used for discriminant function analysis. The best sensitivity and specificity values gave the cut-off point of the scale. According to the cut-off point, the positive predictive value (PPV) and the negative predictive value (NPV) were calculated.

ResultsParticipants’ characteristics

The descriptive characteristics of participants are given in Table 1. The mean age of the mothers was 30.8 ± 5.4 (Min = 19 - Max = 43) years. 36.8% of them had 15 years or more education and 90.2% were from nuclear families. Income-expenditure status of 68.4% of them was equal. 90.2% of them drank tea every day, 50.4% drank coffee every day, and 92.5% were non-smokers. The mothers were drinking 2.8 ± 2.1 (Min = 0 – Max = 10) cups of tea, 0.6 ± 0.7 (Min = 0–Max = 3) cups of coffee per day on average. They smoked an average of 3.2 ± 1.1 (Min = 1 – Max = 5) cigarettes a day. 55.2% of infants were boys and 71.2% were born by cesarean section. The mean gestational week of the infants was 38.4 ± 1.3 (Min = 37- Max = 41), the mean birth weight was 3.239 ± 14.3 (Min = 2550 – Max = 4000) g, and the mean birth height was 50.3 ± 1.9 (Min = 45 – Max = 54) cm. 44.8% of them were the first child, 64.8% were exclusively breastfed. The mean age of the infants was 10.0 ± 2.4 (Min = 7–Max = 16) weeks during the study.

Table 1.

Descriptive characteristics of participants (n = 133).

Characteristics 
Mother's educational level
≥15 years  49  36.8 
≥11 years  48  36.1 
≥8 years  36  27.1 
Family type     
Nuclear  120  90.2 
Extended  13  9.8 
Family income status     
Income less than expenses  30  22.6 
Income equal to expenses  91  68.4 
Income more than expenses  12  9.0 
Mother's daily tea drinking     
Yes  120  90.2 
No  13  9.8 
Mother's daily coffee drinking     
Yes  67  50.4 
No  66  49.6 
Mother's smoking status     
Yes  10  7.5 
No  123  92.5 
Infant's sex     
Girl  60  44.8 
Boy  73  55.2 
Infant's birth method     
Vaginal delivery  38  28.8 
Cesarean section  95  71.2 
Order of the infant     
First  56  44.8 
Second  52  41.6 
Third and more  25  13.6 
Infant's feeding style     
Exclusive breastfeeding  86  64.8 
Only formula  14  10.4 
Breast milk + formula  33  24.8 
Total  133  100.0 
Results of the Rome IV criteria and the Infant Colic Scale

According to the Rome IV criteria, 32.3% of infants were crying at least 3 days a week and 23.3% of them were crying for more than 3 h a day. Crying was uncontrollable in 22.6% of infants. 58.6% of them had a history of vomiting, 12% had a history of respiratory problems, and 8.3% had a history of atopy. According to the Rome IV criteria as the gold standard, 26.3% of infants had colic.

“Cow's milk/soy protein allergy/intolerance subscale” of the ICS the total mean score was 5.7 ± 3.61, “Immature digestive system subscale” total mean score was 7.3 ± 3.05, “Immature central nervous system subscale” total mean score was 23.2 ± 5.26, “Problematic infant subscale” total mean score as 12.7 ± 4.85, and “Parent - baby interaction problem baby subscale” was determined as 10.4 ± 3.92. The mean score obtained from the ICS was 59.4 ± 13.7 (Min = 28.0- Max = 88.0) (Table 2).

Table 2.

The distribution of total mean scores of the Infant Colic Scale total and subscales.

Items  Mean  SD  Min-Max 
Cow's milk/soy protein allergy /intolerance subscale  5.7  3.61  2.0–10.0 
1.What mom eats whether the baby has colic or not  2.8  1.91  1.0–6.0 
2. What mom eats affects how bad the colic is  3.0  1.91  1.0–6.0 
Immature digestive system subscale  7.3  3.05  2.0–12.0 
3. My baby does not usually vomit  3.7  1.79  1.0–6.0 
4. My baby also vomits between feeding  3.5  1.69  1.0–6.0 
Immature central nervous system subscale  23.2  5.26  7.0–42.0 
5. My baby is jittery  3.2  1.74  1.0–6.0 
6. Colic occurs when my baby has had a busy day  3.4  1.74  1.0–6.0 
7. My baby does not need to be rocked to sleep  4.0  1.85  1.0–6.0 
8. Colic is not related to my baby being tired  3.5  1.91  1.0–6.0 
9. My baby can go to sleep by him or herself  3.4  1.73  1.0–6.0 
10.My baby is always in motion when awake  2.2  1.02  1.0–6.0 
11. My baby sleeps at different times every day  3.6  1.80  1.0–6.0 
Problematic infant subscale  12.7  4.85  4.0–24.0 
12. My baby is cranky most of the time  3.1  1.67  1.0–6.0 
13. My baby does not cry easily  3.5  1.60  1.0–6.0 
14. My baby is happy most of the time  2.5  1.32  1.0–6.0 
15. My baby waits calmly while I get the food ready  3.7  1.70  1.0–6.0 
Parent -baby interaction+problem baby subscale  10.4  3.96  4.0–24.0 
16. When my baby starts to fuss, nothing I do helps  3.1  1.60  1.0–6.0 
17. When the colic starts, I can soothe him or her  2.5  1.27  1.0–6.0 
18. When the colic starts, nothing I do helps  2.7  1.43  1.0–6.0 
19. I can tell what my baby wants when he or she starts to cry  2.2  1.12  1.0–6.0 
Total  59.4  13.7  28.0–88.0 

SD, standard deviation.

Predictive validity results of the Infant Colic Scale

In this study, it was found that the area under the ROC curve was 87.4% (SE = 0.30, p = 0.001, 95% CI = 0.815–0.934), and the sensitivity and specificity values for determination with the least error were found to be at the ICS cut-off point of 60.5. According to this cut-off point, the ICS has 88.6% sensitivity, 70.5% specificity, 51% PPV, and 94% NPV in detecting colic in infants (Table 3).

Table 3.

Cut-off points, sensitivity and specificity, positive and negative predictive values of the Turkish Version of the Infant Colic Scale in the diagnosis of infantile colic according to the Rome IV Criteria (n = 133).

Cut-off point  AUC  Sensitivity (%)  Specificity (%)  PPV (%)  NPV (%) 
58.5  0.874  91.4  67.3  50.0  95.0 
59.5  95%  88.6  69.4  50.0  94.0 
60.5  CI = 0.815–0.934  88.6  70.5  51.0  94.0 
61.5  SE = 0.30  85.7  70.5  50.0  93.0 
62.5  p = 0.001  80.0  71.4  50.0  90.0 

AUC, area of under curve, PPP, positive predictive value, NPV, negative predictive value.

58.5, 59.5, 61.5, and 62.5 were determined to be the other closest cut-off points for the diagnosis of colic by the ICS. Sensitivity, specificity, PPV, and NPV for these cut-off points are shown in Table 3.

Discriminant function analysis was performed to determine the scope of the ICS in the groups with or without infantile colic. It was found that the ICS could accurately predict 75.2% of colic and non-colic infants with a cut-off point of 60.5 (p = 0.001) (Table 4). The predictive power was obtained by dividing true positive values and true negative values by the total number of cases. According to the logistic regression analysis, the testing power of the ICS was determined to be 75.2%.

Table 4.

Discriminant Function Analysis results (n = 133).

According to the ICS's cut-off point for 60.5According to Rome IV criteriap-valuea 
ColicNon-Colic 
 
Colic  31  88.6  29  29.6  0.001
Non-Colic  11.4  69  70.4 

ICS, Infant Colic Scale.

a

McNemar test.

Discussion

Diagnosis of infantile colic with Rome IV criteria requires experience.6 Infantile colic can also be diagnosed with the ICS. The ICS contains more questions than Rome IV but provides ease of diagnosis based on the cut-off point. According to the cut-off point for the ICS scale, healthcare professionals working in the field of infant health can decide whether an infant has colic. The use of the scale also provides information on the etiology of colic. In this study, 26.3% of the infants had colic according to the Rome IV criteria, and the sensitivity and specificity values for determination with the least error were found to be at the cut-off point of 60.5.

Determining the cut-off point of a measurement tool closely affects the sensitivity and specificity of that measurement tool. If a low cut-off point is selected, a large number of normal individuals may also be diagnosed as patients. However, all patients may be identified. In this case, the sensitivity and specificity of the test are also low. On the contrary, when a high cut-off point is selected, all healthy individuals can be identified. Nevertheless, some of the real patients may also be diagnosed as healthy. In this case, the sensitivity of the test is low, but the specificity is high. Therefore, patients and healthy individuals may be misdiagnosed (false positive and false negative) at close rates, and it is appropriate to select the cut-off point at which the sensitivity and specificity of the screening test are affected at close rates.23

The first rule for determining the cut-off point is to take at least a 70% value first for both specificity and sensitivity. The second rule is to take the cut-off value, which gives the highest sensitivity value.23 According to these criteria, it was determined that the sensitivity and specificity values for colic infants to be determined with the least error were at the ICS cut-off point of 60.5. According to this cut-off point, the ICS has 88.6% sensitivity, 70.5% specificity, 51.0% PPV, and 94.0% NPV in detecting colic infants. In another study, since the cut-off value for the ICS was not calculated, the conformity of the 60.5 cut-off point could not be compared. However, theoretically, the sensitivity, specificity, PPV, and NPV of the cut-off point of 60.5 for the ICS scale were found to be acceptable and valid for the diagnosis of colic.

The height of the area under the curve (AUC) shows that the test has reached the highest sensitivity and specificity values with the cut-off point, and the determined values are significantly higher than the parametric values of sensitivity = 0.50 and specificity = 0.50.23 The AUC of 87.4% indicates a significantly higher area (p = 0.001).

Discriminant function analysis was performed to determine the scope of the ICS in predicting colic and non-colic groups. It was determined that the ICS was able to accurately predict 75.2% of colic and non-colic infants with a cut-off point of 60.5, and this rate had good predictive power.

Limitations

This study has some limitations. The study was conducted in a pediatric clinic. Since the predictive validity of the ICS was not examined in another study/culture, the results could not be compared.

Conclusions

In this study, the predictive validity of the Turkish version of the ICS was found to be at a good level with high sensitivity and acceptable specificity for a cut-off point of 60.5. Healthcare professionals working in this field can use the ICS to exclude colic in infants. It can also be recommended to test the predictive validity of the ICS in another population.

Acknowledgments

The authors would like to thank the mothers and their infants who participated in the study and the researchers who adapted the ICS to Turkish society and gave permission to use it in the study, and we also would like to thank Prof. Dr Sadık Akşit for his contribution about research design/concept.

References
[1]
H Karp.
The Happiest Baby on the Block; Fully Revised and Updated Second Edition: The New Way to Calm Crying and Help Your Newborn Baby Sleep Longer.
Bantam, (2015),
[2]
MA Wessel, JC Cobb, EB Jackson, GS Harris, AC Detwiler.
Paroxysmal fussing in infancy, sometimes called “colic”.
Pediatrics, 14 (1954), pp. 421-435
[3]
D Wolke, A Bilgin, M. Samara.
Systematic review and meta-analysis: fussing and crying durations and prevalence of colic in infants.
J Pediatr, 185 (2017),
[4]
S García Marqués, R Chillón Martínez, S González Zapata, M Rebollo Salas, JJ Jiménez Rejano.
Tools assessment and diagnosis to infant colic: a systematic review.
Child Care Health Dev, 43 (2017), pp. 481-488
[5]
NF Steutel, MA Benninga, MW Langendam, JJ Korterink, F Indrio, H Szajewska, et al.
Developing a core outcome set for infant colic for primary, secondary and tertiary care settings: a prospective study.
[6]
DF Baaleman, C Di Lorenzo, MA Benninga, M Saps.
The effects of the Rome IV criteria on pediatric gastrointestinal practice.
Curr Gastroenterol Rep, 22 (2020), pp. 21
[7]
J Zeevenhooven, IJ Koppen, MA. Benninga.
The new Rome IV criteria for functional gastrointestinal disorders in infants and toddlers.
Pediatr Gastroenterol Hepatol Nutr, 20 (2017), pp. 1-13
[8]
SA Reijneveld, MF van der Wal, E Brugman, RA Sing.
Verloove-Vanhorick SP. Infant crying and abuse.
Lancet, 364 (2004), pp. 1340-1342
[9]
S Daelemans, L Peeters, B Hauser, Y. Vandenplas.
Recent advances in understanding and managing infantile colic.
[10]
A Bilgin, D. Wolke.
Development of comorbid crying, sleeping, feeding problems across infancy: Neurodevelopmental vulnerability and parenting.
Early Hum Dev, 109 (2017), pp. 37-43
[11]
D. Greeff.
Infant colic.
Prof Nurs Today, 21 (2017), pp. 30-33
[12]
DA. Drossman.
Functional gastrointestinal disorders: history, pathophysiology, clinical features and Rome IV.
[13]
A Sridevi, H Jonathan, BA Wicaksono, B Hegar.
Indonesian pediatricians’ knowledge of Rome IV criteria and their therapeutic approach to infantile colic.
Paediatr Indones, 62 (2022),
[14]
D Yılmaz, Ö Yayıcı Köken, E Gürkaş, N Ceylan, AN Çıtak Kurt.
Infantile spasms: knowledge, attitude, and practice of pediatricians in Turkey.
Epilepsy Behav, 126 (2022),
[15]
ML Cirgin Ellett, D Murphy, L Stroud, RA Shelton, A Sullivan, SG Ellett, et al.
Development and psychometric testing of the infant colic scale.
Gastroenterol Nurs, 26 (2003), pp. 96-103
[16]
B Cetinkaya, Z. Başbakkal.
A validity and reliability study investigating the Turkish version of the Infant Colic Scale.
Gastroenterol Nurs, 30 (2007), pp. 84-90
[17]
RM. Warner.
Applied Statistics: From Bivariate Through Multivariate Analyses Sage Publications.
Sage Publications, (2008),
[18]
A. Field.
Discovering Statistics Using SPSS.
3rd ed., Sage Publications, (2009),
[20]
C Canivet, B Hagander, I Jakobsson, J. Lanke.
Infantile colic–less common than previously estimated?.
Acta Paediatr, 85 (1996), pp. 454-458
[21]
V Sounderajah, H Ashrafian, RM Golub, S Shetty, J De Fauw, L Hooft, et al.
Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol.
[22]
NE Dubois, KE. Gregory.
Characterizing the intestinal microbiome in infantile colic: findings based on an integrative review of the literature.
Biol Res Nurs, 18 (2016), pp. 307-315
[23]
W Zhu, N Zeng, N. Wang.
Sensitivity, specificity, accuracy, associated confidence interval and ROC analysis With practical SAS implementations.
NESUG Proceedings: Health Care and Life Sciences,
Copyright © 2022. Sociedade Brasileira de Pediatria
Idiomas
Jornal de Pediatria (English Edition)

Subscribe to our newsletter

Article options
Tools
en pt
Taxa de publicaçao Publication fee
Os artigos submetidos a partir de 1º de setembro de 2018, que forem aceitos para publicação no Jornal de Pediatria, estarão sujeitos a uma taxa para que tenham sua publicação garantida. O artigo aceito somente será publicado após a comprovação do pagamento da taxa de publicação. Ao submeterem o manuscrito a este jornal, os autores concordam com esses termos. A submissão dos manuscritos continua gratuita. Para mais informações, contate assessoria@jped.com.br. Articles submitted as of September 1, 2018, which are accepted for publication in the Jornal de Pediatria, will be subject to a fee to have their publication guaranteed. The accepted article will only be published after proof of the publication fee payment. By submitting the manuscript to this journal, the authors agree to these terms. Manuscript submission remains free of charge. For more information, contact assessoria@jped.com.br.
Cookies policy Política de cookies
To improve our services and products, we use "cookies" (own or third parties authorized) to show advertising related to client preferences through the analyses of navigation customer behavior. Continuing navigation will be considered as acceptance of this use. You can change the settings or obtain more information by clicking here. Utilizamos cookies próprios e de terceiros para melhorar nossos serviços e mostrar publicidade relacionada às suas preferências, analisando seus hábitos de navegação. Se continuar a navegar, consideramos que aceita o seu uso. Você pode alterar a configuração ou obter mais informações aqui.