Journal Information
Vol. 97. Issue 3.
Pages 256-259 (May - June 2021)
Share
Share
Download PDF
More article options
Vol. 97. Issue 3.
Pages 256-259 (May - June 2021)
Editorial
Open Access
Merging Pediatric Index of Mortality (a physiologic instability measure), lactate, and Systemic Inflammation Mortality Risk to better predict outcome in pediatric sepsis
Visits
2284
Christopher M. Horvat, Dennis W. Simon, Zachary Aldewereld, Idris Evans, Rajesh Aneja, Joseph A. Carcillo
Corresponding author
carcilloja@ccm.upmc.edu

Corresponding author.
University of Pittsburgh Medical Center (UPMC), Children’s Hospital of Pittsburgh, Department of Critical Care Medicine, Pittsburgh, PA, USA
Related content
J Pediatr (Rio J). 2021;97:287-9410.1016/j.jped.2020.07.008
Cristian Tedesco Tonial, Caroline Abud Drumond Costa, Gabriela Rupp Hanzen Andrades, Francielly Crestani, Francisco Bruno, Jefferson Pedro Piva, Pedro Celiny Ramos Garcia
This item has received

Under a Creative Commons license
Article information
Full Text
Bibliography
Download PDF
Statistics
Full Text

One in five deaths occur globally as a result of sepsis, with the majority occurring in newborns and children in resource-poor settings.1 Whereas public health efforts in Brazil have reduced pediatric deaths from sepsis by 75%, mortality in hospitalized children remains stubbornly high at above 20%, particularly in the São Paulo region.2 In today’s journal, Tonial et al. report the first attempt at merging PIM2, a measure of physiologic instability, with lactate, and Systemic Inflammation Mortality Risk in order to better predict mortality in pediatric sepsis.3 They found that the Youden’s J test best cut-off value PIM2 score has an accuracy (true positives + true negatives/all tests) of 0.880, which increases to 0.945 with the addition of the best Youden’s test cutoffs for lactate plus the two components of Systemic Inflammation Mortality Risk, namely C-reactive protein and ferritin, in predicting pediatric sepsis mortality in their PICU.

The PIM2, PRISM IV, TOPICC, PEMOD, and PELOD-2 scores are among the most commonly used measures of physiological instability and organ dysfunction-related mortality risk in pediatric intensive care. Over the years, the PIM2, PRISM IV, and TOPICC scores have added high-risk and low-risk diagnoses to further improve their ability to predict mortality and morbidity. Specifically, the PIM2 score records lowest systolic blood pressure, pupillary reaction, paO2, and base excess, as well as highest FiO2, with added patient care characteristics including mechanical ventilation, elective admission, post-surgical admission, cardiopulmonary bypass, and high-risk diagnosis (cardiac arrest, cerebral hemorrhage, hypoplastic left heart syndrome, acute or chronic liver failure, or neurodegenerative disease) or low risk diagnosis (asthma, bronchiolitis, obstructive sleep apnea, diabetic ketoacidosis, or central apnea).4

Children who die from septic shock do so in two epochs: those who die within the first 48 h of unremitting shock, and those who die 7 or more days later of multiple organ dysfunction syndrome due to an inability to remove infection, or control inflammation and coagulation.5–8 Use of septic shock recognition tools as well as timely resuscitation and antibiotic treatment in the emergency department results in reduced mortality.9 In the PICU, de Oliveira and colleagues showed that children in São Paulo who die from septic shock can be saved with continuous monitoring of ScVO2, reducing mortality from 42% to 11% when therapies were directed at keeping ScVO2 > 70% and maintaining normal perfusion pressure for age, defined by Mean Arterial Pressure minus Central Venous Pressure.10 ScVO2 > 70% is attained by targeting a goal cardiac index between 3.3 and 6.0 L/min/m2 with normal perfusion pressure using variable combinations of fluid loading or diuresis, and inotropes with vasopressors or vasodilators depending on changing hemodynamic phenotypes over time (including hypovolemic, hypervolemic, hypodynamic, hyperdynamic, vasodilated, and vasoconstricted phenotypes).10–15

Measurement of ScVO2 requires placement of a central venous catheter with a sampling port at the RA-SVC or RA-IVC junction. ScVO2 is decreased when oxygen extraction is increased. This can occur with low cardiac output < 3.3 L/min/m2, causing low oxygen delivery, or very high cardiac output > 6.0 L/min/m2, causing increased cardiac oxygen consumption.14 For clinicians unable to measure ScVO2, capillary refill < 3 s with normal pulses is the next best alternative; however, it is next best as the patients in the de Oliveira study who showed 42% mortality were those for whom cardiovascular therapies were targeted to capillary refill < 3 s, not ScVO2 > 70%.10 When ScVO2 < 70% is accompanied by lactate > 4 there is life-threatening oxygen debt.15 Pediatric septic shock non-survivors remain in a very high oxygen extraction state for the first 48 h of PICU presentation, whereas survivors normalize their very high oxygen extraction by attaining cardiac index goals of 3.3–6.0 L/min/m2 with the aid of bedside hemodynamic support.15 Unlike adults, essentially all deaths in the first 48 h of pediatric septic shock are due to oxygen debt.11 Elevation of lactate in the absence of a low ScVO2 does not indicate oxygen debt but rather metabolic alteration or Type II lactatemia. Lactate likely improves PIM2 prediction of death only in children with oxygen debt, not in those without oxygen debt.

Despite ScVO2 directed resuscitation, 11% of children still died in the São Paulo Brazilian PICU setting.10 Our colleagues from Porto Alegre were the first to recognize that these children are dying from uncontrolled inflammation leading to Multiple Organ Dysfunction Syndrome and ensuant Multiple Organ Failure, when they reported the relationship of hyperferritinemia to late (>7 days) MOF-related deaths.16 Of course, it had been recognized for a very long time that uncontrolled systemic inflammation kills patients. Roger Bone elegantly described a progression from systemic inflammatory response (SIRS), with high TNF alpha and IL-1 produced to kill infection, to compensatory inflammatory response (CARS), with high IL-10 released to control inflammation; to immunologic dissonance with very high levels of IL-6 and IL-10 indicative of uncontrolled inflammation and inability to kill infection. Mortality was highest with immunologic dissonance.17

When clinicians applied Bone’s conceptual model at the bedside it worked to a large degree, but not completely so. Interleukin-6 induces production and release of the pattern recognition receptor C-reactive protein (CRP), which binds bacteria as well as necrotic host cells to complement and delivers these antigens to resident macrophages for processing and clearance by the reticuloendothelial system. Indeed, patients with very high CRP levels were noted to be sicker, and persistently high CRP levels were related to unremitting infection, sometimes due to incorrect antibiotics.18 These patients could be saved with effective source control and appropriate use of sensitive antibiotics. In Brazil, this means that every hospital laboratory must be able to perform antimicrobial sensitivity testing. Nevertheless, many patients dying with MOF who received what was perceived as adequate source control and antibiotic therapies were not identifiable by CRP production alone.19

The Porto Alegre group showed that these MOF patients could be recognized by the presence of very high ferritin levels in conjunction with high CRP levels.16 Wang and colleagues have recently explained the scientific basis for this phenomenon, showing that the sequence of antigen exposures predicts development of hyperferritinemia and MOF in sepsis. Exposure of rodents to viral followed by bacterial antigen leads to hyperferritinemic sepsis, MOF, and death caused by macrophage activation that is both T-cell and interferon gamma independent, whereas exposure to viral:viral, bacterial:bacterial, or bacterial:viral antigen sequences does not induce hyperferritinemic sepsis, MOF, or high systemic inflammation mortality risk.20 At the Brazilian bedside, hyperferritinemic sepsis should raise suspicion for intracellular organisms such as Leishmaniasis, Babesiosus, Dengue virus, COVID-19, and DNA viruses.21,22 This should alert the physician to order different diagnostic tests and consider different treatment options. In addition to anti-viral and anti-parasitic medications, anti-inflammatory therapies can be considered to quell macrophage activation syndrome. Ferritin itself is pro-inflammatory, participating in feed forward inflammation during inflammasome activation. It is also immune suppressive, causing concomitant IL-10 production and inhibition of lymphopoiesis, leading to anergy.23 Ideal therapies for hyperferritinemic sepsis need to kill viruses/intracellular organisms, reduce inflammation, and restore immunity.

Analysis of paired CRP and ferritin levels in children provides the Systemic Inflammation Mortality Risk for MOF deaths beyond 7 days.19,24,25 Low CRP and low ferritin are associated with low ‘near zero’ risk; elevated CRP or ferritin alone confers intermediate ‘single digit percentage’ risks; and combined high CRP and high ferritin confer very high mortality risks of 22–45%. These deaths in patients with high Systemic Inflammation Mortality Risk occur in part due to the development of inflammation pathobiology phenotypes, including: (1) immunoparalysis: a condition characterized by a whole blood ex vivo TNF-alpha response to endotoxin <200 pg/mL for longer than three days with unremitting infection, that is reversible with proper antibiotics, source control and targeted immune modulation with low dose GM-CSF26–28; (2) thrombocytopenia-associated MOF: a condition characterized by thrombotic microangiopathy, AKI, elevated LDH, and ADAM TS13 activity < 57% that is reversible with plasma exchange27–31; and (3) hyper-inflammation32 related to sequential liver failure-associated MOF, a condition characterized by virus-induced lymphoproliferative disease that is reversible with antivirals and anti-proliferative mAbs such as Rituximab for EBV infection,33 and macrophage activation syndrome, a condition characterized by hyperferritinemia with disseminated intravascular coagulation and hepatobiliary dysfunction that is reversible with methylprednisone, interleukin 1 receptor antagonist, IVIG, and plasma exchange.27,28,32,34,35

As low-cost and widely available biomarkers, the combination of CRP and ferritin provides a simple and accessible approach to assessment of systemic inflammation mortality risk among children suspected or confirmed to have severe and/or systemic infection. Our own work examining children with sepsis, as well as a diagnostically diverse cohort of all hospitalized children at our own institution, demonstrated that children with both elevated CRP and ferritin are substantially and significantly more likely to die than children with elevation of one or neither of the biomarkers.19,24,25 Incorporating CRP and ferritin in prediction models that simultaneously leverage other available discrete data at the bedside may be an avenue for developing meaningful clinical decision support tools for general risk assessment in the PICU, though much work remains in this arena. While PIM2, PRISM IV, PELOD-2 and other comparable scores have demonstrated good predictive validity in cohort studies, their utility in predicting individual risk is unclear and even controversial. A model that demonstrates excellent outcome discrimination by assessment of the area under the receiver operating characteristics curve in a cohort does not necessarily indicate that a high positive predictive value will be observed when using the model at the bedside.36 While the present work by Tonial et al. requires additional validation to better understand the potential value in individual risk stratification,3 it does add to a growing body of evidence indicating that both CRP and ferritin should be incorporated into the standard laboratory evaluation of children presenting with concern for serious infection or confirmed sepsis.19 Merging physiologic instability scoring systems with the Systemic Inflammation Mortality Risk using paired CRP and ferritin is an exciting way forward for resource-poor and resource-rich settings alike.

In the resource-rich setting, proteomics and transcriptomics are considered the superior approach to pediatric sepsis mortality risk stratification and disease progression tracking. Specific to proteomics, the PERSEVERE biomarkers C–C chemokine ligand 3 (CCL3), interleukin 8 (IL8), heat shock protein 70 kDa 1B, granzyme B, and matrix metallopeptidase 8 used in the Pediatric Sepsis Biomarker Risk Model had an area under the receiver operating characteristic curve of 0.73 (95% CI, 0.59−0.87; p = 0.002) for estimating the risk of hospital mortality in US children with community-acquired septic shock.37 In multivariable analyses, the Pediatric Sepsis Biomarker Risk Model was not independently associated with increased odds of the composite outcome of mortality or persistent, serious deterioration of health-related quality of life greater than 25% below baseline. A new decision tree using the Pediatric Sepsis Biomarker Risk Model biomarkers had an area under the receiver operating characteristic curve of 0.87 (95% CI, 0.80−0.95) for estimating the risk of persistent, serious deterioration in health-related quality of life at 3 months among children who survived septic shock.37 Specific to transcriptomic prognostic models for 30-day mortality in five cohorts of community-onset sepsis patients including children and adults showing summary AUROCs ranging from 0.765 to 0.89, similar performance was observed in four cohorts of hospital-acquired sepsis. These transcriptomic analyses identified adaptive, coagulopathic, and inflammatory endotypes.38 Normalization of these pathologic transcriptomes over time was associated with response to therapies in children with sepsis.39 Until proteomics and transcriptomics become available with lower resource needs, the merging of PIM2, lactate, and Systemic Inflammation Mortality Risk provides a practical path to better mortality risk stratification. Further work will be needed to determine whether tracking the physiologic components of PIM2, lactate, CRP, and ferritin over time will help bedside clinicians assess response to therapies and improve outcomes.

Conflicts of interest

Dr Carcillo is funded by grant 2R01GM108618 from by the National institutes of Health for this work. The other authors have no conflicts of interest regarding this work.

References
[1]
K.E. Rudd, S.C. Johnson, K.M. Agesa, K.A. Shackelford, D. Tsoi, D.R. Kievlan, et al.
Global, regional, and national sepsis incidence and mortality, 1990–2017: analysis for the Global Burden of Disease Study.
[2]
C.M. Mangia, N. Kissoon, O.A. Branchini, M.C. Andrade, B.I. Kopelman, J. Carcillo.
Bacterial sepsis in Brazilian children: a trend analysis from 1992 to 2006.
[3]
C.T. Tonial, C.A. Costa, G.R. Andrades, F. Crestani, F. Bruno, J.P. Piva, et al.
Performance of prognostic markers in pediatric sepsis.
J Pediatr (Rio J), 97 (2021), pp. 287-294
[4]
M.P. Arias Lopez, A.L. Fernández, M.E. Ratto, L. Saligari, A.S. Serrate, I.J. Ko, et al.
Pediatric Index of Mortality 2 as a predictor of death risk in children admitted to pediatric intensive care units in Latin America: a prospective, multicenter study.
J Crit Care, 30 (2015), pp. 1324-1330
[5]
S.L. Weiss, F. Balamuth, J. Hensley, J.C. Fitzgerald, J. Bush, V.M. Nadkarni, et al.
The epidemiology of hospital death following pediatric severe sepsis: when, why, and how children with sepsis die.
Pediatr Crit Care Med, 18 (2017), pp. 823-830
[6]
J.C. Lin, P.C. Spinella, J.C. Fitzgerald, M. Tucci, J.L. Bush, V.M. Nadkarni, et al.
New or progressive multiple organ dysfunction syndrome in pediatric severe sepsis: a sepsis phenotype with higher morbidity and mortality.
Pediatr Crit Care Med., 18 (2017), pp. 8-16
[7]
B.S. Podd, D.W. Simon, S. Lopez, A. Nowalk, R. Aneja, J.A. Carcillo.
Rationale for adjunctive therapies for pediatric sepsis induced multiple organ failure.
Pediatr Clin North Am, 64 (2017), pp. 1071-1088
[8]
J.A. Carcillo, B. Podd, R. Aneja, S.L. Weiss, M.W. Hall, T.T. Cornell, et al.
Pathophysiology of pediatric multiple organ dysfunction syndrome.
Pediatr Crit Care Med, 18 (2017), pp. S32-S45
[9]
I.V. Evans, G.S. Phillips, E.R. Alpern, D.C. Angus, M.E. Friedrich, N. Kissoon, et al.
Association between the New York sepsis care mandate and in-hospital mortality for pediatric sepsis.
JAMA, 320 (2018), pp. 358-367
[10]
C.F. de Oliveira, D.S. de Oliveira, A.F. Gottschald, J.D. Moura, G.A. Costa, A.C. Ventura, et al.
ACCM/PALS haemodynamic support guidelines for paediatric septic shock: an outcomes comparison with and without monitoring central venous oxygen saturation.
Intensive Care Med, 34 (2008), pp. 1065-1075
[11]
C.F. de Oliveira, E.J. Troster, J.A. Carcillo.
A beneficial role of central venous oxygen saturation-targeted septic shock management in children: follow the pediatric story, not only the adult story.
Pediatr Crit Care Med, 15 (2014), pp. 380-382
[12]
G. Ceneviva, J.A. Paschall, F. Maffei, J.A. Carcillo.
Hemodynamic support in fluid-refractory pediatric septic shock.
Pediatrics, 102 (1998), pp. e19
[13]
A.L. Davis, J.A. Carcillo, R.K. Aneja, A.J. Deymann, J.C. Lin, T.C. Nguyen, et al.
American College of Critical Care Medicine clinical practice parameters for hemodynamic support of pediatric and neonatal septic shock.
Crit Care Med, 45 (2017), pp. 1061-1093
[14]
A. Deep, C.D. Goonasekera, Y. Wang, J. Brierley.
Evolution of haemodynamics and outcome of fluid-refractory septic shock in children.
Intensive Care Med, 39 (2013), pp. 1602-1609
[15]
C.D. Goonasekera, J.A. Carcillo, A. Deep.
Oxygen delivery and oxygen consumption in pediatric fluid refractory septic shock during the first 42 h of therapy and their relationship to 28-day outcome.
Front Pediatr, 6 (2018), pp. 314
[16]
P.C. Garcia, F. Longhi, R.G. Branco, J.P. Piva, D. Lacks, R.C. Tasker.
Ferritin levels in children with severe sepsis and septic shock.
Acta Paediatr, 96 (2007), pp. 1829-1831
[17]
R.C. Bone.
Immunologic dissonance: a continuing evolution in our understanding of the systemic inflammatory response syndrome (SIRS) and the multiple organ dysfunction syndrome (MODS).
Ann Intern Med, 125 (1996), pp. 680-687
[18]
X. Schmit, J.L. Vincent.
The time course of blood C-reactive protein concentrations in relation to the response to initial antimicrobial therapy in patients with sepsis.
Infection, 36 (2008), pp. 213-219
[19]
M.D. Taylor, V. Allada, M.L. Moritz, A.J. Nowalk, R. Sindhi, R.K. Aneja, et al.
Use of C-reactive protein and ferritin biomarkers in daily pediatric practice.
Pediatr Rev, 41 (2020), pp. 172-183
[20]
A. Wang, S.D. Pope, J.S. Weinstein, S. Yu, C. Zhang, C.J. Booth, et al.
Specific sequences of infectious challenge lead to secondary hemophagocytic lymphohistiocytosis-like disease in mice.
Proc Natl Acad Sci U S A, 116 (2019), pp. 2200-2209
[21]
J.A. Carcillo, K.K. Kernan, C.M. Horvat, D.W. Simon, R.K. Aneja.
Why and how is hyperferritinemic sepsis different from sepsis without hyperferritinemia?.
Pediatr Crit Care Med, 21 (2020), pp. 509-512
[22]
J.A. Carcillo, D.W. Simon, B.S. Podd.
How we manage hyperferritinemic sepsis-related multiple organ dysfunction syndrome/macrophage activation syndrome/secondary hemophagocytic lymphohistiocytosis histiocytosis.
Pediatr Crit Care Med, 16 (2015), pp. 598-600
[23]
K.F. Kernan, J.A. Carcillo.
Hyperferritinemia and inflammation.
Int Immunol, 29 (2017), pp. 401-409
[24]
C.M. Horvat, J. Bell, S. Kantawala, A.K. Au, R.S. Clark, J.A. Carcillo.
C-Reactive protein and ferritin are associated with organ dysfunction and mortality in hospitalized children.
Clin Pediatr (Phila), 58 (2019), pp. 752-760
[25]
J.A. Carcillo, K. Sward, E.S. Halstead, R. Telford, A. Jimenez-Bacardi, B. Shakoory, et al.
A systemic inflammation mortality risk assessment contingency table for severe sepsis.
Pediatr Crit Care Med, 18 (2017), pp. 143-150
[26]
M.W. Hall, N.L. Knatz, C. Vetterly, S. Tomarello, M.D. Wewers, H.D. Volk, et al.
Immunoparalysis and nosocomial infection in children with multiple organ dysfunction syndrome.
Intensive Care Med, 37 (2011), pp. 525-532
[27]
J.A. Carcillo, R.A. Berg, D. Wessel, M. Pollack, K. Meert, M. Hall, et al.
A multicenter network assessment of three inflammation phenotypes in pediatric sepsis induced multiple organ failure.
Pedatr Crit Care Med, 20 (2019), pp. 1137-1146
[28]
J.A. Carcillo, E.S. Halstead, M.W. Hall, T.C. Nguyen, R. Reeder, R. Aneja, et al.
Three hypothetical inflammation pathobiology phenotypes and pediatric sepsis-induced multiple organ failure outcome.
Pediatr Crit Care Med, 18 (2017), pp. 513-523
[29]
T.C. Nguyen, Y.Y. Han, J.E. Kiss, M.W. Hall, A.C. Hassett, R. Jaffe, et al.
Intensive plasma exchange increases a disintegrin and metalloprotease with thrombospondin motifs-13 activity and reverses organ dysfunction in children with thrombocytopenia-associated multiple organ failure.
Crit Care Med, 36 (2008), pp. 2878-2887
[30]
E. Sevketoglu, D. Yildizdas, O.O. Horoz, H.S. Kihtir, T. Kendirli, S. Bayraktar, et al.
Use of therapeutic plasma exchange in children with thrombocytopenia-associated multiple organ failure in the Turkish thrombocytopenia-associated multiple organ failure network.
Pediatr Crit Care Med, 15 (2014), pp. e354-9
[31]
J.D. Fortenberry, T. Nguyen, J.R. Grunwell, R.K. Aneja, D. Wheeler, M. Hall, et al.
Thrombocytopenia-associated multiple organ failure (TAMOF) network study group.
Crit Care Med, 47 (2019), pp. e173-e181
[32]
D. Demirkol, D. Yildizdas, B. Bayrakci, B. Karapinar, T. Kendirli, T.F. Koroglu, et al.
Hyperferritinemia in the critically ill child with secondary hemophagocytic lymphohistiocytosis/sepsis/multiple organ dysfunction syndrome/macrophage activation syndrome: what is the treatment?.
Crit Care, 16 (2012), pp. R52
[33]
L. Doughty, R.S. Clark, S.S. Kaplan, H. Sasser, J. Carcillo.
sFas and sFas ligand and pediatric sepsis-induced multiple organ failure syndrome.
Pediatr Res, 52 (2002), pp. 922-927
[34]
B. Shakoory, J.A. Carcillo, W.W. Chatham, R.L. Amdur, H. Zhao, C.A. Dinarello, et al.
Interleukin-1 receptor blockade is associated with reduced mortality in sepsis patients with features of macrophage activation syndrome: reanalysis of a prior phase III trial.
Crit Care Med, 44 (2016), pp. 275-281
[35]
U. Emmenegger, U. Frey, A. Reimers, C. Fux, D. Semela, P. Cottagnoud, et al.
Hyperferritinemia as indicator for intravenous immunoglobulin treatment in reactive macrophage activation syndromes.
Am J Hematol, 68 (2001), pp. 4-10
[36]
T. Saito, M. Rehmsmeier.
The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets.
[37]
H.R. Wong, R.W. Reeder, R. Banks, R.A. Berg, K.L. Meert, M.W. Hall, et al.
Biomarkers for estimating risk of hospital mortality and long-term quality-of-life morbidity after surviving pediatric septic shock: a secondary analysis of the life after pediatric sepsis evaluation investigation.
Pediatr Crit Care Med, 22 (2021), pp. 8-15
[38]
T.E. Sweeney, T.D. Azad, M. Donato, W.A. Haynes, T.M. Perumal, R. Henao, et al.
Unsupervised analysis of transcriptomics in bacterial sepsis across multiple datasets reveals three robust clusters.
Crit Care Med, 46 (2018), pp. 915-925
[39]
H.R. Wong, N.Z. Cvijanovich, N. Anas, G.L. Allen, N.J. Thomas, M.T. Bigham, et al.
Endotype transitions during the acute phase of pediatric septic shock reflect changing risk and treatment response.
Crit Care Med, 46 (2018), pp. e242-e249

See article by Tonial et al. in pages 287–94.

Idiomas
Jornal de Pediatria (English Edition)
Article options
Tools
en pt
Taxa de publicaçao Publication fee
Os artigos submetidos a partir de 1º de setembro de 2018, que forem aceitos para publicação no Jornal de Pediatria, estarão sujeitos a uma taxa para que tenham sua publicação garantida. O artigo aceito somente será publicado após a comprovação do pagamento da taxa de publicação. Ao submeterem o manuscrito a este jornal, os autores concordam com esses termos. A submissão dos manuscritos continua gratuita. Para mais informações, contate assessoria@jped.com.br. Articles submitted as of September 1, 2018, which are accepted for publication in the Jornal de Pediatria, will be subject to a fee to have their publication guaranteed. The accepted article will only be published after proof of the publication fee payment. By submitting the manuscript to this journal, the authors agree to these terms. Manuscript submission remains free of charge. For more information, contact assessoria@jped.com.br.
Cookies policy Política de cookies
To improve our services and products, we use "cookies" (own or third parties authorized) to show advertising related to client preferences through the analyses of navigation customer behavior. Continuing navigation will be considered as acceptance of this use. You can change the settings or obtain more information by clicking here. Utilizamos cookies próprios e de terceiros para melhorar nossos serviços e mostrar publicidade relacionada às suas preferências, analisando seus hábitos de navegação. Se continuar a navegar, consideramos que aceita o seu uso. Você pode alterar a configuração ou obter mais informações aqui.