Journal Information
Vol. 91. Issue 1.
Pages 87-92 (January - February 2015)
Share
Share
Download PDF
More article options
Visits
...
Vol. 91. Issue 1.
Pages 87-92 (January - February 2015)
Original article
Open Access
Evaluation of clinical and laboratory variables associated with anemia in pediatric patients on hemodialysis
Avaliação de variáveis clínicas e laboratoriais associadas à anemia em pacientes pediátricos em hemodiálise
Visits
...
Johnathan S. de Freitas
Corresponding author
sanfet2010@hotmail.com

Corresponding author.
, Paulo Sucasas Costa, Luciane Rezende Costa, Alessandra V. Naghettini
Universidade Federal de Goiás (UFG), Goiânia, GO, Brazil
Article information
Abstract
Full Text
Bibliography
Download PDF
Statistics
Figures (1)
Tables (2)
Table 1. Association between clinical and laboratory variables and the occurrence of anemia in children with chronic kidney disease on hemodialysis (357 files of 29 patients).
Table 2. Clinical factors predictive of anemia in children and adolescents on hemodialysis.
Show moreShow less
Abstract
Objective

To identify the occurrence of anemia in pediatric patients on hemodialysis and the association between hemoglobin levels and anemia in CKD-related variables.

Methods

This was a retrospective study. Patients aged up to 18 years with chronic kidney disease undergoing hemodialysis at this service between January of 2009 and December of 2010 were selected. Clinical and laboratory data were obtained from medical records. Statistical analysis was performed with chi-squared test, Student's t-test and general estimating equations (GEE) using SPSS 20.0, assuming a significance level of 5%.

Results

A total of 357 medical records depicting the monthly evolution of 29 patients were analyzed. The most common etiology for chronic kidney disease was malformations of the genitourinary tract (28%). Hemoglobin showed a mean (standard deviation) value of 9.20 (1.8) g/dL, with the occurrence of anemia in 65.3% of cases. Anemia was associated with hospitalization; antibiotic use; transfusion; use of intravenous iron hydroxide; low values of creatinine, hematocrit, and albumin; and high values of ferritin, aluminum, and equilibrated Kt/V (p<0.05). The odds ratio for anemia with the use of intravenous iron hydroxide was 0.36 (95% CI: 0.25 to 0.89), i.e., a 2.78-fold higher chance of developing anemia without the use of this medication.

Conclusions

Anemia predominated in children and adolescents with chronic kidney disease; intravenous iron hydroxide use was a protective factor.

Keywords:
Anemia
Adolescent
Child
Kidney dialysis
Renal failure
Resumo
Objetivo

Identificar a ocorrência de anemia entre pacientes pediátricos em hemodiálise e a associação entre os valores de hemoglobina e variáveis relacionadas à anemia na DRC.

Métodos

Estudo retrospectivo. Selecionados pacientes até 18 anos de idade com doença renal crônica em hemodiálise no serviço entre janeiro de 2009 a dezembro de 2010. Verificados prontuários para coleta de dados clínicos e laboratoriais. Análise estatística com testes de qui-quadrado, t de Student e General Estimating Equations (GEE) em programa Statistical Package for the Social Sciences 20.0, assumindo-se nível de significância de 5%.

Resultados

Analisadas 357 fichas de evolução médica mensal de 29 pacientes. A etiologia mais frequente para a doença renal crônica foram as malformações do trato genito-urinário (28%). Hemoglobina apresentou valor médio (desvio padrão) de 9,20 (1,8) g/dL, com ocorrência de anemia em 65,3% das consultas. Anemia associou-se a internação, uso de antibiótico, transfusão, uso de hidróxido de ferro endovenoso, valores baixos de creatinina, hematócrito e albumina e valores altos de ferritina, alumínio e Kt/V equilibrado (p<0,05). A odds ratio para anemia com uso de hidróxido de ferro endovenoso foi 0,36 (95% IC 0,25-0,89), ou seja, uma chance 2,78 vezes maior de desenvolver anemia sem o uso dessa medicação.

Conclusões

A anemia predominou em crianças e adolescentes com doença renal crônica, tendo como fator protetor o uso de hidróxido de ferro endovenoso.

Palavras-chave:
Anemia
Adolescente
Criança
Diálise renal
Insuficiência renal
Full Text
Introduction

The prevalence of renal replacement therapy in children aged 0-19 years in Brazil is of 23:1,000,000 in the related age range. The survival of these children is approximately 30-fold lower than that of their healthy peers. The main cause of mortality in this group is cardiovascular disease, accounting for 40% of deaths, and anemia is identified as a factor related to higher rates of mortality.1,2

The prevalence of anemia in Brazilian children without a diagnosis of any disease ranges from 25.6% to 63.7%,3,4 while in children with chronic kidney disease (CKD), chronic hyporegenerative anemia is common. If untreated, it can lead to growth and development impairment, left ventricular hypertrophy, and tachycardia. When on hemodialysis, blood loss in the dialysis circuit and blood collections due to frequent examinations contribute to further exacerbation of anemia.5

Anemia is a limiting survival factor of children on hemodialysis. Thus, this study aimed to identify the occurrence of anemia in pediatric patients on hemodialysis and the association between hemoglobin levels and anemia-related variables in CKD.

MethodsStudy design and setting

This was a retrospective cohort study approved by the Ethics Committee in Human and Animal Medical Research, of Hospital das Clínicas da Universidade Federal de Goiás (HC/UFG). Data were collected at the Renal Replacement Therapy Service, Hemodialysis Sector, HC/UFG.

Study participants

The study participants were subjects undergoing hemodialysis in HC/UFG between January of 2009 and December of 2010 who met the inclusion criteria: age<18 years and diagnosis of CKD. Cases whose medical records were not available were excluded.

Data collection and variables

An expert researcher (a pediatric nephrologist) collected the data on the variables of interest from patients’ records, specifically the records of monthly medical progress, and recorded them in digital form. Record survey was conducted at the Hemodialysis Sector and the Division of Medical Records and Health Information of HC/UFG.

The dependent variable was the occurrence of anemia, measured by hemoglobin levels (mg/dL).

The independent variables were: clinical variables - hospitalizations, antibiotic use in the month (whether the patient used or not), transfusions of packed red blood cells, and main drugs used to treat anemia (folic acid, B complex, vitamin C, erythropoietin, and intravenous iron hydroxide); laboratory variables - serum tests performed as routine at the service, defined by Resolution No. 154 of the Brazilian National Health Surveillance Agency, with relevant adjustments: urea at pre-hemodialysis session, creatinine, hemoglobin, hematocrit, pH, bicarbonate, ferritin, transferrin saturation index (TSI), iron, parathyroid hormone (PTH), albumin, and aluminum.6 Equilibrated Kt/V (EKT/V) was calculated as proposed by Fischbach et al.7

Statistical Analysis

Data were tabulated and analyzed using descriptive and inferential statistics using IBM SPSS Statistics software, release 20 (IBM Corporation, NY, USA).

After exploratory data analysis, continuous variables were categorized based on clinical parameters to better interpret the results, considering the limited number of subjects included in the study.

Serum hemoglobin values were classified as adequate or inadequate (anemia) as recommended by the National Institute for Health and Clinical Excellence (NICE) criteria of 2011,8 which considers both age groups (0-2 years and over 2 years). Student's t-test and the chi-squared test were then used to test associations between the described variables and the occurrence of anemia.

This longitudinal observational study included repeated measurements for the same variable in the same subject. As the dependent variable was categorical (having or not anemia), the analysis of variance for repeated measures could not be used. Moreover, multiple linear regression would be contraindicated because the assumption of independence of random variables could not be met, as the data had an interdependent association (repeated measures). Thus, the statistical analysis that would better determine predictive variables for anemia were the generalized estimating equations (GEEs).9

The GEE approach, which is an extension of generalized linear models, was developed to produce more efficient and less biased regression estimates for use with correlated data, as repeated longitudinal measures.10 Thus, GEE analyzes data depending on exposure over successive periods of time. For the analysis of this study, a model for the dependent variable “anemia” was created. The independent variables were “hospitalization”, “infection of double-lumen catheter (DLC)”, “antibiotic use,” “use of intravenous iron hydroxide.” Missing data were treated by the GEE mechanism that uses all available data to include evasion data. The odds ratio (OR) and confidence intervals (95% CI) were calculated to reflect the possible associations between predictive factors and dependent variable.

All statistical tests were two-tailed, with a significance level of 0.05.

Results

A total of 31 patients were included, but two were excluded due to inaccessible files. The final sample consisted of 29 patients, corresponding to 357 files of monthly clinical evolution throughout 24 months. Each patient was followed-up for one to 23 consultations. The mean age of patients at baseline was 10 years and 3 months (standard deviation [SD] 34.6 months). Of the 29 patients included, 21 were males (72.4%). The etiologies of CKD with their respective percentages are shown in Fig. 1.

Figure 1.

Etiology of chronic kidney disease.

HUS, hemolytic uremic syndrome; ARPKD, autosomal recessive polycystic kidney disease; FSGS, focal and segmental glomerulosclerosis; GUT, genitourinary tract.

Total=100%.

(0.13MB).

All patients but one had anemia at some point during their follow-up, which corresponded to 65.3% of consultations (n=233). The mean hemoglobin (g/dL) for the categorized groups were: anemic - 8.2g/dL (SD 1.2); non-anemic - 11.2g / dL (SD 1.0) (p<0.001). The mean hemoglobin value for the total sample was 9.2 (SD 1.8).

The bivariate analysis showed that anemia was associated with several clinical situations (Table 1): need for hospitalization, antibiotic use, transfusion, and less frequent use of intravenous iron hydroxide (p<0.05). As for the laboratory findings, anemia was related to lower levels of creatinine, hematocrit, and albumin, as well as to higher values of ferritin and aluminum, and EKt/V (Table 1).

Table 1.

Association between clinical and laboratory variables and the occurrence of anemia in children with chronic kidney disease on hemodialysis (357 files of 29 patients).

Independent variables  Total sample  Anemia
      Yes (n=233)  No (n=124)   
Medical history, n (%)
Need for hospitalization  316  52 (14.6%)  43 (20.7%)  9 (8.3%)  0.005a 
Antibiotic use  316  60 (16.8%)  48 (23.1%)  12 (11.1%)  0.010a 
Need for transfusion  316  11 (3.1%)  56 (26.9%)  3 (2.8%)  < 0.001a 
Double-lumen catheter infection  316  59 (16.5%)  10 (4.8%)  1 (0.9%)  0.105a 
Main medications used in the treatment of anemia, n (%)
Folic acid  339  333 (93.3%)  222 (98.7%)  111 (97.4%)  0.392a 
Vitamin B complex  339  332 (93.0%)  221 (98.2%)  111 (97.4%)  0.601a 
Vitamin C  339  327 (91.6%)  219 (97.3%)  108 (94.7%)  0.222a 
Erythropoietin  321  303 (84.9%)  201 (94.4%)  102 (94.4%)  0.977a 
Intravenous iron hydroxide  318  160 (44.8%)  89 (42.4%)  71 (65.7%)  < 0.001a 
Laboratory exams, mean (standard deviation)
Pre-hemodialysis session urea (mg/dL)  353  140.5 (42.3)  143.2 (42.9)  135.5 (40.8)  0.104b 
Creatinine (mg/dL)  352  6.2 (2.1)  5.9 (2.1)  6.6 (2.1)  0.002b 
Hematocrit (%)  356  28.3 (5.7)  25.1 (4.0)  34.3 (3.0)  < 0.001b 
Power of hydrogen (pH) (AV)  119  7.4 (0.6)  7.4 (0.1)  7.4 (0.1)  0.177b 
Bicarbonate (mmol/L)  120  20.2 (3.8)  19.8 (3.4)  20.7 (4.1)  0.197b 
Ferritin (ng/L)  351  709.3 (460.0)  754.0 (479.1)  625.1 (413.1)  0.012b 
Transferrin saturation index (TSI) (%)  357  35.5 (19.4)  36.0 (20.2)  34.5 (17.9)  0.474b 
Iron (uG/dL)  340  75.7 (93.5)  74.7 (95.9)  78.0 (89.8)  0.754b 
Parathormone (PTH) (pg/mL)  347  520.3 (495.1)  526.2 (519.4)  513.2 (450.7)  0.815b 
Albumin (g/dL)  343  3.9 (0.5)  3.8 (0.6)  4.1 (0.2)  < 0.001b 
Aluminum (ug/dL)  167  24.1 (28.6)  28.0 (30.8)  16.0 (21.6)  0.011b 
Equilibrated Kt/V (eKt/V) (AV)  294  1.9 (0.4)  2.0 (0.4)  1.8 (0.4)  0.004b 

Values in bold show statistically significant associations (p < 0.05).

AV, absolute value.

a

Pearson's chi-squared test.

b

Student's t-test.

Based on the results of the chi-squared and Student's t-test, the authors sought to determine the predictive clinical factors for anemia in this group of patients, considering the clinical variables that reached p<0.2 in the bivariate analysis. Among these, the variable “need for transfusion” was excluded, as it is not logical to evaluate this variable as a predictor of anemia. Thus, through GEE, it was demonstrated that the use of intravenous iron hydroxide was a protective factor for anemia (Table 2): the odds the children on hemodialysis to have anemia when using this medication was 0.36 times the odds of those who did not use it. Calculating the inverse of 0.36, it is observed that the chance of children who did not use intravenous iron hydroxide to have anemia was 2.78 times the chance of those who used it. The other variables did not show such significant predictors in the GEE.

Table 2.

Clinical factors predictive of anemia in children and adolescents on hemodialysis.

Independent variables  Odds ratio  95% Confidence intervalp value 
    Minimum  Maximum   
Need for hospitalization  1.00  0.52  1.93  0.988 
Antibiotic use  1.97  0.89  4.35  0.095 
Double-lumen catheter infection  4.13  0.49  34.60  0.191 
Use of intravenous iron hydroxide  0.36  0.25  0.89  < 0.001 
Discussion

This article emphasizes the high prevalence of anemia in children undergoing hemodialysis, in spite of the administration of recombinant human erythropoietin, and reaffirms the importance of intravenous iron supplementation in this population.

In relation to the clinical aspects of this study, it is noteworthy that, among the causes of CKD, congenital malformations of the urinary tract were the most prevalent, consistent with data obtained in developed and developing countries. However, neurogenic bladder was the second most prevalent cause observed in the present study and by researchers in Turkey, while glomerulonephritis and hereditary nephropathies were observed in developed countries. This may reflect a delay in the diagnosis and treatment of urological problems in developing countries.1,11–13

It is worth mentioning the use of medications for the treatment of CKD-associated anemia: 94.4% used erythropoietin; 50.5% used intravenous iron hydroxide in the present study. In the Brazilian dialysis census of 2011, prescriptions of these medications were 80.0% and 53.1%, respectively.14

The target-hemoglobin values for children and adolescents with CKD in the literature are divergent and have been revised over the past years.12,15,16 It is known that, in children and adolescents, age and gender should be taken into account in order to define target hemoglobin (Hgb) values.17 The recommendation of the last NICE criteria8 to define Hgb in patients as adequate or inadequate was adopted; therefore, the present study showed that Hgb values below the expected (anemia) were found in most monthly records of patients (65.3%), albeit still above those found in the last Brazilian dialysis census (39.3%).14

The authors believe that this difference is due to the study methodology, in which the same patient with anemia was reviewed during successive months, overestimating the frequency of anemia. In other studies, the hemoglobin value is usually considered in a single annual measurement (only one month of the year); the authors believe that this, conversely, may underestimate the prevalence of anemia. The mean Hgb observed was 9.2 (SD 1.8) g/dL. Comparatively, American data show a mean Hgb of 11.5 (SD 1.6) g/dL, with 68% of patients showing the target Hgb values.2 Data from the United Kingdom showed that most children on renal replacement therapy are anemic (47% of children on hemodialysis).18 A Polish study evidenced a mean Hgb of 10.91 (SD 1.2) g/dL.19

It is known that serum Hgb varies over the months, and withdrawal of erythropoietin in the last 60 days and hospitalization are related to Hgb<11g/dL. The longer the hospital length of stay, the greater the chance of more significant decreases in Hgb values.20 A 14.6% rate of hospitalization was observed in the present study, which may also have contributed to the anemia in the present population. Appropriate values of serum albumin are related to adequate Hgb values.21 Adequate mean serum albumin levels were observed in the present study, in agreement with other authors.15,19

Good markers of ferritin and transferrin saturation index (TSI) were identified, as the iron stores are replenished intravenously, as recommended in the literature for over a decade for hemodialysis patients receiving human erythropoietin.22,23

In 91.6% of the files, patients were receiving vitamin C, which is important for release of iron stored in the body, ensuring its availability for erythropoiesis.24,25

When the GEE was performed, intravenous iron hydroxide received during that month showed to be protective against anemia. The variable “transfused” was removed from the GEE analysis, as it ultimately reflects the event throughout the month. A blood sample was collected for the monthly tests on the first Thursday of each month. Those who met the criteria for receiving packed red blood cells (including the low Hg values) received the blood product in subsequent sessions. Thus, the answer “yes” to the variable “transfused” meant that, during that month, hemoglobin levels were very low and the patient required blood transfusion. Therefore, the transfusion event could be erroneously indicated as a risk factor for anemia in the GEE.

The main limitation of this study was its retrospective design, with analysis of data found in medical records. The lack of control in data collection and information loss constitute biases of retrospective studies. These were observed in the present study, compromising the sample and decreasing the power of data representation. The longitudinal repeated measures were treated by GEE.

In conclusion, the study population consisted of children and adolescents with end-stage renal disease on hemodialysis. Anemia was prevalent and the use of intravenous iron hydroxide was a protective factor.

Conflicts of interest

The authors declare to have no conflicts of interest.

References
[1]
J. Harambat, K.J. van Stralen, J.J. Kim, J. Tizard.
Epidemiology of chronic kidney disease in children.
Pediatr Nephrol., 27 (2012), pp. 363-373
[2]
A.M. Neu, D.L. Frankenfield.
Clinical outcomes in pediatric hemodialysis patients in the USA: lessons from CMS’ ESRD CPM project.
Pediatr Nephrol., 24 (2009), pp. 1287-1295
[3]
J.T. Costa, M.M. Bracco, P.A. Gomes, R.Q. Gurgel.
Prevalence of anemia among preschoolers and response to iron supplementation.
J Pediatr (Rio J)., 87 (2011), pp. 76-79
[4]
G.A. Bortolini, M.R. Vitolo.
Relationship between iron deficiency and anemia in children younger than 4 years.
J Pediatr (Rio J)., 86 (2010), pp. 488-492
[5]
D. Müller, S.L. Goldstein.
Hemodialysis in children with end-stage renal disease.
Nat Rev Nephrol., 7 (2011), pp. 650-658
[6]
Brasil. Agência Nacional de Vigilância Sanitária. Resolução RDC 154 de 15 de junho de 2004. Estabelece o regulamento técnico para o funcionamento dos serviços de diálise.
Diário Oficial da União, 115 (2004), pp. 64-69
[7]
M. Fischbach, A. Edefonti, C. Schröder, A. Watson, European Pediatric Dialysis Working Group.
Hemodialysis in children: general practical guidelines.
Pediatr Nephrol., 20 (2005), pp. 1054-1066
[8]
National Clinical Guideline Centre. Anaemia management in people with chronic kidney disease. London (UK): National Institute for Health and Clinical Excellence (NICE); Clinical guideline; no. 114. 2011; 38.
[9]
L.S. Guimarães, V.N. Hirakata.
Uso do modelo de equações de estimativas generalizadas na análise de dados longitudinais.
Rev HCPA., 32 (2012), pp. 501-511
[10]
K.Y. Liang, S.L. Zeger.
Longitudinal data analysis using generalized linear models.
Biometrika., 73 (1986), pp. 13-22
[11]
Kidney Disease: Improving Global Outcomes (KDIGO) CKD-MBD Work Group.
KDIGO clinical practice guideline for the diagnosis, evaluation, prevention, and treatment of chronic kidney disease-mineral and bone disorder (CKD-MBD).
Kidney Int Suppl., 113 (2009), pp. S1-S130
[12]
Kidney Disease: Improving Global Outcomes (KDIGO).
KDIGO clinical practice guideline for anemia in chronic kidney disease.
Kidney Int Suppl., 2 (2012), pp. S279-S335
[13]
R.G. VanDeVoorde, G.M. Barletta, D.H. Chand, I.G. Dresner, J. Lane, J.J. Lin, et al.
Blood pressure control in pediatric hemodialysis: the Midwest Pediatric Nephrology Consortium Study.
Pediatr Nephrol., 22 (2007), pp. 547-553
[14]
R.C. Sesso, A.A. Lopes, F.S. Thomé, J.R. Lugon, Y. Watanabe, D.R. Santos.
Diálise crônica no Brasil – relatório do censo brasileiro de diálise, 2011.
J Bras Nefrol., 34 (2012), pp. 272-277
[15]
National Collaborating Centre for Chronic Conditions., Anaemia management in chronic kidney disease., National clinical guideline for management in adults and, children..
Royal College of Physicians, (2006), pp. 172
[16]
KDOQI; National Kidney, Foundation., KDOQI, clinical.
practice guidelines and clinical practice recommendations for anemia in chronic kidney, disease.
Am, J., Kidney, Dis., 47 (2006), pp. s90-s93
[17]
G. Filler, K. Mylrea, J. Feber, H. Wong.
How to define anemia in children with chronic kidney disease?.
Pediatr Nephrol., 22 (2007), pp. 702-707
[18]
R. Pruthi, H. Maxwell, A. Casula, Y. Tse, M.D. Sinha, C. O’Brien, et al.
UK renal registry 14th annual report: chapter 11 clinical, haematological and biochemical parameters is patients receiving renal replacement therapy in paediatric centers in the UK in 2010: national and centre-specific analyses.
Neprhon Clin Pract., 120 (2012), pp. c219-c232
[19]
A. Jander, R. Wiercinski, I. Balasz-Chmielewska, M. Miklaszewska, K. Zachwieja, H. Borzecka, et al.
Anaemia treatment in chronically dialysed children: a multicentre nationwide observational study.
Scand J Urol Nephrol., 46 (2012), pp. 375-380
[20]
D.M. Spiegel, M. Gitlin, T. Mayne.
Factors affecting anemia management in hemodialysis patients: a single-center experience.
Hemodial Int., 12 (2008), pp. 336-341
[21]
L.B. Smith, J.J. Fadrowski, C.J. Howe, B.A. Fivush, A.M. Neu, S.L. Furth.
Secondary hyperparathyroidism and anemia in children treated by hemodialysis.
Am J Kidney Dis., 55 (2010), pp. 326-334
[22]
M.P. Kooistra, E.C. Niemantsverdriet, A. van Es, N.M. Mol-Beermann, A. Struyvenberg, J.J. Marx.
Iron absorption in erythropoietin-treated haemodialysis patients: effect of iron availability, inflammation and aluminium.
Nephrol Dial Transplant., 13 (1998), pp. 82-88
[23]
J. Silva, S. Andrade, H. Ventura, J.P. Santos, S. Colaço, C. Oliveira, et al.
Iron supplementation in haemodialysis - practical clinical guidelines.
Nephrol Dial Transplant., 13 (1998), pp. 2572-2577
[24]
G.J. Handelman.
Vitamin C deficiency in dialysis patients - are we perceiving the tip of an iceberg?.
Nephrol Dial Transplant., 22 (2007), pp. 328-331
[25]
W.H. Hörl, I.C. Macdougall, J. Rossert, R.M. Schaefer.
OPTA - therapy with iron and erythropoiesis-stimulating agents in chronic kidney disease.
Nephrol Dial Transplant., 22 (2007),

Please cite this article as: de Freitas JS, Costa PS, Costa LR, Naghettini AV. Evaluation of clinical and laboratory variables associated with anemia in pediatric patients on hemodialysis. J Pediatr (Rio J). 2015;91:87–92.

Study conducted at Post-Graduation program in Health Sciences, Universidade Federal de Goiás (UFG), Goiânia, GO, Brazil.

Copyright © 2014. Sociedade Brasileira de Pediatria
Idiomas
Jornal de Pediatria (English Edition)

Subscribe to our newsletter

Article options
Tools
en pt
Taxa de publicaçao Publication fee
Os artigos submetidos a partir de 1º de setembro de 2018, que forem aceitos para publicação no Jornal de Pediatria, estarão sujeitos a uma taxa para que tenham sua publicação garantida. O artigo aceito somente será publicado após a comprovação do pagamento da taxa de publicação. Ao submeterem o manuscrito a este jornal, os autores concordam com esses termos. A submissão dos manuscritos continua gratuita. Para mais informações, contate assessoria@jped.com.br. Articles submitted as of September 1, 2018, which are accepted for publication in the Jornal de Pediatria, will be subject to a fee to have their publication guaranteed. The accepted article will only be published after proof of the publication fee payment. By submitting the manuscript to this journal, the authors agree to these terms. Manuscript submission remains free of charge. For more information, contact assessoria@jped.com.br.
Cookies policy Política de cookies
To improve our services and products, we use "cookies" (own or third parties authorized) to show advertising related to client preferences through the analyses of navigation customer behavior. Continuing navigation will be considered as acceptance of this use. You can change the settings or obtain more information by clicking here. Utilizamos cookies próprios e de terceiros para melhorar nossos serviços e mostrar publicidade relacionada às suas preferências, analisando seus hábitos de navegação. Se continuar a navegar, consideramos que aceita o seu uso. Você pode alterar a configuração ou obter mais informações aqui.