Journal Information
Vol. 90. Issue 5.
Pages 523-527 (September - October 2014)
Visits
3685
Vol. 90. Issue 5.
Pages 523-527 (September - October 2014)
Original article
Open Access
Outcome of children with severe acquired aplastic anemia treated with rabbit antithymocyte globulin and cyclosporine A
Resultado de crianças com anemia aplástica grave adquirida tratadas com globulina antitimocítica de coelho e ciclosporina A
Visits
3685
Marlene Pereira Garanitoa,
Corresponding author
marlene.garanito@hc.fm.usp.br

Corresponding author.
, Jorge David Aivazoglou Carneiroa, Vicente Odone Filhoa, Phillip Scheinbergb
a Instituto da Criança, Hospital das Clínicas, Faculdade de Medicina, Universidade de São Paulo (USP), São Paulo, SP, Brazil
b Hospital Beneficência Portuguesa, São Paulo, SP, Brazil
This item has received

Under a Creative Commons license
Article information
Abstract
Full Text
Bibliography
Download PDF
Statistics
Figures (1)
Tables (1)
Table 1. Demographic and hematological characteristics.
Abstract
Objective

To evaluate the outcome of children with severe acquired aplastic anemia treated with rabbit antithymocyte globulin and cyclosporine as first-line treatment at this institution.

Methods

Retrospective analysis of 26 pediatric patients with aplastic anemia, treated between 1996 and 2011 with rabbit antithymocyte globulin plus cyclosporine.

Results

The overall response rate at six months was 34.6% (9/26), and the cumulative incidence of relapse was 26.5% (95% confidence interval [CI]: 1.4%-66%) at 5 years. The cumulative incidence of clonal evolution after immunosuppressive therapy was 8.3% (95% CI: 0.001%-53.7%) at five years with both clonal evolutions in non -responders who acquired monosomy 7 karyotype. The overall survival at five years was 73.6% (95% CI: 49.2%-87.5%).

Conclusions

The present results confirm the poor response rate with rabbit antithymocyte globulin as first therapy in pediatrics patients, similar to what has been reported for patients of all ages. This confirmation is problematic in Brazil, given the lack of horse antithymocyte globulin in many markets outside the United States.

Keywords:
Aplastic anemia
Immunosuppressive therapy
Anti-thymocyte globulin
Pancytopenia
Relapse
Clonal evolution
Resumo
Objetivo

Avaliar o resultado de crianças com anemia aplástica grave adquirida tratadas com globulina antitimocítica de coelho e ciclosporina como tratamento inicial em nosso instituto.

Métodos

Análise retrospectiva de 26 pacientes pediátricos com anemia aplástica tratados entre 1996 e 2011 com globulina antitimocítica de coelho e ciclosporina.

Resultados

A taxa de resposta geral em seis meses foi de 34,6% (9/26), e a incidência acumulada de recorrência foi de 26,5% (intervalo de confiança [IC] de 95%,1,4%-66%) em cinco anos. A incidência acumulada de evolução clonal após a terapia imunossupressora foi de 8,3% (IC 95%, 0,001%-53,7%) em cinco anos, com ambas as evoluções clonais em pacientes sem resposta que adquiriram o cariótipo com monossomia 7. A sobrevida geral em cinco anos foi de 73,6% (IC 95%, 49,2%-87,5%).

Conclusões

Nossos resultados confirmam a baixa taxa de resposta com globulina antitimocítica de coelho como terapia inicial em pacientes pediátricos, da mesma forma como relatado para pacientes de todas as idades. Essa confirmação é problemática em nosso país devido à falta de globulina antitimocítica de cavalo em muitos mercados fora dos Estados Unidos, incluindo o Brasil.

Palavras-chave:
Anemia aplástica
Terapia imunossupressora
Globulina antitimocítica
Pancitopenia
Recorrência
Evolução clonal
Full Text
Introduction

Severe aplastic anemia (SAA) is a rare hematological disease characterized by pancytopenia and a hypocellular bone marrow. In SAA, cellular marrow elements are replaced by fat as a result of an immune-mediated destruction of stem and progenitor cells.1 Until recently, it was believed that fat replacement was a benign process; however, recent data suggest that it might be a negative regulator of hematopoiesis, contributing to marrow failure.2 Hematopoiesis can be restored in SAA following hematopoietic stem cell transplantation (HSCT) or immunosuppressive therapy (IST). In children and young adults, HSCT is preferred when a histocompatible sibling donor is available; for all other patients, IST is often employed as first therapy.3–5 The standard IST is with a combination of horse antithymocyte globulin (ATG) and cyclosporine (CsA).6 More potent lymphocytotoxic agents, such as rabbit ATG, alemtuzumab, and cyclophosphamide, have yielded disappointing results in treatment-naïve SAA due to lack of efficacy and/or increased toxicity.7–10 Response rates to horse ATG/CsA have been consistent across studies in the US, Europe, and Japan, and have varied between 60% and 75%.1,4,11–13 In general, children have a higher hematologic response rate in the 70% to 80% range, while older adults (> 40-50 years of age) have reported response rates in the 50% to 60% range.14–17

Rabbit ATG is manufactured similarly to horse ATG, but has greater lymphocytotoxic properties on a weight basis.18,19 Human T-cells derived from a thymus or T-cell line is used to sensitize an animal, whether horse or rabbit, which will produce polyclonal antibodies with a multitude of specificities to molecules expressed in human T cells. This polyclonal sera is then purified for administration in humans. Rabbit ATG has been successful in salvaging SAA patients after initial horse ATG failure and in kidney allograft, and was shown to be superior to horse ATG in head-to-head comparison.20–22 However, when given as first therapy, outcomes with rabbit ATG were inferior to horse ATG in a randomized study.8 Follow-up retrospective reports have confirmed a lower response rate in patients treated with rabbit ATG as first therapy when compared to horse ATG.23–26 However, the majority of the reports have not focused on children. This article aimed to report the results in pediatric patients who received rabbit ATG as first therapy for SAA treated at the Instituto da Criança of the Universidade de São Paulo, São Paulo, Brazil.

Patients and MethodsPatients

This study included consecutive patients with SAA who received rabbit ATG/CsA between August of 1996 and of June 2011 at the Instituto da Criança of the Universidade de São Paulo. Due to the unavailability of the horse ATG in this service and in Brazil since 2007, rabbit ATG became the standard immunosuppressor in SAA patients without an HLA-identical sibling donor. All patients met the criteria for SAA, defined as a bone marrow cellularity of less than 30% and severe pancytopenia with at least two of the following peripheral blood count criteria: (1) absolute neutrophil count (ANC) < 0,5 x 109/lL (2) absolute reticulocyte count (ARC) < 60x109/L; platelet count < 20x109/L.27 Exclusion criteria were: (1) abnormal cytogenetics, (2) bone marrow morphology consistent with myelodysplasia, and (3) diagnosis of Fanconi anemia. Bone marrow biopsy and aspirate, including cytogenetics, were performed before initiating therapy. Fanconi anemia was excluded by the absence of chromosomal changes after exposure in vitro of lymphocytes to diepoxibutane (Deb-test). Patients were hospitalized for the administration of rabbit ATG and discharged when clinically stable, usually after approximately three weeks. The local medical ethics committee approved this study, and data were obtained from written and computerized material records.

Treatment regimen

An initial intravenous test dose was performed on all patients to assess for allergic hypersensitivity. Rabbit ATG (Timoglobulina®, Genzyme, Cambridge, MA, USA) was administered at a dose of 5mg/kg/d i.v for five consecutive days. Serum sickness prophylaxis was with methylprednisolone at 2mg/kg/d was given prior to the first dose of ATG, and was continued for ten days and then tapered over the subsequent seven days. Cyclosporine was initiated on day 6 at 10mg/kg/d p.o in divided doses q12h. CsA was administered for at least six months, adjusted to blood levels (therapeutic range between 150 and 250 ng/mL).

Supportive care

Granulocyte colony stimulating factor (G-CSF) was administered at a dose of 5μg/kg subcutaneously from day +1 to day +30 to maintain neutrophils >0.5 x 109/Lto avoid infections. Itraconazole was used as prophylaxis for fungal infection at a dose of 100mg/d for at least one month after rabbit ATG. Other prophylactic antibiotics were not routinely administered.

Red blood cells were transfused in patients with symptomatic anemia or to maintain a hemoglobin level higher than 9g/dL. Platelets were transfused prophylactically in all patients with a platelet count lower than 10 x 109/L. Platelets were transfused at a higher threshold (20 x 109/L) in the presence of fever and/or clinical bleeding.

Definitions

Compete response (CR) was defined as transfusion independence associated with hemoglobin (Hb) > 110g/L, neutrophil count > 1.5 x 109/L, and platelet count > 100 x 109/L. Partial response (PR) was defined as transfusion independence, but not meeting the blood count criteria for CR. All remissions had to be confirmed by two blood counts at least four weeks apart. Response was evaluated at 180 days from treatment. Relapse was indicated by the requirement of red blood cell and/or platelet transfusion after transfusion independence lasting three or more months.

Clonal evolution was defined as the appearance of a new clonal disorder on cytogenetics or characteristic morphologic changes on bone marrow examination.

Statistical analysis

Summary statistics, including means, proportions, and their corresponding standard deviations were used to describe patients’ age, gender, and other baseline characteristics. Sample proportions and their 95% confidence intervals (CIs) were used to describe the six-month response rates for patients categorized by discrete risk factors. Long-term survival probabilities for patients with discrete and continuous baseline risks were evaluated using Kaplan-Meier estimates; patients who were lost to follow-up were counted as censored. Median follow-up was determined by the reverse censoring method. The numerical results were computed using GraphPad Prism (GraphPad, CA, USA).

Results

A total of 26 patients with SAA were treated with rabbit ATG/CsA. The median follow-up for the cohort was 4.3 years (interval: 0.02-12.2). The median age was 8.1 years (range: 1.6-15). The absolute neutrophil count was less than 0.2 x 109/L at presentation in 14 (53.8%) and less than 0.5 x 109/L in 12 (46.2%) patients. In 92.3% of cases, there was no apparent precipitating event (idiopathic SAA), and two patients (7.7%) had SAA following seronegative hepatitis. The interval between diagnosis and start of treatment was a median of 75.7 days (range: 7-300 days). Additional patient characteristics are shown in Table 1.

Table 1.

Demographic and hematological characteristics.

Patient characteristics  Number of patients (n = 26) 
Male/ female  14/12 (53.8% / 46.2%) 
Median age at diagnosis, years  8.1 (1.6-15) 
Cause of aplastic anemia
Idiopatic  24 (92.3%) 
Associated hepatitis  2 (7.7%) 
Severity of disease at diagnosis
Very severe (neutrophil count < 0.2 x 109/L)  14 (53.8%) 
Severe (neutrophil count < 0.5 x 109/L)  12 (46.2%) 
Median time between diagnosis and treatment initiation, days  75.7 (7-300) 
Response to treatment at six months
Complete response  3 (11.6%) 
Partial response  6 (23.0%) 
No response  17 (65.4%) 
Relapse  4 (15.0%) 
Clonal evolution (monosomy 7)  2 (7.6%) 
Outcome
Alive with transfusion independency  16 (61.6%) 
Alive with transfusion dependency  4 (15.4%) 
Dead  6 (23.0%) 
Median time of follow-up, years  4.3 (0.02–12.2) 

The overall response rate at six months was 34.6% (95% CI: 16%-53%). Three patients responded between six and 12 months resulting in a response rate of 46.2% (95% CI: 27%-65%) at this time period. The cumulative incidence of relapse was 26.5% (95% CI: 1.4%-66%) at five years (Fig. 1, top panel). The cumulative incidence of clonal evolution was 8.3% (95% CI: 0.001-53.7%; Fig. 1, middle panel). The two clonal evolutions were in non-responders who acquired a monosomy 7 karyotype, and both died due to infectious complications. The overall survival at five years was 73.6% (95% CI: 49.2%-87.5%; Fig. 1, bottom panel). There were four deaths from complications of SAA (septicemia) and two deaths secondary to clonal evolution.

Figure 1.

Long term outcomes following initial therapy with rabbit ATG. (Top) The cumulative incidence of relapse was 26.5% (95% CI: 0.1%-68%) at five years among responders to rabbit ATG; (Middle) The cumulative incidence of clonal evolution was 8.3% (95% CI: 0.001%-53.7%) among all patients; (Bottom) The overall survival at five years was 73.6% (95% CI: 49.2%-87.5%) among all patients. Patients were censored at the time of death or last follow-up. Median follow-up for all patients was 4.3 years. The graph was truncated at six years. Dotted lines, bottom panel represents 95% CI. Number at risk is depicted for the Kaplan-Meier survival curve only.

(0.12MB).
Discussion

In general, children have a more favorable outcome compared to older patients with aplastic anemia who are treated with IST. The response rates are higher in children and overall survival among responders is excellent.17 Although there hasn’t been a randomized study comparing IST to HSCT in pediatric patients, most patients in this age group undergo a matched related HSCT if a histocompatible sibling is available. However, IST in this age group also produces excellent results as reported by the European Group for Blood and Marrow Transplantation (EBMT), where survival outcomes for IST and HSCT as first therapy were > 90%.28 Most of the IST experience in SAA is with horse ATG; however, since 2007, this formulation is no longer available in many parts of the world including Brazil. Thus, rabbit ATG became the only formulation available outside the United States, and it is used interchangeable with horse ATG by hematologists worldwide. However, outcomes from a large prospective and several other retrospective analysis have demonstrated that rabbit ATG was less efficacious than horse ATG as first therapy in SAA. At this center, rabbit ATG is still used, and a high dose of ATG was adopted as initial therapy in children who were not transplant candidates to verify whether the response would be better when compared with the usual doses.

The authors’ experience with rabbit ATG as first line therapy in a small pediatric cohort was disappointing. Although there was no historical control, the results were far inferior to the 70% 80% response rate reported in the literature with horse ATG in children under the age of 18. The present relative sample size (with wide confidence intervals) is a limitation to our analysis; notwithstanding, the observed response rate in this pediatric cohort was lower than what is observed in this patient population following horse ATG therapy. The experience of only a 34.6% response rate at six months is very similar to the large NIH randomized trial, and is in accordance with other retrospective results.8,24,26 A small retrospective study showed a similarly low response rate in children, where only 13,3% of patients (2/15) responded to rabbit ATG.29 Some reports suggest that the response to rabbit ATG as first therapy is not too dissimilar from what observed with horse ATG; however, the response rate to rabbit ATG in these retrospective analysis tend to be lower than what has been reported in other large studies with this agent.30,31

The present results suggest that the response rate of rabbit ATG as first therapy is poor in pediatrics patients, similarly to what has been reported for patients of all ages. The confirmation of this hypothesis in this patient population is logistically complex, given the lack of horse ATG outside the United States market.

Conflicts of interest

The authors declare no conflicts of interest.

References
[1]
N.S. Young, R.T. Calado, P. Scheinberg.
Current concepts in the pathophysiology and treatment of aplastic anemia.
Blood, 108 (2006), pp. 2509-2519
[2]
O. Naveiras, V. Nardi, P.L. Wenzel, P.V. Hauschka, F. Fahey, G.Q. Daley.
Bone-marrow adipocytes as negative regulators of the haematopoietic microenvironment.
Nature, 460 (2009), pp. 259-263
[3]
E.C. Guinan.
Diagnosis and management of aplastic anemia.
Hematology Am Soc Hematol Educ Program, 2011 (2011), pp. 76-81
[4]
P. Scheinberg, N.S. Young.
How I treat acquired aplastic anemia.
Blood, 120 (2012), pp. 1185-1196
[5]
P. Kurre, F.L. Johnson, H.J. Deeg.
Diagnosis and treatment of children with aplastic anemia.
Pediatr Blood Cancer, 45 (2005), pp. 770-780
[6]
N. Frickhofen, J.P. Kaltwasser, H. Schrezenmeier, A. Raghavachar, H.G. Vogt, F. Herrmann, Treatment of aplastic anemia with antilymphocyte globulin and methylprednisolone with or without cyclosporine. The German Aplastic Anemia Study Group., et al.
N Engl J Med, 324 (1991), pp. 1297-1304
[7]
P. Scheinberg, O. Nunez, B. Weinstein, P. Scheinberg, C.O. Wu, N.S. Young.
Activity of alemtuzumab monotherapy in treatment-naive, relapsed, and refractory severe acquired aplastic anemia.
[8]
P. Scheinberg, O. Nunez, B. Weinstein, P. Scheinberg, A. Biancotto, C.O. Wu, et al.
Horse versus rabbit antithymocyte globulin in acquired aplastic anemia.
N Engl J Med, 365 (2011), pp. 430-438
[9]
P. Scheinberg, D.M. Townsley, B. Dumitriu, O. Rios, B. Weinstein, M. Battiwalla, Even “moderate” dose cyclophosphamide for severe aplastic anemia is associated with significant toxicities and does not prevent relapse and clonal evolution., et al.
ASH Annual Meeting Abstracts, 120 (2012), pp. 1259
[10]
N.S. Young, J.F. Tisdale.
High-dose cyclophosphamide for treatment of aplastic anemia.
Ann Intern Med, 137 (2002), pp. 549-550
[11]
A. Bacigalupo, B. Bruno, P. Saracco, E. Di Bona, A. Locasciulli, F. Locatelli, et al.
Antilymphocyte globulin, cyclosporine, prednisolone, and granulocyte colony-stimulating factor for severe aplastic anemia: an update of the GITMO/EBMT study on 100 patients. European Group for Blood and Marrow Transplantation (EBMT) Working Party on Severe Aplastic Anemia and the Gruppo Italiano Trapianti di Midolio Osseo (GITMO).
Blood, 95 (2000), pp. 1931-1934
[12]
S. Kojima, S. Hibi, Y. Kosaka, M. Yamamoto, M. Tsuchida, H. Mugishina, et al.
Immunosuppressive therapy using antithymocyte globulin, cyclosporine, and danazol with or without human granulocyte colony-stimulating factor in children with acquired aplastic anemia.
Blood, 96 (2000), pp. 2049-2054
[13]
S. Rosenfeld, D. Follmann, O. Nunez, N.S. Young.
Antithymocyte globulin and cyclosporine for severe aplastic anemia: association between hematologic response and long-term outcome.
JAMA, 289 (2003), pp. 1130-1135
[14]
M. Führer, U. Rampf, I. Baumann, A. Faldum, C. Niemeyer, G. Janka-Schaub, et al.
Immunosuppressive therapy for aplastic anemia in children: a more severe disease predicts better survival.
Blood, 106 (2005), pp. 2102-2104
[15]
P. Scheinberg, C.O. Wu, O. Nunez, N.S. Young.
Predicting response to immunosuppressive therapy and survival in severe aplastic anaemia.
Br J Haematol, 144 (2009), pp. 206-216
[16]
A. Tichelli, G. Socie, M. Henry-Amar, J. Marsh, J. Passweg, H. Schrezenmeier, et al.
Effectiveness of immunosuppressive therapy in older patients with aplastic anemia. European Group for Blood and Marrow Transplantation Severe Aplastic Anaemia Working Party.
Ann Intern Med, 130 (1999), pp. 193-201
[17]
P. Scheinberg, C.O. Wu, O. Nunez, N.S. Young.
Long-term outcome of pediatric patients with severe aplastic anemia treated with antithymocyte globulin and cyclosporine.
J Pediatr, 153 (2008), pp. 814-819
[18]
F.T. Thomas, C. Griesedieck, J. Thomas, M. Carver, T. Whitley, R. Warren, et al.
Differential effects of horse ATG and rabbit ATG on T cell and T cell subset levels measured by monoclonal antibodies.
Transplant Proc, 16 (1984), pp. 1561-1563
[19]
P. Scheinberg, S.H. Fischer, L. Li, O. Nunez, C.O. Wu, E.M. Sloand, et al.
Distinct EBV and CMV reactivation patterns following antibody-based immunosuppressive regimens in patients with severe aplastic anemia.
Blood, 109 (2007), pp. 3219-3224
[20]
D.C. Brennan, K. Flavin, J.A. Lowell, T.K. Howard, S. Shenoy, S. Burgers, et al.
A randomized, double-blinded comparison of Thymoglobulin versus Atgam for induction immunosuppressive therapy in adult renal transplant recipients.
Transplantation, 67 (1999), pp. 1011-1018
[21]
E. Di Bona, F. Rodeghiero, B. Bruno, A. Gabbas, P. Foa, A. Locasciulli, et al.
Rabbit antithymocyte globulin (r-ATG) plus cyclosporine and granulocyte colony stimulating factor is an effective treatment for aplastic anaemia patients unresponsive to a first course of intensive immunosuppressive therapy. Gruppo Italiano Trapianto di Midollo Osseo (GITMO).
Br J Haematol, 107 (1999), pp. 330-334
[22]
P. Scheinberg, O. Nunez, N.S. Young.
Retreatment with rabbit anti-thymocyte globulin and ciclosporin for patients with relapsed or refractory severe aplastic anaemia.
Br J Haematol, 133 (2006), pp. 622-627
[23]
P. Scheinberg.
Aplastic anemia: therapeutic updates in immunosuppression and transplantation.
Hematology Am Soc Hematol Educ Program, 2012 (2012), pp. 292-300
[24]
E.H. Atta, D.S. Dias, V.L. Marra, A.M. de Azevedo.
Comparison between horse and rabbit antithymocyte globulin as first-line treatment for patients with severe aplastic anemia: a single-center retrospective study.
Ann Hematol, 89 (2010), pp. 851-859
[25]
M.G. Afable 2nd, M. Shaik, Y. Sugimoto, P. Elson, M. Clemente, H. Makishima, et al.
Efficacy of rabbit anti-thymocyte globulin in severe aplastic anemia.
Haematologica, 96 (2011), pp. 1269-1275
[26]
J.C. Marsh, A. Bacigalupo, H. Schrezenmeier, A. Tichelli, A.M. Risitano, J.R. Passweg, et al.
Prospective study of rabbit antithymocyte globulin and cyclosporine for aplastic anemia from the EBMT Severe Aplastic Anaemia Working Party.
Blood, 119 (2012), pp. 5391-5396
[27]
S.J. Rosenfeld, J. Kimball, D. Vining, N.S. Young.
Intensive immunosuppression with antithymocyte globulin and cyclosporine as treatment for severe acquired aplastic anemia.
Blood, 85 (1995), pp. 3058-3065
[28]
C. Dufour, M. Pillon, R. Oneto, G. Franceschetto, A.P. Bacigalupo, J.R. Passweg, Outcome of aplastic anemia in children. A Survey on behalf of the SAA and PDWP of the EBMT., et al.
ASH Annual Meeting Abstracts, 120 (2012), pp. 643
[29]
D.Y. Karapinar, N. Karadas, Y. Ay, M. Akin, C. Balkan, Y. Aydinok, et al.
Rabbit antithymocyte globulin treatment in childhood acquired severe aplastic anemia.
Pediatr Hematol Oncol, 31 (2014), pp. 20-28
[30]
S.H. Shin, J.H. Yoon, S.A. Yahng, S.E. Lee, B.S. Cho, K. Eom, et al.
The efficacy of rabbit antithymocyte globulin with cyclosporine in comparison to horse antithymocyte globulin as a first-line treatment in adult patients with severe aplastic anemia: a single-center retrospective study.
Ann Hematol, 96 (2013), pp. 817-824
[31]
C. Vallejo, P. Montesinos, A. Rosell, S. Brunet, R. Córdoba, E. Pérez-Ceballos, et al.
Comparison between lymphoglobuline- and thymoglobuline-based immunosuppressive therapy as first-line treatment for patients with aplastic anemia.
ASH Annual Meeting Abstracts, 114 (2009), pp. 3194

Please cite this article as: Garanito MP, Carneiro JD, Odone Filho V, Scheinberg P. Outcome of children with severe acquired aplastic anemia treated with rabbit antithymocyte globulin and cyclosporine A. J Pediatr (Rio J). 2014;90:523–7.

Study conducted at Serviço de Oncologia e Hematologia of Instituto da Criança, Hospital das Clínicas, Faculdade de Medicina, Universidade de São Paulo, São Paulo, SP, Brazil.

Copyright © 2014. Sociedade Brasileira de Pediatria
Idiomas
Jornal de Pediatria (English Edition)
Article options
Tools
en pt
Taxa de publicaçao Publication fee
Os artigos submetidos a partir de 1º de setembro de 2018, que forem aceitos para publicação no Jornal de Pediatria, estarão sujeitos a uma taxa para que tenham sua publicação garantida. O artigo aceito somente será publicado após a comprovação do pagamento da taxa de publicação. Ao submeterem o manuscrito a este jornal, os autores concordam com esses termos. A submissão dos manuscritos continua gratuita. Para mais informações, contate assessoria@jped.com.br. Articles submitted as of September 1, 2018, which are accepted for publication in the Jornal de Pediatria, will be subject to a fee to have their publication guaranteed. The accepted article will only be published after proof of the publication fee payment. By submitting the manuscript to this journal, the authors agree to these terms. Manuscript submission remains free of charge. For more information, contact assessoria@jped.com.br.
Cookies policy Política de cookies
To improve our services and products, we use "cookies" (own or third parties authorized) to show advertising related to client preferences through the analyses of navigation customer behavior. Continuing navigation will be considered as acceptance of this use. You can change the settings or obtain more information by clicking here. Utilizamos cookies próprios e de terceiros para melhorar nossos serviços e mostrar publicidade relacionada às suas preferências, analisando seus hábitos de navegação. Se continuar a navegar, consideramos que aceita o seu uso. Você pode alterar a configuração ou obter mais informações aqui.