Evaluating school and parent reports of the national student achievement testing system (SIMCE) in Chile: Access, comprehension, and use

https://doi.org/10.1016/j.evalprogplan.2008.10.004Get rights and content

Abstract

This evaluation examined school and parent reports of the national student achievement testing system (SIMCE) in Chile regarding three dimensions: access, comprehension, and use. We conducted phone surveys with a representative sample of directors (N = 375), teachers (N = 1145) and parents (N = 625), and we collected more in-depth data through interviews and focus groups in 16 of these schools. The results indicate that access to the reports is not an obstacle to use for school actors, but it is for parents. While summative ratings of the reports in terms of their clarity and utility were generally very positive, the actual recall and interpretation of even basic information (assessed through case scenarios) was incorrect for a majority of teachers and parents, and reported uses of the information were both intended and unintended in nature. We also found some statistically significant differences regarding comprehension and use among subgroups of our sample. Our findings are especially relevant given the expectations attached to the use of the reports for school improvement on the one hand, and parents’ behavior as critical consumers of education on the other hand.

Introduction

For over 25 years now Chile has been one of the pioneers in Latin America regarding the measurement of educational quality. Chile's national student achievement testing system (Sistema de Medición de la Calidad de la Educación, SIMCE), housed within the Ministry of Education, constitutes an important reference in the region because of its high technical quality, sound legal foundation, stable financial backing, census application, and public dissemination of results. In contrast, other Latin American and Caribbean countries’ standardized educational testing systems “have suffered important discontinuities and have had to start over various times,” as the Partnership for Educational Revitalization in the Americas (Programa de la Reforma Educativa en América Latina y el Caribe, PREAL) observes (see Ravela et al., 2008, Consejo Consultivo del PREAL, 2006).

In Chile in 1982 the School Achievement Evaluation Program (Programa de Evaluación del Rendimiento Escolar, PER) offered a first antecendent of a national educational testing system. This initiative tested 4th and 8th grade students and aimed at giving a prominent role to parents by informing them about the educational achievement of their child. Based on a market rationale, the assumption was that parents would exercise school choice based on school quality information, and that school quality would be a function of supply and demand (that is, student enrolment), as regulated by a simple voucher system. In 1988 we can detect the institutional origins of SIMCE, and since 1990 an educational law (Ley Orgánica Constitucional de Enseñanza, LOCE) has provided the legal foundation and operational details for SIMCE to operate on a continuing basis. For example, this law specifies that SIMCE results must be published for each region and each school in national or regional newspapers and additionally be displayed visibly by each school (Ministry of Education, 2003a; Eyzaguirre & Fontaine, 1999). Currently, SIMCE measures national curricular objectives as established by the Ministry of Education for the following four subject areas: Mathematics, Language (Spanish), Social Sciences, and Natural Sciences. Testing is censal and applied to 4th grades at the end of each school year (November), while alternating testing between 8th and 10th grades.

Similar to many achievement testing systems throughout the world, the Chilean system intends to serve a number of purposes: For policy-makers SIMCE provides a general picture of student achievement in the country, identifies the most underachieving schools for special assistance programs, evaluates the effectiveness of such programs, and serves as a basis for rewarding high-achieving schools by giving salary bonuses to its teachers. Teachers are expected to use SIMCE as a formative, diagnostic tool to improve their pedagogical practice in the classroom. For directors and administrators SIMCE should provide relevant indicators to support school-level actions and decision-making. For parents, SIMCE is intended as an accountability tool that strengthens involvement in their children's education and that importantly orients their decisions with regard to school choice. Experts have stated that these multiple purposes are difficult to achieve satisfactorily with one and the same instrument, and that in fact, the formative feedback purpose and the accountability purpose stand in contradiction to each other (see OECD, 2003, Ministry of Education, 2003a). In any case, for achieving these multiple purposes it is of vital importance to effectively communicate SIMCE results to the different audiences.

The evaluation study we report on in this article assessed the SIMCE 2005 school reports aimed at directors and teachers, and parent reports. Specifically, we evaluated the SIMCE 2005 reports that communicated 4th grade test results to schools and parents. Students had been tested at the end of the school year in November of 2005 and reports were disseminated to schools at the beginning of the new school year in April of 2006.

Each school in the country receives from the Ministry of Education an individualized school report. The 70-page SIMCE 2005 school report contained student achievement results by classroom and school, as well as benchmarking indicators meant to help each school compare its performance to demographically similar schools (i.e., similar socioeconomic status and same type of school), schools within its geographical proximity (i.e., within the same school district), the national average, and its own previous performance. The school reports were carton-bound documents offering details such as icons, photos, broad margins, tables and graphs. Each school also received a two-page parent pamphlet with very basic information about school performance. Schools were expected to make copies and distribute these pamphlets to its parents; for this reason the pamphlet was printed in black-and-white and the use of graphics was kept at a minimum. No individual student performance was reported, neither to parents, nor to schools.

Table 1 summarizes the contents of the school and parent reports that were disseminated at the beginning of 2006, and which formed the basis of the evaluation we report on in this article. For actual examples of reports, please refer to the following official website: http://www.simce.cl/index.php?id=334.

In our evaluation, we assessed the school and parent reports in terms of three main dimensions: access to the reports, clarity and comprehension of the contents of the reports, as well as utility and use of the information contained in the reports. The specific evaluation questions are summarized in Table 2. The question of effectiveness of communicating results of educational testing systems has been of interest not only in Chile but also in the entire Latin American region. As stated by Cueto (2005), “the information products do not seem to be utilized sufficiently by some of the intended audiences (for example, school teachers). It seems that in Latin America more effort has been spent on what type of information to collect, and how, than on thinking about how such information could be used for educational improvement.” The purpose of this evaluation study was to provide empirical evidence to inform the reconsideration of SIMCE distribution policies, as well as the redesign and improvement of school and parent reports in terms of their clarity and utility for the year 2007. The SIMCE Unit in the Ministry of Education commissioned and financed the evaluation study, and its professionals were our main stakeholders. However, the ideas expressed in this article are those of the authors and do not necessarily reflect the interpretations of those who commissioned the evaluation.

The goal of this article is to report on the findings of the evaluation in terms of access, clarity and use of SIMCE 2005 school and parent reports. We first describe some background information we judge to be relevant for an international audience so they can better understand the context in which this evaluation took place. The section that follows outlines the methods we applied. The article then details the evaluation findings and finally offers a discussion of these findings.

Section snippets

Broader evaluation context and relevant literature

The Chilean educational system is decentralized and consists of three types of schools: public, private subsidized, and private non-subsidized. The Ministry of Education acts as coordinator by regulating, evaluating and supervising all aspects of education, including educational policies and special programs aimed at improving the quality and equity of the system (for more details see Ministry of Education, 2003b). In 2001 there were approximately 10,800 schools working in the system, 58% of

Evaluation methods

A nationally representative sample of 495 schools was selected for this evaluation. The sampling criteria included socioeconomic status at school level, whether the school was public, private non-subsidized, or private-subsidized (i.e., type of school), average SIMCE score in the previous year, and change in SIMCE score between 2002 and 2005. When we report our findings, wherever relevant we will present comparisons based on these criteria. In the sample schools we evaluated access to SIMCE

School staff's access to SIMCE results and their assessment of school reports

Our survey findings indicate that the school reports that the Ministry of Education sends out to all schools in the country were the main source of information about SIMCE results. In addition, directors and leaders of the TPU sometimes used the Internet as an additional source of information. Almost all directors and TPU leaders reported having received the 2005 school report (99% in both cases). Teachers indicated that they had received the report to a lesser degree (83%). Because school

Discussion

The evaluation arrived at a number of important conclusions, mostly confirming existing evidence on the topic. First of all, access to school reports was not a relevant obstacle for school actors’ use of SIMCE information. However, access at the level of the directors and leaders of the TPU did not automatically translate into access for teachers, and much less so for parents. Here we observed a “double gap”: a gap from directors and TPU leaders to teachers, and a much bigger gap from the

Acknowledgements

The authors would like to thank the tireless and competent cooperation of various SIMCE professionals, especially Lorena Meckes, Elisa de Padua and Yolanda Gana. We would also like to thank two of our colleagues for their excellent contributions to this project: Maria Teresa Aravena and Trinidad Moreno. Finally, we are indebt to Jorge Manzi for his helpful comments on an earlier version of this paper.

Sandy Taut earned her Ph.D. from the Graduate School of Education at the University of California Los Angeles (UCLA), with an emphasis on program evaluation. She currently works as an educational evaluator and researcher at the Measurement Center (MIDE UC) of the Pontificia Universidad Católica de Chile.

References (20)

  • E. Bouregois et al.

    Apprentissage et formation des adultes

    (1999)
  • M. Carnoy et al.

    The limitations of teacher pay incentive programs based on inter-cohort comparisons: The case of Chile's SNED

    Education Finance and Policy

    (2007)
  • Consejo Consultivo del PREAL

    Cantidad sin calidad: Un informe del progreso educativo en Ámerica Latina

    (2006)
  • G. Elacqua et al.

    El consumidor de la educación: El actor olvidado de la libre elección de escuelas en Chile

  • G. Elacqua et al.

    School choice in Chile: Is it class or the classroom?

    Journal of Policy Analysis and Management

    (2006)
  • B. Eyzaguirre et al.

    Qué mide el SIMCE?

    Estudios Públicos

    (1999)
  • Hastings, J., Kane, T., & Staiger, D. (2005). Parental preference and school competition: Evidence from a public choice...
  • P. McEwan et al.

    The effectiveness and efficiency of private schools in Chile's voucher system

    Educational Evaluation and Policy Analysis

    (2000)
There are more references available in the full text version of this article.

Cited by (0)

Sandy Taut earned her Ph.D. from the Graduate School of Education at the University of California Los Angeles (UCLA), with an emphasis on program evaluation. She currently works as an educational evaluator and researcher at the Measurement Center (MIDE UC) of the Pontificia Universidad Católica de Chile.

Flavio Cortés is a sociologist of Catholic University of Chile and received an M.A. in Media Studies from The New School University (New York). Currently he works at the Measurement Center (MIDE UC) of the Pontificia Universidad Católica de Chile, directing projects in educational and social areas.

Christian Sebastian obtained his Ph.D. in psychology from Catholic University in Leuwen, Belgium. He is an assistant professor at the School of Psychology, Pontificia Universidad Católica de Chile.

David Preiss holds a Ph.D. in psychology from Yale University. He is an assistant professor at the School of Psychology, Pontificia Universidad Católica de Chile.

View full text