Copyright © 2026 Apolitical Group Limited
Search across all content
Contact
projectTCESP built a new analysis methodology, the School-Effect Indicator (IEE), which evaluates school effectiveness based on the analysis of factors related to both school and extra-school factors, such as student socioeconomic status.
This information was sourced from "School-Effect Indicator: An analysis of schools, using artificial intelligence, according to the equity principle" by Observatory of Public Sector Innovation (OECD), https://oecd-opsi.org/innovations/school-effect-indicator/. Licensed under Creative Commons Attribution-ShareAlike 3.0 IGO (CC BY-SA 3.0 IGO).
The São Paulo State Court of Auditors (TCESP), in the context of improving its main actions, identifies that the external control environment is being influenced by three main trends, addressed at the XXIII International Organization of Supreme Audit Institutions Congress (INCOSAI), which developed the document “The Moscow Declaration” and presented the guidelines for the triennium 2019-2022 of Supreme Audit Institutions (SAIs) of the Courts of Auditors Associated with INTOSAI (International Organization of Supreme Audit Institutions):
Within this alignment with INTOSAI guidelines, TCESP built a new analysis methodology, the School-Effect Indicator (IEE), which evaluates school effectiveness based on the analysis of factors related to both school and extra-school factors, such as student socioeconomic status. The work carried out was inspired by an article published in 2007 called School-Effect Indicator: a methodology for identifying school success based on data from the Prova Brasil, authored by teachers Amaury Patrick Gremaud, Fabiana de Felicio and Roberta Loboda Biondi, from the National Institute of Educational Studies and Research Anísio Teixeira (INEP.)
By evaluating schools according to the principle of equity, TCESP recognizes the needs of each school group and begins to work to reduce the impact of these differences. The identification of schools that provide their students with a better result than, in theory, is expected of them, brings to light a new inspection methodology, which evaluates education policies for being inclusive, of quality and equitable, aligning up to SDG 4 of the 2030 Agenda. The methodology hypothesis predicts that schools located in neighborhoods/districts whose population has greater purchasing power achieve a higher average in the result of their students in the National Basic Education Assessment System (SAEB). On the other hand, schools whose students belong to the most precarious socioeconomic strata of the population tend to present lower results in the final exam grade.
The project uses Artificial Intelligence to calculate the expected grade of students from each school and compares it with the grade observed in the SAEB exam. The difference between the observed and expected value becomes the added value of the school, that is, how much it has added to the proficiency of its students. Using the average grade of schools and socioeconomic data, we were able to estimate the portion of the grade that belongs to socioeconomic factors and, residually, the value added by schools to the performance of their students.
The School-Effect Indicator (IEE) is created from the residual of the model, that is, the difference between the actual grade of the schoolchild in the SAEB exam and the grade estimated by the model that only considers the socioeconomic factors of the students. It is an estimate of the portion of the grade that is dependent on school factors, and therefore a measure of school effectiveness. Example: A school with a score of 260 points in the SAEB exam, whose score estimated by the model is 250 points, has an IEE of 10 points. We can interpret it as follows: that school achieved 10 points more than what would be expected of schools with a similar student socioeconomic profile.
The IEE evaluation methodology allows the auditor to select the schools that stood out the most (highest IEE) to identify the good practices that possibly influenced the above-expected performance and, thus, such actions can be disseminated to other schools. On the other hand, schools with the worst IEE scores are also selected for evaluation and possible identification of the factors that determined a lower-than-expected performance. As the suggestions are included in the audit report, the public manager becomes aware of the notes and thus can work to suppress the factors causing the low scores and disseminate the good practices recognized by the TCESP assessment. The IEE brings numerous opportunities for conducting research on the learning environment, including good practices and effective processes, that explain why certain schools differ from others operating in similar conditions.
The current School-Effect Indicator analyzes state public schools. At this moment, TCESP and the University of São Paulo, through a cross-sector partnership in Artificial Intelligence, have been working together to improve the model and expand the analysis to municipal public schools, thus covering the entire universe of public schools in the state of São Paulo, whether they are linked to state or municipal managers.
The School-Effect Indicator methodology is innovative because:
After applying the methodology of the School-Effect Indicator, in the audit carried out in 2021 in schools selected as more or less effective, according to the IEE score, 13 recommendations were made for the management of the government of the state of São Paulo, in order to improve school performance based on findings and good practices identified by the TCESP audit. Based on these recommendations, issued in July of this year, TCESP monitors and controls eventual corrections and adoption of measures by government management. The IEE methodology will be extended to municipal schools, under the management of the 644 municipalities in São Paulo (except for the capital of SP, which is governed by the Court of Auditors of the Municipality of SP). To this end, TCESP has a cross-sector partnership with USP to improve the model and hierarchical analysis to include the comparison between different education networks.
The education system in the state of São Paulo has 5,351 teaching units, 5,044 of which are regular, 38 of indigenous education, 4 “quilombolas”, 16 in rural settlements, 153 penitentiary classes and 96 socio-educational service centers. This amounts to a total of 3,466,547 students served in 2021, according to data from the Department of Education. The School-Effect Indicator (IEE) was used for the first time in 2021, at TCESP, as a guiding element for the evaluation of public policies in education. The IEE was able to help the TCESP audit to select schools considered effective by the proposed methodology, identifying educational policies with greater evidence of impact on school performance. These audit findings were compiled in a report that came to the attention of state public managers in the form of 13 recommendations that can contribute to better student performance.
The audit took place throughout 2021 and health restrictions due to the pandemic prevented on site inspections. Further, the atypical operation of schools made observation tests impossible. To overcome this challenge, the solution chosen was to intensify data analysis and videoconference interviews. The investigation regarding the execution of the Education Department's policies was also impacted by the disruption of the outlined plans. Remote learning showed how large socioeconomic conditions can affect students, where not everyone has access to adequate infrastructure and these factors had to be considered in the analyses. Finally, the theme has a close relationship with academic research, requiring specialized knowledge in the area. For this reason, the categorization of the schools studied, although carried out, will not be disclosed, so as not to cause harm to school communities. The analyses, therefore, were carried out for the schools together, avoiding stigmas, inaccuracies, or arbitrariness.
The methodology encompasses new ways of acting by the control bodies, focusing on the analysis of public policies, the use of artificial intelligence and the search for more efficient results with greater impact on public management. The School-Effect Indicator is a new element that needs to be disseminated to auditors, senior leadership, and later, to municipal and state managers. The involvement of the academic community is essential, considering that the topic needs education specialists to validate the analyzes carried out. Universities can also act in the construction of algorithms. Because of this, the cross-sector partnership between TCESP and USP is fundamental for the construction of a more assertive mathematical model. The IEE demonstrates the importance of information transparency and how it is essential for the diagnosis and construction of more effective public policies, even though data from the educational area (INEP) is no longer available as an open source.
The School-Effect Indicator (IEE) can be replicated throughout Brazil, considering that school management is based on national guidelines (Law 9394/96 and Law 14113/20) and the SAEB exam is national. The theme of equity, when raised by the IEE, can be expanded to the Municipal Management Effectiveness Index (IEGM), used throughout the country and winner of an honorable mention of the Innovare Award 2018. The IEGM evaluates the effectiveness of public policies in health, education, planning, fiscal management, civil defense, environment, and governance in information technology. By involving universities in the construction of the IEE methodology, opportunities are created for the elaboration and analysis of the effectiveness of public policies with the participation of education specialists. This increases the robustness of the IEE and facilitates its dissemination to other sectors of society, such as government representatives, civil society organizations and companies.
The idea of the School-Effect Indicator project came to meet a new demand inherent to the process of modernization of control actions, which demands from the Audit Institutions evaluations of the results of public policies, including those related to the SDGs of the 2030 Agenda. The Audit Institutions have been seeking to increase their impact on improving public management. An innovation, in this case, a new methodology for analyzing public policies, according to the principle of equity, can face resistance if the institution's people are not involved, whether they are part of the top leadership or the executors of the actions. The involvement of public workers helps to mitigate resistance, create sympathy, goodwill, and engagement necessary for the project to be successful and welcomed by the institution. Finally, the communication process is fundamental and good feedback to the participants provides a suitable environment for the emergence of innovations.
The School-Effect Indicator project was selected for presentation at the 7th International Seminar on Public Administration Data Analysis, organized by the Federal Court of Auditors (TCU), the Federal Controllership General (CGU) and the National School of Public Administration (ENAP), in Brasilia, in the year 2021. The project is being perfected together with researchers and professors from the University of São Paulo (USP), because of the cross-sector partnership TCESP-USP for projects that use Artificial Intelligence.
Connect with 500,000+ public servants solving your hardest challenges.
Connect with 500,000+ public servants solving your hardest challenges.
Help public servants worldwide learn from your work, what worked, what flopped and what you'd do differently
Log in or sign up to continue the conversation