Malha Fina de Convenios

This information was sourced from "Malha Fina de Convenios" by Observatory of Public Sector Innovation (OECD), https://oecd-opsi.org/innovations/malha-fina-de-convenios/. Licensed under Creative Commons Attribution-ShareAlike 3.0 IGO (CC BY-SA 3.0 IGO).

Innovation Overview

The "Malha Fina de Convenios" was designed to solve the stock problem of presented accounts pending analysis in the process of federal voluntary transfers among subnational entities of the Brazilian federation. The innovation is the use of a machine learning algorithm based on the characteristics of the agreements whose accounts have already been analysed to allow a fast and efficient analysis of covenant accounts sent to the federal agencies in Brazil. Between September 2008 and December 2017, more than 61,000 agreements had their accounts analysed by the grantors, providing a satisfactory amount of data for the learning of the algorithm to provide accurate results.

The application of the algorithm results in the constitution of an individual note for each agreement, varying between 0 and 1. The closer to 0 the note is, the greater the chance that the agreement will have its accounts disapproved. Alternatively, the closer to 1, the greater the chances that the agreement will have its accounts rejected. Consequently, the rejection of the accounts of an agreement entitles the grantor to take the appropriate measures to recover the damage to the Treasury. The value calculated for each agreement is compared to the “cut-off value” established by the federal manager, which also varies from 0 to 1. Thus, all agreements whose score calculated by the algorithm was above this limit would be considered “objectionable”, requiring a conventional analysis.

Thus, for the operation of the algorithm, it is enough that the agency stipulates a minimum score before which all agreements classified below it are approved. As an example, if a specific agency stipulates a score of 0.8 as its threshold, it means that 79.4% of its agreements may be subject to tacit approval, of which 4.62% would be inadvertently approved. It should be noted that the decision on the passing score by the granting body reflects the risk appetite of the federal manager who is transferring the money to the subnational entities.

The life cycle of the transfer of discretionary resources ends with its rendering of accounts and consequent analysis by the transferring body, which opines for the approval or rejection of the accounts. Accountability analysis is a lengthy process and calls for the use of resources for its realisation, in addition to trained public servants. In turn, the “Malha Fina de Convênios” system presents a quick, rational and innovative alternative for the analysis of accountability.

Consequently, the validation of the automated accountability method is fundamental to the continuity of this innovative approach. It allows a disruptive away to analyse 15,300 accounts that represents a liability of almost R$ 17 billion (approximately U$ 3,95 billion). Hence, all the efforts and the bureaucratic body of the transferring agencies installed to analyse pending accounts can be rationalised.

The greatest inherent risk in the process of the “Malha fina de convênios” system is the inadvertent classification of covenants whose accounts were rejected with a score close to zero. Indeed, the machine learning is not infallible, sometimes assigning good scores to bad covenants. The error rate increases as the number of eligible convents submitted for an automatic analyses of accounts, based on the score given by the system, raises. Thus, the heart of the problem is to determine a passing score that will allow a great number of automatic analysis of covenants and a low error of misclassification of bad convents.

The determination of this threshold score considers the cost of the number of public servants and the time that would take to analyse all the accounts pending analyses, one by one, compared to the possibility of approving all of them by simply pushing one simple button. This is the risk appetite.

As one of the ways to add quality to the process - since the predictive model seeks to reproduce a manager action in the analysis of account presentations, with lower cost and time optimization - the internal audit activity performed by Controladoria-Geral da União adopts the concept of continuous audits. Hence, continuous audit was also aggregated to the system "Malha Fina de Convenios" using the “agreement audit trails” methodology, which also contributed to the mitigation of residual risks. The audit trails refer to a comparison of databases in the search for pre-defined patterns that point to signs of improprieties or irregularities. So, besides the score assigned by the machine learning algorithm, the federal manager can also use the alerts given by the audit trails.

In short, this is a disruptive solution to the huge problem of covenants liability in Brazil.

What Makes Your Project Innovative?

The “Malha fina de convenios“ is innovative because it solved a problem of accountability of funds supplied by the Brazilian federal government for the municipalities in a disruptive way, using Artificial Inteligence. In the process of money transfer from the federal government to the subnational level there is an imbalance between the operating capacity of the granting agencies and the volume of work expended to analyse the rendering accounts of the transfers made.
This imbalance generated a liability of more than 15,000 instruments pending analysis, representing almost R$ 17 billion (U$ 3,95 billion) waiting for assurance. In practice, this solution checks the instruments between the Federal Agencies and the subnational level, uses algorithms, and provides a risk grade to measure the probability of approval or disapproval of the accounts. The methodology also combines the issuing of alerts generated in the audit trails in search of predefined patterns of indications of irregularities.

What is the current status of your innovation?

The system “Malha Fina de Convenios” was officially made available to the Federal Public Administration on 11/7/2018 with the publication in the Official Gazette of the Federal Interministerial Instruction Number 5, of November 6, 2018. The aforementioned Normative Instruction stipulated the date of 08/31/2018 as a time limit to define the inventory liability. Thus, all account payments sent to the granting agencies until 08/31/2018 are considered stock and their analysis can be supported by the “Malha fina de convênios”.

Indeed, on February 14th 2019, the Normative Instruction number 1 was published, establishing rules and parameters for the application of the computerised procedure for the rendering analysis of covenants accounts and of the on-lending contracts and records as of September 1, 2018. Hence, the “Malha fina de convenios” is acting at the stock of pending accounts also on covenants that are not in the stock.

Collaborations & Partnerships

Controladoria-Geral da União (CGU) - the union transfers department of the ministry of the economy - developed, supported, and implemented the "Malha Fina de Convenios".

Users, Stakeholders & Beneficiaries

All the Brazilian agencies that transfer money to subnational entities were the beneficiaries of this innovation. As an example these are the top 10 agencies with more covenants in stock waiting for analysis:
Ministry of tourism
Ministry of health
Sports ministry
Justice ministry
Ministry of national integration
Ministry of education
Ministry of agriculture, livestock and supply
Ministry of culture
Ministry of social development
Presidency of the Republic
Ministry of cities

Results, Outcomes & Impacts

The major result so far is that 11 agencies of the federal government published their choices of the passing score, defining their risk appetite to the automated analysis of the covenants.

Therefore, more than 4,000 occurrence of audit trails were shared with federal managers. These trails are categorised in (i) conflict of interest, (ii) non-compliance with standard, and (iii) failure in financial execution. Precisely, 3,044 covenants in the stock were flagged in trails.

Indeed, more than 2,000 covenants were approved by the machine and all the 15,300 stock were classified and prioritised by risk.

Challenges and Failures

The challenges can be summarised in two questions: (i) Would the risk classification of the transfer processes be more assertive if the machine learning algorithm were applied to each granting agency? (ii) Would the disapproval rate curve for covenants behave differently if there was segregation by the granting agency?

These questions are not responded yet, but preliminary tests showed that the process of the "Malha fina de convenios" was not compromised in these cases.

Conditions for Success

The support of the head of the Federal Comptroller Secretary and the total commitment of members of the ministry of the economy were key factors of success.

Additionally, the leadership and guidance of the Auditors of CGU were essential for the achievement. As the preliminary tests did not present good results, there was no feeling of giving up. Continuous perseverance was crucial.

Replication

The system "Malha fina de convênios" can be replicated for all the transfers that are not operationalised on "Plataforma + Brasil". In Brazil, there are several ways and systems that operate transfers from the Union for the subnational entities. The methodology used on the covenants at "Plataforma + Brasil" can be extended to others systems.

Lessons Learned

The life cycle of the transfer of discretionary resources ends with its rendering of accounts and consequent analysis by the transferring body, which opines for the approval or rejection of the accounts. Accountability analysis is a lengthy process and encourages the use of resources for its realisation, in addition to trained public servants. In turn, the “ Malha Fina de Convênios ” system presents a quick and rational alternative for the analysis of accountability, configuring itself in innovation.

The most important lesson learned is that decision making in a bureaucratic organisation must be guided by the transaction cost of the processes. Along this line, the “Malha Fina de Convênios ” system rationalises the use of the workforce in the analysis of rendering of accounts by the granting agencies through the adoption of a risk appetite threshold in which the probable agreements with rejectable accounts would be inadvertently approved.

Budget: Not disclosed