Piloting UNESCO's ethical impact assessment for public-sector AI

The problem

Governments are putting artificial intelligence (AI) into public services, in citizen-facing platforms, administrative processing and decision support. As they do, recognised concerns follow about whether these systems are fair, transparent, private and accountable. The questions are widely acknowledged. What many institutions lack is a structured way to work through them.

In 2021, the United Nations Educational, Scientific and Cultural Organisation (UNESCO) adopted the Recommendation on the Ethics of Artificial Intelligence, the first of its kind anywhere in the world, setting out principles on safety, fairness, privacy, transparency and accountability. Principles, though, are not self-executing. An institution still has to establish whether its particular system handles citizen data appropriately, whether its automated processes are transparent enough for the people affected by them, and who carries responsibility when an automated decision treats someone unfairly.

These are practical questions, and they need a practical method. The gap is the absence of a consistent, repeatable process that institutions with very different levels of AI expertise, operating under different rules and running very different kinds of systems, can all use to examine the ethics of their AI in a structured way.

The solution

UNESCO's answer is the Ethical Impact Assessment (EIA), a structured method an institution can use to surface and act on the ethical risks a given AI system carries, both before it goes live and once it is running. The EIA is one of the practical tools attached to the 2021 Recommendation. UNESCO produced it, and to test how it worked in practice, UNESCO and GobLab UAI piloted it across Latin America with a set of public and private institutions.

The method is organised around two parts. It opens with scoping questions that set out the fundamentals of a project, its purpose, the team behind it, and whether automating the task is the right move at all. It then works through the ethical principles set out in the Recommendation: safety and security; fairness, non-discrimination and diversity; sustainability; privacy and data protection; human oversight and determination; transparency and explainability; accountability and responsibility; and awareness and literacy. Completing it is meant to be collaborative. A team works through it together, typically in one or two working sessions of about half a day each, which brings people from different roles into a structured discussion about a system they share responsibility for.

The pilots applied the same method across different systems and settings. In Colombia, the Alcaldía Mayor de Bogotá conducted an assessment of Chatico, the city's AI chatbot for citizen services, examining how it handled citizen privacy, its transparency, and the governance surrounding it. In Peru, a group of five public bodies, brought together by the Secretariat of Government and Digital Transformation, used it across systems spanning agriculture, energy regulation, environmental certification, and education. In Paraguay, the private firm Excelsis trialled it on a generative AI chatbot it uses for company data. Using one method across systems as different as a citizen chatbot, an energy regulator's tool and a corporate assistant was the point of the exercise: to see whether a single approach could hold up across contexts.

The impact

1. The method gave institutions a structured way to examine new questions
Across the pilots, teams said the assessment prompted them to look closely at aspects of their AI work they had not previously scrutinised, and several called the experience transformative. Participants also reported that working through it initiated previously missing conversations about responsible AI within their organisations.

2. It surfaced concrete, context-specific issues in each setting
In Bogotá, the assessment raised practical questions about how citizen data was managed and whether the governance arrangements around the chatbot were sufficient. In Peru, the institutions identified the need for the method to accommodate systems as different as an agricultural support tool and an energy regulator. In Paraguay, the pilot exposed gaps between the assessment and the country's existing regulatory framework. In each case, the findings were specific enough for the institution to act on, though the published record does not detail what changes followed.

3. The pilots are reshaping the method itself
Teams recommended turning the EIA into an interactive online tool, giving it a modular structure so institutions can focus on the dimensions most relevant to them, writing it in plainer, less technical language, and adding environmental and social indicators alongside the existing ethical ones. UNESCO states that these recommendations are already being used to revise the tool. That is one of the clearer outcomes of the exercise: piloting the method changed the method.

Lessons learned

• A globally designed tool has to bend to local conditions. The pilots operated under different regulatory regimes and institutional structures, with varying levels of AI maturity. The Paraguayan pilot raised questions about aligning the assessment with national regulations, while the Peruvian institutions needed it to span very different kinds of systems. The shared lesson is that a governance tool needs enough flexibility to fit national frameworks and enough modularity for institutions to prioritise what matters to them. For any government adopting an external tool like this, the room for local adaptation is what determines whether it survives beyond the pilot.
• Accessibility decides who can actually use it. Some teams had deep technical knowledge of AI, and others had little. Plainer language and a structure that does not assume specialist expertise were identified as essential if the method is to be used by the full range of institutions that need it. For governments building ethical AI capacity, the level of technical knowledge a tool assumes determines who can use it and who is shut out.

Launch year: 2024

This case study was written with assistance from artificial intelligence.