What you will learn
Find out what uses of AI in the public sector come with higher risks of harm.
What you need to do
Read the article (15 minutes)
Complete the reflection activity (5 minutes)
Search across all content
Find out what uses of AI in the public sector come with higher risks of harm.
Read the article (15 minutes)
Complete the reflection activity (5 minutes)
In the last section, you learned what AI is, how it works and what tasks and problems it can help with. You also got a taster of what this looks like in practice.
In this section, you’ll develop your critical thinking skills by taking stock of the risks and opportunities AI presents. We know that risks are a massive concern for public servants and governments, so rest assured: we’ll foreground responsible AI innovation all the way.
Let’s begin by testing your intuition for high-risk uses of AI.
We’ve listed some potential uses of AI in the public sector below. We’re going to use these examples to test your intuition for risk.
Take a moment to read through the list and then reflect on the questions we ask you in the ‘AI in your world’ box below:
Manage procurement processes
Optimise refuse collection
Detect welfare fraud
Review air cargo records for threats
Predict energy consumption
Respond to local residential issues
Monitor the quality of medical supplies
Predict protests and conflict
Match job seekers with jobs
Identify vulnerable children
Transcribe parliamentary meetings
Predict school dropouts
Predict criminal activity
Manage traffic flow
All uses of AI in the public sector carry risk, but some uses carry greater risks than others.
Keeping the ‘bad outcomes’ of this lesson’s title firmly in mind, what examples from the list do you think risk causing the most harm?
Consider why you made those choices: do they have anything in common?
You’ve picked out the examples you think could cause the most harm, now it’s time to see how high-risk scenarios can play out in practice.
We’ve identified two case studies of AI in the public sector. Case 1 is a real-life example where a government got it very wrong and caused significant harm to citizens. Case 2 shares the findings of a pilot study that prevented bad outcomes in a high-risk scenario.
Navigate to the expandable boxes below and see if the case studies bring to life any of the risks you thought about when reading the list above.
Use case: Detect welfare fraud
Country: The Netherlands
Year: 2004–2021
In 2004, the Dutch tax authorities started using AI to detect misuse of its childcare benefits scheme. The algorithm profiled claimants and labelled claims based on their risk of being false. The algorithm erroneously tagged more than 20,000 parents as fraudsters, disproportionately flagging claims by parents with dual citizenship.1
Between 2005 and 2019, the childcare allowance of a large number of parents was wrongly stopped.2 These low- and middle-income families were further victimised by being forced to pay back large sums of money. Many of these families are still waiting for compensation.3
In 2020, the District Court of The Hague (Rechtbank Den Haag) ruled that the algorithm violated Article 8 of the European Convention on Human Rights: the right to respect for private and family life.4
This use of AI, introduced for a seemingly small administrative function, led to the collapse of the Dutch government cabinet in 2021. In the Netherlands, it is widely known as ‘the childcare benefits scandal’ (kinderopvangtoeslagaffaire).
Watch the news coverage of this case
Bring the real-world impacts of this case study to life by watching the Euronews report from 2021: Dutch government resigns (4 mins).
To explore the impacts of this case more deeply, you can watch the NYU Center for Human Rights and Global Justice’s conversation: How Runaway Algorithms Brought Down the Dutch Government (1 hour 15 mins).
Use case: Identify vulnerable children
Country: United Kingdom
Year: 2020
In 2020, What Works for Children’s Social Care, an organisation funded by the UK Government, published the findings of a project that explored the use of AI in children's services.
In the project, researchers worked with four local authorities in England “to develop AI models to predict outcomes for individual cases. The predictions all focused on a point within the children’s journey where the social worker would be making a decision about whether to intervene in a case or not and the level of intervention required and looked ahead to see whether the case would escalate at a later point in time.”5
The potential rewards of using AI in this context are high. If successful, the outcome would result in protecting millions of vulnerable children, limiting harm by accurately identifying welfare risks at an early stage.
However, the project didn’t find evidence to support the AI models they created. On the contrary, they found that the model missed 4 out of every 5 children at risk, a result that would cause more harm, not less.6
As a result, the project developed a standardised way for local authorities running similar AI pilots to report findings. This would enable transparent communication and collaboration “to ensure that these techniques are used responsibly if they are used at all”.7
Watch a fictional court trial
The Trial - when predictive analytics in social care goes horribly wrong is a fictional account that emphasises the high-risk of harm that using AI presents in this use case (18 mins).
Discover more about the background behind this fictional trial, and the questions it asks the decision-makers, in this article by the video’s co-creator Paul Waller, Visiting Fellow, University of Bradford & Research Principal, Thorney Isle Research.
If you’re interested in reading more about the case studies we’ve used in this lesson, check out the Sources list at the bottom of the page.
As the case studies demonstrate, when AI is used to make predictions about individuals or groups of people, it comes with a greater risk of causing harm.
Not only is using AI to predict people's circumstances and behaviour potentially inaccurate — as you learned from The Netherlands case study — in a worst-case scenario, it can perpetuate bias, break the law and violate human rights.
While the European Parliament considers many AI systems to ‘pose minimal risk’, the EU AI Act proposes different rules for different risk levels:8
Unacceptable use cases are considered a threat to people. They include AI systems that classify people based on behaviour, socioeconomic status or personal characteristics.
High-risk use cases may negatively affect safety or fundamental rights and must be assessed before being implemented. They include AI systems that involve access to public services and benefits, law enforcement and border control.
You’ll learn more about mitigating AI risks in the coming lessons.
Revisit the list of potential uses of AI in the public sector above. What other examples that make predictions about people's circumstances and behaviour stand out to you now?
Don’t worry if you didn’t intuit high-risk examples the first time you saw the list. Finding where you have gaps in knowledge is an essential part of any learning process. Identifying what you don’t know will lead you towards making informed and responsible decisions about AI innovations. Sharing what you have learned with your colleagues will help those in your world to do the same.
In this lesson, you’ve learned that using AI to make predictions about individuals or groups of people can carry high and unacceptable risks of causing harm.
But, as you will learn throughout this course, AI has the potential to improve the lives of citizens and the public service. Not convinced? Head to the next lesson and we’ll show you.
AI systems can be inaccurate and biased, leading governments to target and victimise individuals and groups unfairly.
Inadequately considering the risks of using AI can have lasting impacts on citizens and government.
Risks can be identified and harmful outcomes can be avoided. The EU AI Act poses different rules for ‘unacceptable’ and ‘high-risk’ applications of AI.
“The Dutch Benefits Scandal: A Cautionary Tale for Algorithmic Enforcement – EU Law Enforcement.” Accessed August 18, 2023. https://eulawenforcement.com/?p=7941.
Rijksoverheid (Dutch Central Government). “Herstel Kinderopvangtoeslag (Restore childcare allowance).” Accessed August 18, 2023. https://www.rijksoverheid.nl/onderwerpen/kinderopvangtoeslag/maatregelen-kinderopvangtoeslag.
European Parliament. “Parliamentary Question.” Accessed August 18, 2023. https://www.europarl.europa.eu/doceo/document/O-9-2022-000028_EN.html.
The Library of Congress. “Netherlands: Court Prohibits Government’s Use of AI Software to Detect Welfare Fraud.” Accessed August 18, 2023. https://www.loc.gov/item/global-legal-monitor/2020-03-13/netherlands-court-prohibits-governments-use-of-ai-software-to-detect-welfare-fraud/.
What Works for Children’s Social Care. “Machine Learning in Children’s Services: Does It Work?,” September 10, 2020. https://whatworks-csc.org.uk/research-report/machine-learning-in-childrens-services-does-it-work/, p. 4.
What Works for Children’s Social Care. “Machine Learning in Children’s Services: Does It Work?,” September 10, 2020. https://whatworks-csc.org.uk/research-report/machine-learning-in-childrens-services-does-it-work/, p. 5.
What Works for Children’s Social Care. “Machine Learning in Children’s Services: Does It Work?,” September 10, 2020. https://whatworks-csc.org.uk/research-report/machine-learning-in-childrens-services-does-it-work/.
European Parliament. “EU AI Act: First Regulation on Artificial Intelligence,” August 6, 2023. https://www.europarl.europa.eu/news/en/headlines/society/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence.
⬅️ Previous lesson | Course outline | Next lesson ➡️
This is the logged out version of a short course in Apolitical’s Government AI Campus. To access all interactive elements and download a certificate of completion, sign up for free.