Smart Systems Require Smarter People

We can’t deny that AI is indeed helping governments work faster, detect patterns, improve services and make better use of data, and rightly so. But introducing AI into the public sector is not only a question of what technology can do but there is another question that deserves equal attention:

What happens when it gets it wrong?

The higher the impact of a decision on people’s rights, income, access to services or opportunities, the more important it becomes to think critically about the data being used, the assumptions behind the system, the potential for bias, and the role humans should continue to play in the decision-making process.

Here’s one case from the Netherlands that shows just how serious those consequences can become.

An AI Use Case That Went Wrong: Detecting Welfare Fraud

Country: The Netherlands Period: 2004–2021

In 2004, the Dutch tax authorities began using an algorithmic system to detect potential misuse of its childcare benefits scheme. The AI system profiled claimants and classified applications according to their perceived risk of fraud.

In simple language; the problem was the system got it badly wrong.

More than 20,000 parents were wrongly tagged as fraudsters, with parents holding dual citizenship disproportionately flagged by the system.

Between 2005 and 2019, childcare allowances were wrongly stopped for large numbers of parents. Many low and middle-income families were then required to repay substantial amounts of money despite having done nothing wrong.

The consequences extended far beyond an inaccurate prediction.

Families experienced financial hardship, uncertainty and significant disruption to their lives, while many were left waiting for compensation. The scandal eventually became one of the most prominent examples of what can happen when automated decision-making is introduced into a high-impact public service without sufficient safeguards.

By 2021, the political consequences had become so significant that the Dutch government cabinet resigned.

What began as a small administrative use of technology had become a national political crisis. I don’t think AI can stand up for itself and claim, “But my intentions were clean.” (wink, wink)

This use-case and many more others show that when AI is used to make predictions about individuals or groups of people, it comes with a greater risk of causing harm. This is where critical thinking around AI becomes essential.

Not only is using AI to predict people's circumstances and behaviour potentially inaccurate, as you read from The Netherlands case study, in a worst-case scenario, it can perpetuate bias, break the law and violate human rights. The Dutch case demonstrates the danger clearly.

An algorithm was designed to identify risk, but when the assumptions, data and decision-making process behind that risk assessment produced unfair outcomes, the technology did not simply make a technical error. However, people experienced the consequences.

It’s important to clarify here that not every AI Use Case carries the same risk:

The EU AI Act, for example, takes a risk-based approach to AI. Some AI applications pose relatively limited risk, others require considerably greater scrutiny, and certain uses are considered unacceptable because of the threat they may pose to people's rights and freedoms.

1- Unacceptable-risk AI: certain applications of AI are prohibited because the potential harm is considered incompatible with fundamental rights. Some examples include particular forms of social scoring, manipulation of human behavior, exploitation of people's vulnerabilities and certain forms of predictive policing based on profiling.

Just because an AI system can technically make a prediction about a person does not mean that prediction should be used to make decisions about them.

2- High-risk AI: other applications may be permitted but require significantly stronger safeguards because an incorrect decision could seriously affect someone's safety or fundamental rights. Some examples include AI used in areas such as access to essential public services and benefits, law enforcement, migration, asylum and border management, education, employment, and critical infrastructure

For such systems, the question cannot simply be whether the model achieves good accuracy.

We also need to assess how decisions are made, what data is being used, whether particular groups are disproportionately affected, how humans can intervene and what happens when the system makes a mistake.

So What Should We Ask Before Using AI in Government?

Perhaps the most important skill we need to develop alongside AI adoption is critical thinking. You’ll probably notice this skill showing up in many of my articles, partly because it somehow keeps needing to be said. It should be instinctive, yet somehow, you still manage to treat it like an optional feature.

Before deploying an AI system, particularly one that influences people's lives, I believe we need to ask:

What problem are we actually trying to solve? Is AI genuinely the right solution, or could the problem be addressed through a simpler and safer system?

What data is the AI learning or making decisions from? Is that data accurate, representative and appropriate for the intended use?

Could the system disadvantage certain groups? Are we testing performance across different populations rather than only looking at overall accuracy?

What happens if the AI is wrong? A recommendation algorithm getting a movie suggestion wrong carries a very different consequence from an algorithm incorrectly identifying someone as a fraud risk.

Can the affected person understand or challenge the decision? There should be a route for explanation, review and appeal where decisions have meaningful consequences.

Where should human judgement remain? High-impact decisions should not automatically become fully automated simply because automation is technically possible.

Who is accountable? An organization cannot outsource responsibility to an algorithm. Someone must ultimately own the decision, the system and its consequences.

Responsible AI Is More Than Accurate AI

One of the biggest lessons I am taking from studying "AI in government" at apolitical is that technical performance is only one part of responsible implementation.

A system can be sophisticated and still be inappropriate:

• It can be statistically accurate overall and still create unfair outcomes for particular groups.
• It can improve efficiency while simultaneously introducing risks that were never properly considered.
• And it can automate an existing process while also automating the biases already embedded within that process.

Responsible AI therefore requires more than building a good model. It requires fairness, transparency, accountability, privacy, human oversight, continuous evaluation and a clear understanding of the consequences when the technology fails.

The objective should not be to slow down AI adoption in government but quite the opposite.

The objective should be to make sure that as governments become more ambitious about what AI can enable, they become equally sophisticated about where it should be used, how it should be governed and when humans still need to make the final call.

I’d like to end this article with a quick thought:

As governments move quickly to strengthen digital sovereignty, strategic technological competitiveness and national AI capabilities, it is equally important to slow down long enough to consider the risks, consequences and human impact of how these systems are used.