_This article is written by Melodena Stephens, Professor of Innovation & Technology Governance at Mohammed Bin Rashid School of Government. _

  • The problem: Governments have historically been the leading investors and large adopters of AI, but their oversight has not kept up with public imagination and technological advancement.
  • Why it matters: AI is a tool and how it is built, used, deployed and retired matters — especially when the general population and many decision-makers may not have the necessary literacy to understand the consequences.
  • The solution: By understanding the context of decision-making, policymakers can do better.

AI is a tool that can bring tremendous benefit or harm depending on how it is designed, managed, used or retired. The government is not only a large user of AI but a facilitator and regulator, making its role in governance very complex. Governments have a social contract to deliver public value. And this puts them at a higher level of accountability than other organisations. Delivering public value is not optional; it is a necessity. In addition, as the government is a steward of state resources and a custodian of talent and planetary resources, it is accountable to the current and successive generations.

Governments have been investing in AI for decades since World War I. Much of the fundamental and early applied AI research remains government-funded (for example, USA). Hence, the “new technologies” we see, like generative AI, have, in reality, taken decades to develop, but just days to capture the public imagination. Without oversight, these technologies are way ahead of the regulatory curve. With increasing public-private partnerships, oversight becomes more fuzzy. Self-governance in the private sector struggles with 1) profitability and shareholder pressures and 2) market demand and being first (since often AI can only be profitable at scale). For governments, as the AI race seems to heat up as a proxy cold war, investments in AI are being used to justify: 1) nation first, 2) security and 3) better resource management.

What is missing? We need more policymakers that understand AI in the context of decision-making.

1. Human-centered purpose

Remember your government and department’s purpose. AI needs to be inclusive and benefit all humans (ideally equally). Since so little is known about the black box of AI, it is tempting for policymakers to grasp the **_benefits of AI _**and not do enough due diligence on how AI comes to its conclusions and the potential harms. It is essential to know the error rate and constantly train the system when it is deployed at scale or when data or underlying technology changes. There needs to be a person accountable. In the public sector, we often measure human qualitative variables like quality of life and give them objective measures that may not capture information properly. Take the example of the USA NarxCare score (AI algorithm) assigned by a private company (Appriss, name has recently changed, post Wired article), funded in part by the US Department of Justice, which uses the state database via their prescription drug monitoring programs (PDMP). When hospitals adopted the system, patients were automatically assigned an Overdose Risk Score, indicating opioid prescription abuse. It influenced pain management and treatment. Doctors became reluctant to prescribe medicines for patients with high scores as prescribing medicines to a known opioid user affected the doctor. Only the company knows how the algorithm is calculated (it is a black box). While the intent of the AI system was good, doctors and patients did not have agency and the system could also do great harm. Such a system currently does not need regulatory oversight.

💬 Let the author know what you think about the role of governments in AI governance and accountability by leaving a comment below

Holding generative AI accountable is difficult to do as it can make billions of calculations per second. How would you verify and check this as a human? It takes up to 215 days to find a software vulnerability. To feed the generative AI monster, you need data — where will you get it and endorse its quality? Where is the energy source for these power-intensive AI machines?

AI is a tool that should ideally augment human decision-making, not always replace it.

This low diligence has led to the adoption of systems that have wrongfully terminated people (Houston Independent School District for teacher performance; Michigan state’s MiDAS), accused people of fraud or crimes (UK Post Office scandal with the Fujitsu Horizon System; DNA testing and algorithm matching for crimes), denied them benefits (Child welfare systems; visas), infringed on legal rights (copyright and generative AI, open source) or gave incorrect advice (mental health chatbots). To be human-centred, we should ask questions like: “Will the service still be inclusive?” “If an AI displaces someone’s job, how long does it take the person to get another job, and will they gain in income or have to choose a lower-paying job?” “If there is an issue, who is the person they can reach who has the agency to solve the discrepancy?” “What are the trade-offs?”

2. Transparency in human-AI decision-making

AI is a **tool **that should ideally augment human decision-making, not always replace it. Knowing when to replace a human skill is not easy. Transparency is letting the users and employees know when AI will be used, how it will be used and who is responsible for failures. Decision-making requires data, and even though many data regulations exist, data privacy remains a challenge as it is difficult to identify when a breach has happened, to contain it and to prosecute and get justice. It is problematic that regulations are inconsistent across jurisdictions, data trade deals are increasing and that data brokerage remains a lucrative business.

Laws also struggle with whom to penalise when there is an AI failure. A human is often treated more severely, say for loss of life, than a technology or its owners, which raises the question of fairness. Hence, laws may need to account for this. More training programs are required on human-AI teaming and the role of AI or humans.

For transparency, we need to know when there is a failure and also how the problem can be escalated up the chain of command. Governments are complex organisations, often with silos and little in-house expertise (thanks to outsourcing). This failure in the chain of command in complex decision-making is seen in cases like the Boeing MCAS approvals, the data collection and profiling of citizens (China), or the 2016 Cambridge Analytica-Facebook issue) or the NarxCare opioid control issue.

3. Building, purchasing and planning for AI governance.

Public procurement is 12% of GDP in OECD countries and, according to the World Bank, can be as high as 20%. Here are some key rules to keep in mind in designing better policies around public procurement for the design and deployment or subcontracting of AI systems:

  • Be aware of what is happening: Look at the AI incident database. We can build and deploy better AI systems if we know what mistakes to avoid. RAND recommends foresight to think of what could go wrong. Hack yourself, like the AI Cyber Challenge — this is a good way to test your systems (of course, the assumption is that you have in-house capabilities),
  • Invite diverse stakeholders to the table to understand the implications of adopting AI.
  • IEEE recommends auditing AI. Especially look at APIs and all suppliers. Gartner Inc. said,_ _“Many API breaches have one thing in common: the breached organisation didn’t know about their unsecured API until it was too late. The first step in API security is to discover the APIs the organisation delivers or consumes from third parties.”
  • Plan for obsolescence: Tech upgrades (software and hardware) can cost 10-40% of IT budgets and are increasing. Managing data obsolescence and training will cost more.
  • Plan and train for skills obsolescence. With more dependency on high tech, there is a challenge that the conventional skills we have gained may not be required till there is a critical AI failure, and then we may not have the capacity.
  • Avoid the too-big-to-fail dilemma. It is tempting to ignore the elephant in the room and continue pouring good money into a flawed system.
  • Deescalate the AI cold war race, which has more arsenal with less oversight than the nuclear cold war with about 70,000+ arms.

Contrary to the public perception, government can and should work with its stakeholders to manage AI governance, literacy and purpose better. Playing catch-up ends up with more regulatory costs for business and more lobbying (which disproportionately favours big tech).

Done reading? Make sure to share your own thoughts on the importance of transparency and human-centred AI decision-making by leaving a comment below ⬇️

(Image credit: Unsplash)