This post is written by Pia Andrews, Public Sector Reformer. With thanks to the following for their contributions, peer review, comments & support: Tim de Sousa, Bruce Haefele, Matt Beard, Marcus Wigan, Abhinav Palia, Kathy Reid, Saket Narayan, Morgan Dumitru, Alex Morrison, Geoff Mason and Aurélie Jacquet (Ethical AI Consulting).

  • The problem: Public trust and confidence in public institutions is negatively impacted by opaque, inconsistent and unaccountable systems.
  • Why it matters: Public trust in the public sector has a direct impact on policy effectiveness, as well as social, democratic and economic stability.
  • The solution: The special context of government is critical to understand and apply in the design, delivery and management of government systems. This paper explores practical ways to create trustworthy Artificial Intelligence and Automated Decision Making in the public sector, and equally applies to any and all government systems.

Earning and maintaining public trust in government systems is not a 'nice to have', but rather is critical for stability and public confidence.

As governments increasingly explore and invest in Artificial Intelligence (AI) and Automated Decision Making (ADM) systems, we need to take steps to ensure that these rapidly evolving technologies are used appropriately in the special context of a public service, along with all other systems.

Trust in government is key in the perceived legitimacy of all that the public sector administers, from services to policies and elections. That’s why earning and maintaining public trust in government systems is not a ‘nice to have’, but rather is critical for stability and public confidence. Governments must have the trust of the public to remain legitimate.

The full paper that this blog post was drawn from, including references and further detail, is available here.

Opaque use of AI and ADM is eroding trust in the public sector

Trust can be broadly defined as:

  • a) a willingness to believe that a person or entity is operating in good faith,
  • b) with integrity, and
  • c) in a way that fulfils the individual’s expectations of that person or entity.

Government systems could therefore be considered trustworthy through demonstrating **good faith **(through systemic, measurable and publicly reported commitment to human-centred and humane outcomes), by assuring **high integrity **systems (that are lawful, accurate, consistently applied and appealable), and that meets **public expectations **(by understanding and reflecting public values and needs, doing no harm, being transparent and operating within relevant legal, social, moral and jurisdictional limitations of power).

Citizens and residents are increasingly faced with a “computer says no” situation from public services, and few or no options to seek an explanation or appeal.

Unfortunately, the way in which governments have digitised or automated manual processes across the breadth of public service functions has often resulted in opaque, inscrutable systems, leading to outcomes that are not explainable or traceable back to their legal authority. For instance, many systems produce outputs without recording the correlating legislative rules, data and other factors the output was based upon, requiring manual review in the case of a challenge. This has made it difficult to understand or appeal decisions made by systems, which are in turn not auditable in real time, and therefore incapable of detecting and minimising unintentional consequences. Most public sector systems also don’t measure the human benefits or quality of life impacts (beyond basic satisfaction measures), which makes understanding or driving human outcomes impossible, let alone identifying and mitigating potential harms as they arise.

There are many issues created by opaque and inscrutable systems. Citizens and residents are increasingly faced with a “computer says no” situation from public services, and few or no options to seek an explanation or appeal. Frontline staff have limited means of determining, let alone explaining, the rationale behind decisions generated from opaque software systems. The systems, in turn, are often protected from scrutiny by commercial-in-confidence or proprietary vendor arrangements.An increasing number of systems using Artificial Intelligence (AI) and/or Automated Decision Making (ADM) are machine-learning based, using historical data to train predictive responses that perpetuate historical bias/inequities and legacy or system-based mistakes, and create additional uncertainty for the population.

The special context of government requires a different approach

These issues are anathema to good government. Governments have special context that needs to be considered in the design and delivery of government systems - that is, they are driven by public good, rather than profit, like the private sector. Further, they have greater power and, hence, greater responsibility. This context differentiates government from private sector organisations, and requires a different approach to the design and delivery of systems that ensure and assure public outcomes and are fair, accurate and lawful, to maintain public trust, access to justice, and, by extension, legitimate government.

Unlike the private sector, government organisations are expected to be compliant with Administrative Law, with testable and auditable accountability to the Parliament, people and Auditors General. Access to review of government decisions is a key component of access to justice, so government systems need to maintain the ability to record, explain and monitor automated decision making. Governments also have very specific requirements as set out in privacy and data handling legislation (such as the Australian Privacy Act 1988 (Cth)) to enable transparency of use, appropriate use and disclosure, and protection of personal information and personal privacy.

Different objectives - which require different metrics

Most things done in government are ostensibly in the pursuit of a policy objective and for public good. Measuring and monitoring the policy impact over time is critical to assuring that the policy intent is met, but monitoring and measuring the human impact is also critical to ensuring a net positive impact for society with limited unintended harms. Both types of measurements can feed into policy iteration and responsiveness to change. This is different in the private sector, which is primarily driven by a financial imperative. Finally, there is always a constitutional and/or legislative context and mandate for public institutions, which they need to deliver upon. This context is important for designing systems that are both compliant with legislation and aligned to the purpose and mandate of the specific public institution in their specific jurisdiction, in contrast with commercial systems.

People do not have the choice to not interact with government, and rarely have the choice of which governments to interact with.

With great power comes great responsibility

Finally, and most importantly, governments have high impact levers that no other sectors have. The various branches of government can investigate, penalise, censure, enforce, seize assets, institutionalise, or incarcerate. Strict controls are very important to ensure these powers are not abused and to maintain public confidence and continued legitimacy. The separation between the executive and judicial branches of government is helpful for trust by ensuring that everyone, including government institutions and officials, can be equally held to account under the Rule of Law.

While many people may share large amounts of personal data with private sector technology platforms such as Google or Facebook, they have a choice as to which services to use, and what to share, and those companies do not have the power to imprison them, or take their children away. People do not have the choice to not interact with government, and rarely have the choice of which governments to interact with. This context is extremely important because if government systems are not fair, equitable, lawful, appealable, etc. then the real impact on people’s lives can be, and has been, devastating.

Governments affect people’s lives, ideally for the better, but sometimes for the worse. As governments increasingly digitise, automate and adopt AI/ADM, government systems must be designed to live up to basic expectations of delivering public good, fairly. And if they do not, mechanisms are needed to prove that, and on which to base corrective action.

Designing for trust

So how can you build trustworthy systems in government? It is useful to clearly articulate the requirements for building and maintaining trustworthy systems in the special context of the public sector. These six questions should be put at the heart of the design of all policies and services, to help make them more trustworthy:

  1. How would you audit and monitor the decisions/actions made, their accuracy and their legal authority, in real time?
  2. How would an end user (citizen, resident, etc.) know, understand, challenge and appeal a decision/action?
  3. How would you know whether this action/process is having a fair, positive or negative impact?
  4. How would you ensure, maintain and demonstrate independent oversight and effective governance?
  5. How would you detect, respond to and implement continuous change, external or internal?
  6. How can your organisation operate in a way the public would consider ‘trustworthy’?

Every government system demands a solid answer for all of these questions, but the ultimate test is to ask people what would make a relationship with a particular agency trustworthy (which will vary according to the mandate of the agency) and implement measures accordingly, rather than assume or demand trust. Trust is situational and contingent to the context of legislation and the scope of the institution that seeks to gain and maintain that trust. Mapping the end to end ‘user journey’ or process for these questions inevitably forces us to dig into several likely features of a high trust system.

  • Explainability and decision capture: Have you recorded the events leading up to the decision, the data and legal basis upon which the decision was based, and the decision itself? Explainability of the decision/action itself is **necessary_ _**for public sector ADM systems.
  • Traceability of authority: Can you tie the process and its outcome back to legislative, regulatory, delegated, or policy authorities? This requires prescriptive legislative and regulatory rules to be digitally available, testable and traceable by machines, and clarity on what rules require human judgement with digital access to case law to draw on precedent.
  • Monitoring for accuracy: Is there a means to check the consistency, accuracy and predictability of your system outputs? Is the test suite or verification mechanism publicly available for anyone to test against? Are you monitoring for patterns of unusual trends that need to be understood or investigated?
  • Operational responsiveness: There is no point having a monitoring system if there is no operational model around it with skilled staff able to take action or escalate an issue. A trustworthy system needs to be a combination of people and technology capabilities.

2. How would an end user know, challenge, understand and appeal a decision/action?

  • Decision discoverability: How do you communicate the decision to the affected person? How can they access their record of decisions and trust the record was not changed? How can the affected person understand the decision: the rules, the evidence that they were applied to, and how that led to the decision?
  • **Ease of access to appeal: **How can the end user (citizen, resident, etc.) appeal the decision? What is that ‘user journey’ and how can you ensure a dignified experience throughout?
  • Consistency of application: Is the system applying rules and decisions consistently, with consistent outcomes that don’t unintentionally discriminate against or bias a person or group?
  • A right to explanation: The onus must be on the government entity to explain the decision/action, but this explanation is only available through traditional appeals or FOI processes, which are a barrier to explanation. It might be helpful to introduce an explicit right of access by person to reasons for decisions affecting that person_ _(as found in Section 23 of the Official Information Act in New Zealand), creating a legal obligation on departments to explain decisions to the people about which decisions are made.

3. How would you know whether this action/process is having a fair, positive or negative impact?

  • Human outcomes measurement framework: Is your system measuring the human impacts from your AI/ADM system, both directly and broadly? Can you measure the real human impact of change to your service or policy? How can you detect and mitigate unintended harm?
  • Societal based measurement framework: Is your system measuring or motivating good outcomes for society more broadly?
  • Immediate and long-term timeframes: Can you investigate the decision in real time, at an individual, system and societal level? Can you model change across whole-of-government?
  • Impact data collection: Are you collecting relevant data (from your system or from other sources) to understand the impact of your system? Is this data protected against misuse? Are you ensuring a privacy-by-design approach, to meet the letter and spirit of the Privacy Act?

4. How would you maintain independent oversight and effective governance?

  • A library of models and algorithms for AI/ADM systems: Can the public find, understand and test the building blocks of your government system? If not, can at least the relevant oversight/governance bodies do so (particularly in the case of auditing agencies and the Parliament, to oversee less transparent systems)? What skills and tools are needed to audit and monitor such systems?
  • End-to-end data/software/algorithm assurance: Can you test, audit and monitor the full software and data supply chain to ensure no interference in the resulting outputs/decisions, as well as objective validation of data provenance and quality? Do you test and monitor how different systems/rules/data interact, to avoid harmful unintended consequences?
  • Participatory governance: Is there a broad and diverse representation of the society served included in the oversight and scrutiny of the system?
  • Public reporting: What visibility does the public have on the operations, policies, services and administration? What do they need, to trust that specific public sector organisation?

5. How would you detect, respond to and implement continuous change, external or internal?

  • Detecting and responsive to change: Are you monitoring for human impacts over time, benefits and harm, and for extrinsic or unexpected changes that would trigger a policy or implementation change? Do you have an operational model that supports continuous evidence-based change and improvements to policies or services?
  • Highly skilled workforce: Public institutions need a highly skilled workforce that can effectively deliver core functions with integrity, including in this context, AI/ADM systems, monitoring/measurement and auditing. Researchers and vendors can be engaged to support capability uplift, and provide platforms/tools as is appropriate, but for public institutions to have public trust, they have to be able to deliver well the core functions they are responsible for. This means having the skills to appropriately specify vendor deliverables, and hold vendors to account for delivering them.
  • Feedback mechanisms: Is there easy to access means for anyone to provide feedback to your system, including staff and citizens? Is feedback collected throughout the process of delivery a service/decision? Are new insights being continuously created and actioned? Is it monitored, analysed, prioritised and actioned to feed continuous improvement?

Generally speaking, when users do not trust a system, they avoid contact with it or route around it.

6. How can you operate in a way the public would consider ‘trustworthy’?

The Australian Privacy Commissioner’s report on community attitudes about privacy and trust identifies that the public has some common and increasing concerns: worries about data and privacy protection (including personal location data), to know when their information is used in ADM systems and how that will affect them, access to justice, etc.

Generally speaking, when users do not trust a system, they avoid contact with it or route around it. Accordingly, a lack of trust can hobble or comprise the effectiveness of a system, and the relationship between the person/community and the organisation that runs the system. Below are some ideas to consider in designing, delivering and operating a system in a way that the public could consider trustworthy:

  • Participatory administration: public institutions should continuously engage a diverse representation of public experience, backgrounds, skills, and perspectives in the development, implementation and operations of the system.
  • Dignified experience: does the person affected have a dignified experience, where they aren’t required to overshare personal information, feel respected, are supported to succeed in their task, get a helpful service that anticipates their context and needs, etc?
  • User controls: Do citizens/residents/etc. have control over their own consents and data? Are data utilities available (like a means test Application Programmable Interface, or age verification service) to avoid unnecessary data sharing?
  • Data and decision provenance: public sector organisations need to be able to trace, track and demonstrate provenance of data, decisions, rules and outcomes.
  • Whistleblowing: Are there strong whistleblower protections, as a last resort?

Recommendations

The following recommendations, combined with a traditional human review framework/mechanism, would help ensure an approach that is auditable, appealable, testable, human-centred and considered trustworthy, creating measurably ethical and fair government systems that benefit people.

Recommendations to mandate/legislate:

  1. Establish a right of access by person to reasons of decision, in line with Section 23 of the Official Information Act in New Zealand, creating a legal obligation on departments to provide explainability to the people about which decisions are made.
  2. A systems requirement for immutably recording all decisions that impact a person, with explainability and references to legal authorities captured in real time, and a statement of reason (refer to the Australian Ombudsman’s paper on automated decision making) for auditing and appeal.
  3. Ensure all new public servants and contractors/consultants working with the public sector are educated and informed about the special context of government, to maintain operational integrity, and to ensure mechanisms imported from other sectors are not blindly applied.
  4. Publish a publicly available reference implementation of high impact existing legislation/regulations as code, with a diverse range of test cases for reuse.
  5. Establish real time monitoring of decisions, with patterns analysis & escalation mechanisms.
  6. Establish public feedback mechanisms for all government systems, including AI/ADM systems, a public “bug reporting” and feature request tool.
  7. Mandate the requirements to publish operational information relating to government ADM systems. This necessarily requires consideration about which systems, models, algorithms, data, etc should be open source for scrutiny/accountability, creating procurement/development requirements.
  8. Establish human-outcomes measurement framework (such as the New South Wales Government Human Services Outcomes Framework) to incentivise systems and shape investment/prioritisation.
  9. Monitor human outcome measures at budget, project, service and programme levels to motivate net positive human outcomes by understanding and measuring for human impact (and harm), with escalation mechanisms. Harm in this sense, is when the human outcomes measures start to trend in the wrong direction, either at an individual, demographic or community level.
  10. Establish a dedicated whole-of-government function, unconstrained by portfolio, for exploring, understanding and escalating issues that might put at risk public trust and confidence.
  11. Include a requirement in drafting guidelines to draft all new regulations/legislation as human and machine readable from the start, to deliver clearer and better authoritative (human readable) legislation/regulation, and a reference implementation (machine readable) for reuse.
  12. Establish public visibility of and participation in oversight for government AI/ADM systems.
  13. Publish government models, anonymised training data, algorithms and other relevant AI/ADM technical artefacts using relevant mechanisms, as open source for public scrutiny and testing.
  14. Establish a mandatory Government Service Standard that requires public services to measure and monitor human and policy impact alongside standard user and performance measures.
  15. Map the end to end software, data and communication supply chains for critical digital services, systems and infrastructure, and establish high veracity mechanisms with monitoring

Recommendations to **guide and support **government entities using AI/ADM:

  1. Establish consistent and standardised rigorous testing of AI/ADM systems from different perspectives, ensuring the outcomes align with the expectations, and is representational.
  2. Establish common requirements for quality management and end to end provenance for data.
  3. Consider and adopt the Assessment List for the Trustworthy use of AI framework for government use and leverage the Canadian Government Algorithmic Impact Assessment as a means of determining the level of risk (which could leverage traditional models like the ANAO risk framework), and design governance and independent oversight accordingly.
  4. Establish public participation in policy development and service design as normal practice, independent from formal communications and PR activities.
  5. Establish an ‘open by default’ culture in public service, where ‘need to know’ is by exception.
  6. Establish a culture of seeking and valuing feedback (with support in programme and resource planning) including end user feedback, and peer review from independent experts.
  7. Ensure there are easy to access whistleblower and public reporting of issues or concerns.

Conclusion

The use of AI/ML and ADM systems in the public sector could provide a lot of value to government and society more broadly, but without clear guardrails and controls, could equally perpetuate and accelerate inequity and public distrust. It is critical that public institutions carefully navigate this space and go well beyond the benchmark of minimum or principles based compliance, and towards measurable and test-driven best practice, whilst exemplifying good public service through being lawful, ethical, accountable, fair and values-based.

Good is not just measured in what we do, but in how we do it.

To be considered trustworthy, government systems must demonstrate good faith (through systemic, measurable and publicly reported commitment to human-centred and humane outcomes), must assure high integrity (that are lawful, accurate, consistently applied and appealable), and must meet public expectations (by understanding and reflecting public values and needs, doing no harm, being transparent and operating within relevant legal, social, moral and jurisdictional limitations of power). Public trust and confidence is the most critical enabler for public institutions to ensure the pursuit and delivery of genuine, accountable, just and equitable outcomes for everyone.

👋 You can create a post like this one! Share your thoughts with a community of public servants. Learn more

(Image Credit: Unsplash)