This post is written by Paul Waller, Visiting Fellow, University of Bradford & Research Principal, Thorney Isle Research.

The problem: Research has shown that the use of algorithmic methods to predict the circumstances or behaviour of people has many problems and can lead to bad outcomes.

Why it matters: Such methods, sometimes labelled artificial intelligence, machine learning algorithms, automated decision-making, predictive analytics, data analytics, or similar, have been promoted to, and sometimes tried out by government agencies. They have been used in the fields of social services, criminal justice, policing, benefits fraud, and others.

The Solution: Public sector bodies must follow the rules that exist and take great care. This article is fictional, but the story is made up around findings reported by researchers to act as a warning about what could happen. See also an extended video of the story.

The likelihood is that institutional pressure and aversion to personal risk by front-line workers would, over time, lead to the system prediction being acted on without any further consideration.

Background

Between 2017 and 2019, several local authorities in England piloted the use of predictive analytics in social care, for adults and children. Many of the pilots were stopped before being put into live operation. Academics have studied some and found numerous issues and a lack of detailed information on results; others have flagged up many risks.

One of the issues raised was how social workers would deal with the output of a predictive system conflicting with their professional judgement. The likelihood is that institutional pressure and aversion to personal risk by front-line workers would, over time, lead to the system prediction being acted on without any further consideration.

This problem would almost inevitably lead to both unnecessary interventions that disrupt the lives of families and waste resources, and no interventions in critical cases where harm later happens. The latter situation could be regarded as negligence: if the risk were known, and someone died as a result, it might be considered Gross Negligence Manslaughter.

The story that follows is fictional. There is no suggestion that such an event happened or that any local authority officials behaved in such a manner. However, had these predictive analytics systems been adopted widely, it is quite possible that such a scenario would have played out. So this story is written to emphasise the dangers of such systems in social care, and by extension in any decision making in public services.

On trial: the decision-makers, not the algorithm

The Director of Children’s Services and the Chief Technology Officer (CTO) of a local authority are on trial for Gross Negligence Manslaughter following the death in 2018 of Child D as a result of child abuse in a family on the radar of the authority’s Social Care team. The court case was begun by the prosecution but then suspended due to the Covid-19 pandemic. In 2017, the authority had purchased and put into operation a system using predictive analytics to identify families where there was a risk of serious problems occurring that put children at risk. The system had not flagged Child D’s family to the Social Care team.

In her defence, the Director of Children’s Services said that once it was in place, it became apparent that, politically and personally, it would be disastrous if a prediction from such a system was ignored and something bad happened. So she gave instructions for the Social Care team to focus on the cases flagged by it above all others. This meant all their effort was expended on them alone. She believed it was likely that, without the system, her team would have intervened with Child D’s family and thus the death would probably have been prevented.

The CTO is now called to the stand.

The prosecution questions the CTO

Prosecuting Counsel: Why was this predictive analytics system introduced? What prompted its adoption?

Defendant: We brought it in because of the government’s Troubled Families programme recommendation and funding, widespread messaging about the ability of such systems, the need to make savings, pressures to do better with less in family social care, and it ticked the digital transformation box.

PC: How did you select the system?

D: We saw what other authorities were using. We looked at material published in local government and elsewhere, and case studies by the consultants and vendors. Then we conducted a procurement process – lightweight as this was a pilot.

PC: It is known that such systems are very dependent on the quality of data on which they are trained and that are provided as input. Did you seek professional advice from data analytics experts on the data, and the risks from potential bias within it?

D: It’s not my field of expertise and we don’t employ anyone with the qualifications to analyse it. The consultants gave us assurances that the system had been widely used and trained on data similar to ours.

PC: What impact assessments were carried out?

D: Only on data protection. We put it to our GDPR (General Data Protection Regulation) team and its lawyers. On their advice a privacy statement was put on the web site to explain the GDPR basis for the data processing.

PC: What consultation took place with the families that would be covered by the system?

D: None. We were advised that if they knew they might start gaming the system, or try to opt out, which would defeat the point of it.

PC: Would you therefore agree that those subject to this prediction algorithm had no knowledge of, no understanding of, and gave no consent to its use?

D: Yes, I have to accept that.

PC: If it was a pilot, how would you tell if the system was working well? Was there an evaluation set up to spot any positive or negative impacts?

D: No, we had no baseline against which to conduct a comparison. We would get feedback from the Social Care team on effects on its and the community.

PC: What reasoning was available to the Social Care team from the system about how it reached its conclusions? Were its workings transparent?

D: The output gave a profile of each family and a short reason why the case was flagged, but not detail on the analytic process. The vendors insisted that their methodology was commercially confidential. I’m not sure we’d have understood it anyway.

PC: We heard that the team’s performance was being judged on how many of the “high risk” cases put out by the system each month were followed up. Were you aware of how many of those seemed, when investigated by the Social Care team, to not be a problem but since they had been flagged by the system, they had to continue to put resources into them?

D: I don’t know the numbers but I did hear them grumbling about wasting time visiting families that seemed fine.

PC: We heard that the team was instinctively concerned about Child D’s family, but the system never identified it as a high risk. Were you giving them a system without understanding the chance of it making a wrong prediction one way or another?

D: I guess so, but these things are advocated by lots of people who I presume know more than me.

PC: Do you agree that the lack of explanation from the system about how it reached its conclusions made it much harder for the Social Care team to apply their professional judgement with confidence, rather than having to accept the outputs blind?

D: I suppose so.

PC: To sum up, I submit to the court that the imposition of a predictive analytical system of this nature, with no understanding of its effectiveness, working, or impacts, allied with procedural changes based on the wrong assumption of 100% accuracy, prevented the Social Care team from exercising its professional judgement and intervening with Child D’s family, thus tragically leading to the death of Child D. We can understand how incentives from outside, widespread hype, overstated claims by vendors and consultants, and financial pressures led to the authority considering the use of predictive analytics. However, we have seen how a chain of events, actively led and strongly advocated by the Chief Technology Officer, displayed a catastrophic level of negligence that fell far below any interpretation of the terms “professional” or “ethical”. I submit that this level of negligence, that contributed with significant probability to the death of Child D, is sufficient to be gross and thus criminal and a conviction for Gross Negligent Manslaughter is fully justified.

PC: Your Honour, that is all from the prosecution.

Judge: Court is adjourned.

What is your verdict — guilty or not guilty?

👋 You can create a post like this one! Share your thoughts with a community of public servants. Learn more

(Image Credit: Unsplash)