This article is written by Anna Thomas, co-founder and director of the Institute for the Future of Work in the UK and independent member of the CDEI Bias Review Steering Group.

At the beginning of August, young people across the UK got a rude awakening when they received their A-level grades. Due to the Covid-19 pandemic, and the cancellation of exams, Ofqual designed an algorithm which generated A-level predictions using the historical performance data from each school, a rank provided by the teachers, and prior attainment of each group. This standardisation process was applied for schools with more than five students for a subject, which meant that “small” schools escaped Ofqual’s moderation.

But when the grades were announced, nearly 40% of the students had been adjusted or “downgraded”, meaning that a high proportion of final grades were lower than the predicted grades assessed by their teachers. Moreover, Ofqual’s downgrading was not evenly distributed: disadvantaged state schools were more likely to see year by year variation in grades, and less likely to use the ‘small school’ caveat. This means that state schools were more likely to be disadvantaged by a model which paid very little attention to the individual characteristics of each student. A public outcry ensued, as teachers, parents and students all protested the unfairness of the system.

To counter the uncomfortable truth that automated standardisation will pay no attention to individual personhood, algorithms must not be outsourced to technologists: it must be co-developed

The public outcry is about more than the threatened prospects of hundreds and thousands of hopeful job entrants across the country. It also reflects frustration at the absence of meaningful human accountability and governance over automated systems which can project, en masse, past inequalities into the future.

The debate, so far, has focused on the need for a technical review of the automated process. But technological solutions are not enough, and cannot be seen in isolation. To avoid a form of “techno-chauvinism,” we need to ask wider questions about the legal, social and political infrastructures in which this situation could come about.

In other words, it’s imperative that policymakers learn how to implement the principles of fair, equitable and inclusive design and use of algorithms to avoid a repeat of this debacle - and make sure technology works to promote the public good. So what lessons should policymakers learn from this? Although no substitute for ministerial accountability, here I offer three key takeaways.

A path to better tech

First, more inclusive methods of systems design and testing are needed urgently. Transparency about technical aspects of a model is important but was not the main problem in Ofqual’s case. To counter the uncomfortable truth that automated standardisation will pay no attention to individual personhood, algorithms must not be outsourced to technologists: it must be co-developed.

Policy-makers must be prepared to think big and entertain proposals for new routes for algorithmic accountability

In Ofqual’s case, social scientists and students would have pointed out the problems of using ranking and historic data; lawyers would have pointed to the public sector equality duty, and the principles of fairness in administrative and data protection law; experts, like the UK Royal Statistical Society would have pointed to the technical limitations of the model; teachers unions would have identified the unequal outcomes for demographic groups and schools.

The Institute for the Future of Work is developing an inclusive social policy methodology and tool with world-leaders in systems and design thinking which might help support co-development of tools like the Ofqual algorithm.

Second, policy-makers across Departments must reduce silos and work much more closely to align policy and regulation. It undermines the Chancellor’s support for young people in his Plan for Jobs if job entrants are unfairly deprived of grade boundaries they need to work; or if the Department for Work and Pensions (DWP) concurrently require this unfortunate year group to prove they are looking for jobs which do not exist, in order to avoid sanctions. And it undermines the Cabinet Office’s commitment to address structural inequalities, for the DfE to proceed with the Ofqual algorithm which has not been designed with equality of treatment between demographic groups and socio-economic backgrounds in mind, or if adjustments to mitigate unequal impacts have not been made.

Closer working between departments would also have drawn attention to obvious obstacles or conflicts, such as the dual role of the Ofqual Chair, who heads the Centre for Data Ethics and Innovation.

To help align work in different spheres, the Commission on the Future of Work has proposed a cross-department Work 5.0 Strategy which would re-orient policy-making towards a central, cross-cutting policy objective.

Last, policy-makers must be prepared to think big and entertain proposals for new routes for algorithmic accountability. We should now be rigorously critiquing relevant legal and other accountability frameworks to make sure they fit for the age of the automated system – and ensure the accountability gaps exposed over the last week are filled. We can do better than require individual schools or students to object to unfair grading ex post facto.

If we can learn anything from this, it is that technology is only as good as the humans developing, applying and governing it; and the human values that underpin these activities

Technology policy should require advance assessment and consideration of collective, as well as individual, harms and impacts on equality. There is now a strong case to include consideration of socio-economic disadvantage, which in the UK is not currently protected by the Equality Act (s1 Equality Act is only in force in Scotland).

And we will need proactive, affirmative duties for both the public and private sector to monitor, evaluate and report on adverse impacts on equality to ensure that new forms of collective harm are not boldly projected into the future.

Eyes will now turn on the Center for Data Ethics and Innovation’s forthcoming Bias Review, which will be expected to rise to these new challenges.

Restoring trust

If we can learn anything from this, it is that technology is only as good as the humans developing, applying and governing it; and the human values that underpin these activities. The Ofqual algorithmic standardisation system didn’t have mysterious powers: it was a socio-technical system. Humans decided to deploy the algorithm and determine its remit and purpose: to predict the grade distribution for each school, rather than predicting the grades of individual students.

Humans designed the algorithm; humans selected the data points, variables and weighting; and humans decided how historic data, which reflects historic inequalities, should be used; and that teacher-assessed grades should not be used. And at a political level, humans decide how to regulate, govern and oversee these decisions too.

Technology is needed more than ever to support the challenges of the pandemic. It should be used to bring people and policy makers together, leverage human strengths, and improve access to work, work and wellbeing. Policymakers must now work hard to do this – and regain public trust in technology.

The Institute for the Future of Work’s consultation on equality impact assessments is here — readers are invited to complete it before 27 August. IFOW’s Equality Task Force will be reporting in October 2020. Anna Thomas is also an independent member of the CDEI Bias Review Steering Group.

(Picture credit: Unsplash)