Protesters are taking to the streets of cities around the world to lament the untimely loss of black lives in the United States, but their cries of anguish are extending further than that.

More systemic racism and biases are being uncovered, and one of the key targets of campaigners and protesters’ ire is data collection and processing. AI has become a key part of the way governments work across the world, driving efficiencies at a time that budgets are being slashed. But the more widely it’s being used, the more its blind spots are being uncovered.

“It’s intended to be objective, but these things have caused discrimination by not taking into account the needs, likes and experiences of women and minorities,” said Deniz Erden, research fellow at HIIG Berlin, who studies algorithmic discrimination. That’s because the code is designed to replicate our society almost perfectly – and that includes all its biases and ugly bits.

“The workforce is predominately able-bodied white men,” she added. Any data-driven systems are therefore set up to perpetuate that to the best of its ability.

Dark-skinned women are misidentified by facial recognition systems one in every three times

That’s due to the datasets on which these systems are built not being representative of the full gamut of society. “While women and ethnic minorities and other parts of society, like disabled and old people are left out of the datasets,” Erden said. As a result, the systems don’t account for their needs and can misfire or inaccurately report on them.

One of the key lightning rods for the Black Lives Matter protests taking place worldwide is the perpetual surveillance of black and ethnic minority people by facial recognition software that misidentifies them. Dark-skinned women are misidentified by facial recognition systems one in every three times, according to research by the Massachusetts Institute of Technology. That can impact their ability to access government services and can misidentify them in crimes.

“Black people are twice as likely to die in police custody, and they’re using software that’s probably incorrect to take those people protesting against police custody into police custody,” said Charlene Prempeh of A Vibe Called Tech, a lobby group focused on improving representation in technology. For that reason, IBM and Amazon have both announced they’d pause supplying facial recognition systems to law enforcement until they can resolve the issues.

But the answer to the problem isn’t to include more diverse faces in the datasets – that perpetuates more surveillance on a population already well used to it and increases the problems many people are taking to the streets for.

Root and branch reform is needed, including clearer codes around the ethics of using data. Governments have rushed into adopting technological solutions at the expense of due care. “If there isn’t a solid governance framework, there is no limit on how [the data captured] could be used,” said Lofred Madzou, artificial intelligence lead at the World Economic Forum. He believes protesters “need to actively advocate for the enforced regulation of surveillance technologies.”

The answer isn’t to rush back into the old ways of working, but instead to think deeper about the impacts

Cities need to use the coronavirus crisis as an opportunity to reflect on their data use practices. A number of American cities, including Seattle, San Francisco and Oakland, have strict policies in place about how and when they can use data – and have taken steps to stop the implicit biases contained within them. A recent police reform bill would ban using facial recognition on body cameras worn by American police.

As well as following the lead of Seattle, San Francisco and Oakland, they could take inspiration from the European Union, whose rules around the use of artificial intelligence promote an “appropriate ethical and legal framework” around the use of data and AI.

Among its core principles is that “AI systems should integrate safety and security-by-design mechanisms to ensure that they are verifiably safe at every step, taking at heart the physical and mental safety of all concerned”.

When coming out of the coronavirus, and resetting local economies and governance structures, the answer isn’t to rush back into the old ways of working, but instead to think deeper about the impacts – both intentional and unintentional – that big tech changes can have. - Chris Stokel-Walker

(Photo credit: Tony Zhen/Unsplash)

Make sure to share your own thoughts with the author by leaving a comment below