This article is written by Martin Hackl, Chief Digital Officer of the Austrian Federal Ministry of Justice.

Around the world, we are seeing a growing movement for “Open Justice” and transparency.

Like other “open” movements, proponents of open justice argue that citizens should have the right to access court documents, and that transparency will improve the accountability and effectiveness of the judicial system.

But making these judicial verdicts freely accessible creates a challenge: how do you ensure that the privacy of citizens isn’t violated?

Currently, in most European countries, court verdicts are anonymised by hand, which obviously takes a lot of time and human resources, therefore limiting the number of documents that can be made available to the public. However, this repetitive task follows clear patterns, which means it is excellently positioned to be solved by using machine learning/deep learning algorithms. These algorithms make it possible to respond to deviations from existing patterns as well as to make context-related “decisions” (e.g. to how to deal with certain text passages), making them superior to solely rule-based approaches.

In Austria's judiciary we developed a machine learning algorithm to anonymise this information. We are still testing the algorithm to make sure it lives up to the stringent quality controls, but there are several key lessons that we can draw out for public servants in other countries and fields who are experimenting with machine learning.

In contrast to conventional software projects, the development of projects with AI requires an adapted approach since the development is focused on the training of models with high quality data rather than coding business rules. Therefore our process of developing a machine learning system can be divided into three stages; starting with collecting existing quality assured data, through to generating a machine learning model and iterating its performance. Here are some key tips to bear in mind at each stage.

1) Start by collecting your existing data and become aware of their quality

To enable an algorithm to anonymise text, it must first be trained. To do that, a corresponding amount of data is necessary.

In Austria only the Supreme Court is currently required by law to publish their verdicts. When these verdicts are published, they are manually anonymised using black markers. This means that the original text is still available in internal databases, and therefore, they provide an excellent database for training algorithms against a “gold standard” of approximately 66.000 court verdicts.

There is huge potential for artificial intelligence to improve the work we do in the public sector

Another valuable source are the verdicts of other courts that are stored digitally in internal applications. This database consists of approximately 800.000 un-annotated documents. Additionally, both datasets are complemented with registered data consisting of information on parties to the proceeding. These three databases together make up a perfect foundation to develop very accurate algorithms.

Beside court verdicts and corresponding register data, publicly available dictionaries (e.g. addresses, company registers) are integrated to complement the data set to further optimise model training.

2) Then choose a model — there is not one to rule them all

We knew from the start that we had excellent data to train our algorithm. The next step involved choosing our machine learning approach.

Fortunately, a number of freely available open source libraries for NLP (such as Spacy or Stanford-NLP) can be found on the Internet today. These also offer pre-trained models for recognising names or locations. But how do you bring this into account with the specific rules for the names to be anonymised in court verdicts?

In our approach we decided therefore to implement a combined use of different technologies and models to ensure the best possible outcomes while taking advantage of the existing open source models. So out-of-the box- as well as custom natural language processing (NLP) and machine learning (ML) algorithms are deployed together with traditional methods (such as fuzzy search or regular expression based methods) to follow a best-of-breed approach for compensating weaknesses of individual tools and leveraging their strengths.

Repetitive tasks which follow a clear pattern are bound to be made more efficient by the use of artificial intelligence and this will only speed up as the technology matures. Because anonymising court verdicts currently eat up such vast amounts of human resources, handing the job over to an algorithm would free these people to perform other complex and value-adding tasks

From an architectural perspective, this led to the development of small and targeted models rather than using a major “universal model” to ensure flexibility and quick adaptability to new requirements. This combination of different state-of-the-art approaches was selected to compensate weaknesses of individual tools and leverage their strengths and is therefore primed to deliver the best possible outcome for our use case.

3) Lastly, be sure to iterate and evaluate

Having a large, high quality dataset and an appropriate architecture are two major assets that increase the likelihood of success.

After defining and clarifying the requirements for anonymisation, the models are trained on the dataset. In short sprint cycles, a randomly chosen sample of court decisions is processed using the developed algorithms and undergo an immediate, thorough human review. Additionally overall key performance indicators for the quality of the anonymisation are calculated, such as the algorithm’s ability to recognise only those names and addresses that really need to be anonymised, to assess the overall impact of the changes.

The most important key figures are called: precision, recall and the (aggregated) F1-score. The generated feedback and figures feed directly back into the next development cycle to ensure a targeted improvement and optimisation of the models. This iterative approach enables the project team to immediately follow up any weaknesses in the models and make corrections right before the next training cycles.

As of June 2020, the court decisions are anonymised by the AI and then published in the courts’ internal collection. While 99.5% of the information that needs to be anonymised is recognised correctly (which speaks to our KPI about recall), the algorithm is still too strict with certain information and is anonymising too much (which is captured by the KPI about precision). In order to maintain the legibility of the decisions, this is the starting point for further improvements that are still required before the algorithm can go live.

But remember, artificial intelligence is not infallible

As we’ve seen in this one simple example, there is huge potential for artificial intelligence to improve the work we do in the public sector. Repetitive tasks which follow a clear pattern are bound to be made more efficient by the use of artificial intelligence and this will only speed up as the technology matures. Because anonymising court decisions currently eat up such vast amounts of human resources, handing the job over to an algorithm would free these people to perform other complex and value-adding tasks.

Even though the algorithms developed recently over the course of the last months and years have improved significantly, there is no guarantee that they are working perfectly and are always accurate. Especially since there is always the possibility that new constellations and circumstances are depicted in court decisions and that these differ significantly from previous patterns, which can lead to faults in the anonymisation results. Furthermore, the words and phrases that are used are sometimes ambiguous, and the meaning behind an expression is very context-related, potentially leading to sub-optimal outputs. Since the use of AI in the judicial area must also pay attention to ethical risks, it is important to take care of balanced training data sets that do not contain any bias.

Keep in mind, that no system or algorithm — just like human beings — is infallible

All of these aspects point to the need for quality assurance of automatically processed documents and generated anonymisation outputs. In our case, certain quality criteria have been implemented, as reviewing each processed document would be nearly as time-consuming as doing all the anonymisation manually. These criteria are based on defined parameters and determine whether a machine-based anonymisation is — with a certain probability — correct.

Depending on the variety and granularity, a categorisation into different quality levels (e.g. green, yellow, red) can be implemented. In due consideration of these quality levels, the human capacities for quality assurance can focus on documents, which have been presumed to be processed and anonymised incorrectly (the red ones).  The green ones can be published automatically without any human interaction.

In summary, artificial intelligence offers great opportunities for repetitive tasks that follow clear patterns. However, keep in mind, that no system or algorithm — just like human beings — is infallible. If you have learned to deal with such failures you can also start to outsource your repetitive activities to artificial intelligence. Here we go! — Martin Hackl

(Picture credit: Newspaper Club//Flickr)