This article is written by Tim Hill, Data Standards and Technical Lead at the Open Data Institute

In case there was any doubt left in the matter, the publication of the UK’s Committee for Standards in Public Life report on artificial intelligence in government on 10 February 2020 ended it completely: machine learning has moved out of the world of the research lab and into government. “Artificial intelligence – and in particular, machine learning” it declares, "will transform the way public sector organisations make decisions and deliver public services."

The addition of this powerful new technique to the policymaker’s toolkit, however, is not without risk. To the well-known adage that, to a man with a hammer, everything looks like a nail should be added another saying, well-known amongst data scientists. "With four parameters, I can fit an elephant" the mathematician John von Neumann once quipped. "With five, I can make it wiggle its trunk".

Don’t worry if you don’t get the joke: von Neumann’s point is a simple one. In the complex data domains in which machine learning is typically employed, it is easy to come up with spurious — or even absurd — results.

Below, then, are four crucial issues to consider to make sure your use of machine learning isn’t just banging screws into walls with a quadratically-configurable hammer – but instead adds unique value and insight to their domains.

Have you got the right team?

Successful machine learning projects involve more than just experts in ML. At a minimum, they require:

  • Expertise in machine-learning
  • Software engineering skills
  • A deep understanding of the dataset(s) being analysed — how they were collected, how they relate to each other, how they are processed, etc.
  • Domain knowledge
  • Stakeholder representation
  • Management skills sufficient to coordinate all of the above

Just because there are six bullet-points listed above doesn’t necessarily mean there needs to be six separate individuals or teams involved; but care needs to be taken to ensure each of these roles is properly filled by people with a proven track record in the area. In particular, it’s tempting to assume that the ML and software engineering roles can be carried out by the same people, simply because both involve computers. But in fact the two skillsets are distinct — and expecting an analytics specialist to undertake software engineering may quickly lead to unmaintainable code or worse. Equally, general domain knowledge and understanding of a specific data collection and its provenance should not be conflated. It could be the case that the same person or team possesses both — but this cannot simply be assumed.

With regard to stakeholder representation, remember that the public at large should be considered a "stakeholder" in public sector projects. Questions of bias and the ethical use of ML will need to be examined carefully; and of course the Nolan Principles apply throughout.

Can ML answer the questions most relevant to your project or domain with the data to hand?

This can be surprisingly difficult to assess, which is why it’s so important to have the right team in place before you start examining your domain and trawling through your data: very often, determining what role machine learning can play will require expert assessment both of the problem that needs to be solved and of the data available.

Both of these questions, furthermore, might involve prolonged investigation: data-cleaning can often take between 60% and 90% of total project time, and it pays to see how much is required or possible early on. Public sector projects, furthermore, will often be tightly regulated, with data scattered over several sites for security reasons and sometimes-complex data protection and anonymisation rules needing to be met.

Care needs to be taken to avoid using machine learning for the sake of it

In cases where necessary data is absent or of poor quality, it can sometimes pay to think creatively and see what aspects of the problem machine learning may still be useful in solving. At this point, however, care needs to be taken to avoid using ML for the sake of it: without a well-defined question, attempts simply to run common machine-learning algorithms over existing datasets will almost certainly yield spurious and/or meaningless results.

One useful sanity-test to check the potential utility of ML is whether or not you have a good baseline to compare its performance with. What baseline should be used will depend on the domain. Scientific and medical domains will often have well-established statistical tools for predictive tasks, the accuracy of which can serve as a useful comparator for ML results; at other times a more data-analytical approach (e.g., assigning the majority class in classification tasks) will be appropriate. If consultation with both domain and data analysis experts fails to yield a useful baseline heuristic, you probably need to do more data and domain exploration before re-engaging with ML. On the other hand, if you do have a baseline, but it’s low, chances are good that ML can significantly improve on it.

Can you be agile?

The word "agile" has been so overused for a decade now that it often seems meaningless: even the most sclerotic organisations claim to be "agile" and to use "agile methodologies" now.

But machine learning really demands these. Particularly in the early stages, machine learning involves frequent iterations and team consultations to identify meaningful features, refine parameters, and develop infrastructure. Early results may seem absurd and complex models might well need to be junked in the early phases. Furthermore, one implication of the need for expert assessment of the problem in its early stages is that it may turn out machine learning isn’t the right approach in the first place. As Jurgen Mitsch, Principal Analyst at the Nottingham University Hospital’s NHS Trust puts it, it’s vital that machine learning efforts ‘start small and fail fast’ in order to find and refine the right approach.

Machine learning is a profoundly collaborative enterprise

One implication of this is that machine-learning approaches will need upper-management buy-in over the medium- to long-term if they are to stand much chance of succeeding. If quick wins are needed to secure further funding, then ML will almost certainly not be viable. As machine-learning technology matures and becomes more embedded in public-sector culture, it may be the case that iterations can be sped up and results be assured. But we are not there now, and foresight and long-term support will be required if this vision is to be achieved.

Are you developing a shared vocabulary?

As noted above, machine learning is a profoundly collaborative enterprise, which furthermore involves people deeply specialised in their respective fields.

This can at times make communication difficult – and at precisely the times when it’s most important to be clear. Data analytics has a vast and sometimes forbidding technical vocabulary, which can overlap with but is not identical to that used by data specialists. This, in turn, has some – but not many – points of contact with software engineering. And of course, domain experts will often have their own jargon, which can often be impenetrable to outsiders. A successful project will need to overcome these barriers if it’s going to yield insight.

The complexity of machine learning domains almost guarantees that everyone in the room will have "stupid questions" for each other, and ensuring these can be aired early and often is vital for success

On an administrative level, this means finding tools that allow every member of the team to communicate and share relevant information asynchronously. Many of the tools and platforms used for agile software development methodologies — e.g. Kanban boards, Jira, Azure DevOps, GitHub, etc. – are suitable for this, but one size will not fit all. Take time to experiment and play with a variety of technologies to find the mix that’s right for you.

Not everything, however, can or should be done online. Particularly in the early stages, it’s important to hold round-table plenary meetings fairly frequently, in order that the whole team can establish its shared vocabulary as quickly as possible. Within these meetings, care will need to be taken to allow plenty of time for discussion, clarifying questions, and paraphrasing – and within an environment that is non-judgemental and open. The complexity of machine learning domains almost guarantees that everyone in the room will have "stupid questions" for each other, and ensuring these can be aired early and often is vital for success.

The point of the stupid questions, furthermore, extends out beyond the team, into the "real world" where the effects of your project need to be felt. One common criticism of machine learning in healthcare is that its results are communicated in such a technical and confusing way they become useless to clinicians.

If you can’t create a plain-English summary of the decisions that have been arrived at and the techniques that are being used each day, it’s a good indication your team needs to work more on creating a shared vocabulary – both for improved understanding and for real-world impact. — Tim Hill

(Picture credit: Unsplash)