Once you’ve learned what machine learning is — and the ways that government can use it — it’s time to dive deeper into how the process actually works. Whether you’re thinking of collaborating with experts to design an intervention or just looking to understand and explore the approach, it’s vital that you get a grounding in the basics of how to design and run a machine learning project.
Apolitical spoke to Dr Crina Grosan, Senior Lecturer at the Department of Computer Science at Brunel University and specialist in artificial intelligence and machine learning. Grosan has experience teaching machine learning to policymakers, for instance, she currently runs a UK Government Digital Service Academy masterclass. She shared her guidance for running a machine learning project, broken down into five key stages:
- Defining the problem
- Preparing the data
- Exploring the data and formulating a hypothesis
- Generating your machine learning model
- Evaluating and refining your model
Grosan uses a scenario to demonstrate how to run a machine learning project in practical terms, from data collection through to generating and refining your machine learning model.
“An example I give to my students is data on primary school test results across London boroughs,” said Grosan. “I give them the results, along with other datasets on the boroughs including employment rates, income levels and geographical positioning. Then I ask if they can identify any problems and whether the data can tell them anything.”
1. Defining the problem
The problem can be defined either before or after you’ve prepared and explored the data, depending on the situation.
Problem = the question or issue you are trying to address
In some cases, you begin with a problem and then look to collect relevant data to address or solve it.
In the example of children’s test scores, you might begin with the problem that children in particular boroughs are performing poorly compared to those in other boroughs. The next step would be to collect and explore data to uncover why this is.
In other scenarios, you start with data but without a clear problem. Then, by conducting an exploratory analysis of the data, you find the problem you want to solve and then formulate a hypothesis to test.
Hypothesis = the supposition you want to test
For instance, you might already have data on children’s test results, unemployment rates, income levels and geographical positioning, along with various other indicators by borough. You can then explore the connections between these indicators in order to identify a problem.
For example, you might find that children tend to perform poorly in boroughs where unemployment rates are high. On this basis, you could formulate a hypothesis: the poor performance of the students is linked to the unemployment rates in the borough.
2. Preparing the data
Preparation can be broken down into two parts: collection and preprocessing.
Collection = the process of gathering data
Preprocessing = transforming raw data into an understandable and structured format
First, you collect the data. If you have multiple sources, integrate them by combining various datasets.
Then you begin preprocessing the data. The goal is to transform the data into an understandable format that can be used for machine learning.
This means cleaning the data because there might be a lot of noise — essentially meaningless and redundant data. You’ll need to remove outliers, repetitions and inconsistencies, and normalise the data.
For instance, it could be that one person is 120 years old while everyone else in the dataset is no older than 70. In this case, the 120-year-old may be an outlier and so could be eliminated to avoid skewing the addition you want to make.
You also need to remove inconsistencies. For instance, after matching various datasets by comparing them with one another, you might find that you have two different phone numbers for the same name. If the phone number is important for the analysis, then clean the wrong one.
There are also ways to normalise the data. It could be that for some records you have values in the order of millions. For others, you might have values between 0 and 1. This means that, when you combine those sets of values, the smaller ones won’t contribute to the outcome because they are insignificant compared to the large values. That’s why you normalise them so they’re all within the same range — in this case, by making them all between 0 and 1.
3. Exploring the data and formulating a hypothesis
The next step is to do an exploratory analysis of the data you have.
Exploratory analysis = initial investigation of data using statistics to find patterns
For instance, you might find the percentage pass rate of children by borough.
You might also want to start making visual plots to map connections between indicators in the data.
Indicator = a measurable characteristic from the data
For instance, connections between test scores and income levels, unemployment rates or geographical positioning.
This analysis will help you to understand the problem and formulate one or more hypotheses.
4. Generating your machine learning model
Now you’re ready to start building your model.
Model = A machine learning system based on mathematical relationships
The model should test your hypothesis: in this case, it will investigate the relationship between children’s test scores and unemployment rates.
Based on the data you have from other boroughs, with your machine learning model you can predict that if the unemployment is between w and x, then the students’ results will be between y and z.
In this example, the unemployment rates are the “indicator” and the test scores are the “output.”
Output = A value that the model generates based on mathematical relationships
The relationship between the indicator(s) and the output(s) can be expressed in various ways, for instance as a rule — “if the indicator is x, then the output will be y” — or as a mathematical formula. Using the relationship your model has generated, you’ll be able to evaluate the accuracy of your hypothesis.
In this case, you’ll be able to determine whether there is a connection between test results and unemployment rates and, if so, what that connection is.
Once you’ve confirmed your hypothesis, you can then use your model to generate predictions. Whenever you change any of the indicator values, your model will be able to generate new output values, based on the relationship between the two. This allows you to investigate questions such as: if you lower the unemployment rate to x, what effect will this have on the test scores?
Therefore, rather than immediately making changes in real life, you can simulate them with machine learning to make predictions. This helps to guide interventions that can be put into practice later. Perhaps, in some boroughs, measures could be taken to boost employment which could improve children's’ attainment.
5 . Evaluating and refining your model
You can input known data into your machine learning model to evaluate its accuracy.
Evaluation = Assessing the accuracy of your machine learning model
For instance, if you know the unemployment rate for a borough and the average exam results, you could input the unemployment rate to test what average exam results the model generates. Apply the machine learning model with various different values until you are satisfied with the results it gives you. This is called “training.”
Training = Testing and improving your machine learning model
Your model will approximate and never be exact for all of the data indicators you have. So use performance metrics to evaluate how close the model is to reality.
Performance metrics = Measurements for evaluating your machine learning model
Going back to the example, for some of the boroughs it may be close to 100% accuracy, but for others, it could be worse. Still, you can measure an overall performance level across all of the boroughs.
Performance levels can also help you to test the validity of your hypothesis. Sometimes we spot relationships that aren’t there. For instance, it could be that the correlation between low test scores and high unemployment rates is a coincidence or that low test scores actually cause high unemployment rather than the inverse. In cases like these, a low accuracy rate during training can invalidate your hypothesis.
However, low accuracy doesn’t necessarily mean that your hypothesis isn’t valid. It might be that the machine learning model parameters are not the right ones. Try changing them or test a few other models in case the initial one wasn’t fit for the problem at hand. However, if none of these changes leads to improvements, you might need to abandon your hypothesis and test another one instead.
And when you’re evaluating your model, don't forget to consider the type of technology you’re using to run the machine learning algorithms.
For instance, if you’re rushing to reach a decision, you may need to use a method that is not time-consuming. Or if you don’t have very high-performance computers available, you may choose one that runs on a computer with a simple specification. Luckily, there are several methods so make sure to choose the right one for you.
So how can you improve your model’s performance level? By changing some of the parameters in the model’s relationship, you can get a better approximation of reality. This is called refinement.
Refinement = Improving the accuracy of your machine learning model
Evaluation and refinement go hand in hand — keep on evaluating and refining until you are satisfied with your results. — Anna Goulden
(Picture credit: Unsplash)
Make sure to share your own thoughts with the author by leaving a comment below
Log in or sign up to continue the conversation