Britain’s pensions regulator has developed a machine learning model that can quickly identify billion dollar schemes at risk of collapse. One such scheme for the British company BHS collapsed in 2016, cutting pension payouts for more than 13,000 people. Currently, trained experts require three years to assess some 6,000 schemes on their books, which cover $1.9 trillion and 11 million people. The new system is capable of scanning this pool for high-risk cases every day.
Results & Impact
A machine learning model could allow The Pensions Regulator (TPR) to assess 6,000 pensions schemes for risk every day, rather than once every three years. It can correctly classify 65-70% of schemes, and 85% of the remaining 30% to within a single grade of risk. Around 13% of members are concentrated in schemes classified as weak, with a further 19% in schemes tending to weakness. The total liability of these schemes is $1.9 trillion, with approximately $532 billion of this held by funds which are either weak or which tend towards weakness. If schemes fail, their members are placed onto a lifeboat scheme which pays out significantly lower benefits. Over 13,000 members of the BHS scheme were hit with a 10% cut in their pensions when it collapsed in 2016.
The Pensions Regulator (TPR), the Government Digital Service
An analyst has developed a machine learning model to act as a horizon scanner for risky pensions. Currently, trained experts assess each pension manually once every three years. The process is slow and expensive, meaning that assessments quickly fall out of date. The new model, trained on previous case data going back to 2005, uses a machine learning algorithm to track the financial and qualitative data of the schemes and rates them for risk based on previous patterns. Automation hadn’t been used previously by TPR because the schemes and the evaluation process were believed too complex to turn into a model. The model is currently being used monthly to spot weak funds before they fail, and which could otherwise be missed inside the three-year timetable. In the future, analysts hope to use the model to classify extreme cases automatically, leaving experts free to deal with more complex cases.
General public, the elderly
Cost & Value
The project was completed by analyst Sam Blundy, who created the model as part of his day-to-day work at TPR. He worked at the UK Digital Service's accelerator training programme for a day a week to create the model, which was funded by the digital service itself. The project hasn’t required any extra funding.
Creating the model has been arduous because of the nature of TPR’s data. The types of schemes they regulate are many and varied. It has proved difficult to turn qualitative data into quantitative, and to train the algorithm to cope with missing values in the historical data. Blundy has also had to complete the work alongside his existing duties as a pensions regulator. The pilot is currently in a trial phase and is up for review in November. Encouraging use of the tool alongside existing processes is likely to take time, as is building trust in its efficacy.
TPR is using predictive analytics elsewhere to identify which employers are unlikely to submit pension information for review. Around 40,000 businesses are required to submit information showing that they are enrolling staff in pension plans. Through using their historical data, TPR was able to create a machine learning model to predict future behaviour: whether the firm will make its return on time, be late, or not make a return at all.
The UK’s pensions regulator is using machine learning to help prevent billion dollar pension schemes from collapsing.
In the wake of the collapse of the British company BHS and its pension scheme in 2016, which left thousands of members with reduced pensions, a parliamentary committee recommended more frequent valuations for stability. Now an analyst at The Pensions Regulator (TPR) has developed a tool to help solve the problem.
Eleven million people in the UK are covered by 6,000 ‘defined benefit’ pension schemes worth a combined $1.9 trillion. Each of these currently has to be assessed by expert professionals every three years. Defined benefit (DB) schemes reward staff based on how much they earn and how long they work, regardless of the performance of the investments in the fund. If schemes fail, their members are placed onto a lifeboat scheme which pays out significantly lower benefits.
BHS ran a DB scheme until its collapse in 2016 – it was assessed in 2009 and 2012, but the subsequent valuations took too long, allowing the fund to deteriorate while they took place. Over 20,000 people were hit with a 10% cut to their pensions. The parliamentary committee reporting on the affair described the current valuation process as “clunky and can be concentrated at stages when a scheme is in severe stress or has already collapsed.” It recommended more frequent valuations to reflect risk. The new tool proposes to do just this.
“There’s not a one-size-fits-all approach to this. It’s generally conducted by expert professionals, which makes it quite a high cost and limited resource,” said Sam Blundy, the analyst at TPR who developed the model. “Two thousand every year is a significant number of assessments to be making with quite a limited amount of expert resources. As a result, the assessment that we have is generally quite quickly out-of-date.”
Blundy has developed a new system which is able to automate the work done by these professionals and rate the pension schemes for risk instantly. Currently, the experts have to assess each employer’s pension scheme individually for its health based on each employer’s financial information. They then rate each employer’s scheme from one to four.
Approximately 28% of the total liability is held by funds assessed as being either weak or which tend towards weakness. Around 13% of the total members are concentrated in schemes classified as weak, with a further 19% in schemes tending to weakness. Should a defined benefit scheme fail, the government can guarantee many of the savings as a last resort. TPR acts as a backstop to ensure sponsoring employers honour their promises but its powers “are retrospective and often exercised through drawn-out legal battles,” said the parliamentary committee.
Blundy saw an opportunity to automate the system. “This kind of approach follows on from a prevailing institutional cultural view that the problem or phenomenon of evaluating a company’s ability to support its scheme was too difficult and too nebulous a concept to model mathematically,” he said.
“Because we’ve been doing this since 2005, there was a large stock of institutional knowledge codified that reflected ten years of these assessments. All of that expert judgement had been distilled and hadn’t yet been used in a systematic fashion and I saw this as an opportunity for machine learning. I just had to gather the data,” said Blundy.
Using this data from past assessments, Blundy has built a machine-learning algorithm to identify trends and use them to make judgements of new cases. The new model can correctly classify 65-70% of schemes, and rates 85% of the remaining 30% to within a single grade.
“It’s pretty accurate,” said Blundy. “In terms of the complexity of the phenomena we’re modelling, 65-70% is a pretty good result and I think it’s certainly far better than what it was thought I’d be able to achieve.”
While Blundy will need to improve the accuracy of the model further to implement it fully, it is already being used as a monthly horizon scanner, allowing analysts to spot risky cases they would otherwise miss. The current three-year evaluation process means that assessments are quickly out-of-date. The new model can provide a regular check-up to point the pensions experts in the right direction.
Blundy plans “to plough ahead with the things the model can do but our expert professionals are unable to do, which is namely to provide a point in time assessment across all 6,000 pension schemes on a weekly or monthly basis.”
The main hurdle Blundy has encountered so far is convincing colleagues that the model is a viable alternative to professional evaluation of pension providers. The model is therefore his first step in making the case that machine learning can make a significant difference to the way TPR operates.
“What I would like to do as an analyst is to get people to consider whether this could be used as a front line filter,” said Blundy. “Do we really need for all of our lowest risk cases to be assessed by an expert? Can we more efficiently deploy our experts towards the higher risk element of the [pensions] universe?”
Blundy entered the Government Digital Service’s Data Science Accelerator in November 2016 to develop the model. The Accelerator provides civil servants from across the service with training in programming languages and data modelling under the tutelage of a mentor. Over the course of three months, Blundy learnt a new programming language, R, from scratch, which, combined with the dataset he had assembled over the previous two years, he used to build the model.
The model is unlikely to fully supersede the role of the pension professionals because of the high stakes of the task and the nature of the assessment, and it is unclear whether the collapse of the BHS fund could have been averted had problems been spotted sooner. TPR would still have needed other powers to intervene effectively. While Blundy has built an accurate and effective forecasting model, it is likely to be used in combination with human experts for some time to come.
(Picture Credit: Flickr/Paul Townsend)