Evidence in policymaking: The basics

In 1984, a farmer in the south-eastern part of England called a local vet to investigate why one of his cows was acting erratically. The diagnosis: bovine spongiform encephalitis (BSE), the first reported case, which would lead to one of the biggest health scandals in the UK.

As years elapsed since that first case, the government acted cautiously — it avoided introducing schemes outlawing the slaughter of infected cows that would then enter the food chain. Officials had no substantial evidence that BSE could be transferred to humans – such evidence didn’t yet exist and the government was slow to commission it – so they deemed the meat safe. The result: 178 deaths were reported due to BSE.

Using evidence can be essential for evaluating the societal impact of a program or policy, and for making informed responses in the event of a crisis. But putting in place the right mechanisms to obtain evidence and giving public servants the technical knowledge to find it are crucial.

Evidence is core to policymaking

Reports detailing the bungled response to the BSE crisis not only revealed grave mismanagement but a failure to relay concerns to the public.

It found that there were consistent delays in gathering quality evidence to evaluate the effects BSE-contaminated meat had when consumed by humans.

It took two years for policymakers to set up a research committee to advise on evidence for the BSE epidemic.

Meanwhile, ministers weren’t briefed on research in a timely manner while information sharing between the Department of Health and Social Care and the Department of Environment, Food, and Rural Affairs (DEFRA) was slow to materialise.

“Systems inside government departments are not set up well to provide the advice, and public servants don’t know where to turn, so not everyone has pieces of evidence at their fingertips,” said Louise Shaxson, head of digital societies at the Overseas Development Institute think tank (ODI).

But quality evidence is not always easy to gather. Public servants may not have the skills or the right mechanisms in place within their department to obtain it.

If there is an outbreak of a new disease, for example, policy teams should rely on research to shape their response, but they may not know which scientists or statisticians to contact to get information, said Shaxson.

An advisory committee comprised of a broad range of experts from different research and academic institutions can help assess the quality of evidence, provide guidance on the evidence gathered and ensure that information is free of bias.

Having “knowledge brokers” – a scientific advisor for example – to help public servants evaluate and interpret evidence is another way to support public servants through the process. A later article in this guide tackles this concept in more depth.

Following the BSE scandal, DEFRA has since developed one of the most extensive evidence-gathering strategies within the UK government. The overall goal is to improve access to evidence and ensure the department is making the most of public research investments.

Part of the plan is to work with partner organisations and external experts on solving problems through evidence sharing. For instance, to gather more evidence on water pollution, a Demonstration Test Catchment (DTC) research platform was set up so that data points and research can be shared between the department and external researchers.

DEFRA has also established a “Quality Framework” that covers different elements of evidence quality such as assurance, control, assessment, governance, corporate processes and communications. DEFRA requires the framework to be used by all public servants within the department and by external partners to ensure decision-makers have the assurance that evidence is consistent and of high-quality.

Stay away from Google

Building networks between policymakers and science teams, creating evidence-based strategies and having systems in place within a department so public servants know where they can quickly access evidence are all important.

But public servants without a science background may lack the skills to use technical methods such as systematic reviews to assess the available evidence.

Shaxson said there is a tendency for public servants to begin by simply Googling research to support their efforts, which she deemed “one of the worst things that can be done”.

Instead, setting out the scope of the policy issue and determining what questions need to be addressed should be the first step in the process.

Once the scope and goal of the policy are established it’s then easier to gather evidence that either supports or challenges those outcomes.

Categories of evidence

ODI categorises evidence in four different ways: statistical and administrative data, analytical evidence, evidence from citizens and stakeholders and evidence from evaluations. No one type is more important than the others.

Statistical and administrative data is classified as data used for government record-keeping, including birth rates or greenhouse gas emissions.

Analytical evidence refers to evidence that emerges from social science and natural science research to explain causality.

Evidence from citizens and stakeholders brings in insights and expertise from the people closest to the policy.

Lastly, evaluations look at past programs to review how they worked and for whom.

“If you’re weak in any of those [research] areas your decisions are going to be weak. If you don’t have a reasonable balance then you’re going to make poor decisions,” said Shaxon.

For instance, policymakers often focus on analysis, evaluation and statistics but fail to adequately insert public knowledge and opinion into the process.

Being thoughtful about the way evidence is delivered can also improve the policymaking process. For instance, Shaxon said when she worked in the public service, policy teams would receive 600-page reports, which were too time-consuming or complicated to review. Asking institutions to synthesise reports to 25-30 pages with a focus on the relevant policy questions and goals may be better for informing decisions.

Requesting that institutions do a departmental workshop on the findings is another alternative, she added.

“The trend is that as we see the problems we’re facing as more complex and interconnected, collecting and using evidence helps reduce the risk of making a poor decision,” said Shaxon. – Amelia Axelsen

(Picture credit: Pixabay)