Evidence hacks: Choosing studies in a hurry

This article is written by Michael Sanders, the executive director of the What Works Centre for Children’s Social Care.

If you ask most civil servants whether they think using evidence is important in their job, you’d get a resounding yes. There are very few people who would say that policy should not be based on the best available evidence. Despite this, the use of evidence can often be patchy, or even non-existent. Why?

Aside from a few very rare cases, the dearth of evidence is not due to malice or incompetence. It’s because officials are working under the pressure of deadlines and need to quickly condense multiple sources of information — from inside government, from stakeholders, and from research. A lot of evidence comes from academic papers that aren’t exactly quick or easy reads, so it’s perhaps not surprising that it’s the first source of information to get discarded under pressure.

There’s obviously a lot that the research community could do to communicate their findings better, but what if you’re a  civil servant in a hurry? What can you do to improve your use of evidence? Having worked as a consumer as well as a producer of evidence in roles in government, at the Behavioural Insights Team, as an academic and most recently at What Works for Children’s Social Care, I know both how important research evidence is, and how hard it can be to make time for.

To start with, there are a few questions you can ask of any research — be that academic papers or think tank reports — you come across. These questions will help you to quickly decide whether you want to read further, and whether a source is likely to be helpful in deciding what to do.

Reviews and Reviewers

Before you start, it’s worth asking yourself what kinds of papers you want to include. After all, reading hundreds of individual studies is incredibly time-consuming. In addition — and perhaps even more frustrating — the conclusions are often contradictory. Systematic reviews, meta-analyses, and other forms of summary paper condense the findings from an entire area of research into a single digestible form, and can be more useful than the sum of their parts. For a growing number of policy areas, there are also organisations like the What Works Centres which review the evidence and present bitesize policy recommendations.

Once you have collated your research, here are some critical questions that will help judge its merits:

What kind of language are you looking at?

Are you looking at a paper that’s making causal claims, or not? Does it say “X causes Y” (a causal claim), or “X is associated with Y” (a correlational claim)? If the latter, you shouldn’t use it to inform statements about the impacts of a particular policy, or the causes of a particular problem. If the former, you maybe still shouldn’t…

Comparing apples and bears

Most causal claims should be supported by some kind of counterfactual. In other words, the researchers should be comparing a group that something happened to (i.e a new policy or intervention) with another group that it didn’t happen to (also known as a control group). If they’re not doing this, you can probably disregard any causal claims.

The two groups should also be similar before it makes sense to compare them. If the authors are making a comparison, ask yourself whether the groups are really similar. In this case, you can look at statistics (known as balance checks) to help you — most papers will include balance or comparison tables that tell you how similar the groups are — but in some cases, your gut instinct will also be enough. Are young people at Further Education colleges in the North East of England similar to young people in sixth form colleges in London? Probably not. Are families that narrowly qualify for tax benefits because of their income broadly the same as those that narrowly missed out for the same reason? Probably yes. This might not sound like a common flaw in research papers, but it is a lot more prevalent than you would think.

Is it big enough?

If a paper is using statistical analysis, you should look at the sample size. Anything with a small number of people — anything less than a hundred people receiving an intervention is a good rule of thumb – should be treated with some scepticism unless the research is carried out in a very controlled environment where there’s not a lot of variation. Studies that rely on small samples are often not representative, and they often fail the “balance” checks described above, and will find it hard to statistically detect impacts even if they exist.

Too good to be true?

If a study’s findings look too good to be true, they probably are. This may be a harsh rule of thumb – after all, there are some “Unicorn” type interventions that do have very large effects.

But if you’re skimming over lots of research to try and work out what to include and what to discard, implausibly large effects are a good place to start. Selection bias and other statistical biases, as well as publication biases (the tendency for only positive or novel results to get published), all favour large effects. Most real effects, particularly in a policy context, are pretty modest.

Little, often, and fun

Perhaps the most useful piece of advice I can give doesn’t relate to any single study or topic area. Instead, it relates to the way you approach evidence in the first place. If you get in the habit of reading evidence regularly, even in small amounts, the act of doing so inevitably becomes easier, and if you’re anything like me, you’ll already have a stock of potentially useful things to read. As well as the What Works Network, some newspaper columnists, podcasts and outlets like Apolitical and The Conversation regularly feature research findings to help you keep up to date.

Staying on top of research and evidence can be time-consuming, but it’s an important part of an official’s role to understand the current state of the knowledge in their policy area. These tips can help with this, but they’re no substitute for allowing enough time in the working week for you and the teams you lead to stay informed. – Michael Sanders

(Picture credit: Unsplash)