There is a placard sometimes seen at political rallies, which goes something like this "What do we want? Evidence-based change. When do we want it? After peer review."
Cynicism aside, most reasonable people want to implement policies that work: that is, policies that achieve the intended goal without too high a financial cost and too many negative externalities. Fortunately, many academics are able and willing to turn their research resources—provided in large part through public funding—to figuring out which policies work, where, when, and why. But academic journal articles are not written primarily for policy makers: they are written and then peer-reviewed by researchers for other researchers. Researchers are often interested in contributing to policy-making, but the journey from a piece of published research to a viable policy is rarely simple.
Here are three questions that policy-minded readers should ask about any research report they read:
1. Is the finding robust?
The peer review process puts a paper before other experts, whose main job it is to evaluate whether the research has been done properly: is the sample adequate?, are the measures good?, are the statistical analyses appropriate?, etc. But there is increasing evidence that peer reviewers—even for well-respected journals—are not very good are identifying methodological inadequacies, particularly those involving statistical analyses. This has led to recent claims that “most research findings are false” and that there is now a crisis, especially in the biomedical and social scientific fields that are potentially most useful to policy-makers.
Given the failures of professional researchers of detecting questionable research practices within the guild, it is probably too much to ask that policy-makers do so instead. The best course of action is therefore to find out whether a result has been replicated, preferably by multiple research groups. Systematic reviews and meta-analyses of multiple studies can also one useful for identifying how robust or fickle an intervention might be.
2. What is the context of the reported intervention?
A well-conducted study can tell us whether a policy works at a particular time and place: what it cannot do is tell us whether the exact same policy would work in our current context. As it often does elsewhere, context matters in evidence-based policy-making. There are probably no policies that are inherently effective or ineffective: whether a policy works will depend on a whole host of contextual factors, including infrastructural, political, and cultural ones that may be difficult to measure. It seems uncontroversial that bicycle helmets are a good way to prevent injuries: but meta-analyses have found high levels of variation across studies, probably due to differences in physical and social environments.
There are two recommendable courses of action here. First, policy-makers should aim to find research conducted in contexts as similar as possible to their own. Failing that, however, they may also look to see if an intervention has been shown to work in quite diverse contexts. The more universally effective a policy has been, the more likely it is to also be effective in a novel context.
3. What do we know about the causes of the effect?
For practical reasons, most of the available data relevant to public policy is correlational: academics look across jurisdictions for statistical relationships between policies and outcomes. For example, we might want to know whether minimum wages are associated with employment rates, or whether legal restrictions on firearms are associated with gun-related deaths. The easiest way to do this is to compare different jurisdictions that vary on both of the factors of interest. But this method does not tell us whether the policy has caused the outcome. For example, there is some evidence that at least since WWII, the US economy has done better under Democratic presidents than Republican presidents. However, it’s not clear whether this is due to Democratic fiscal and monetary policies, or whether the Democrats have just had better luck on external events, such as fluctuations in oil prices and technological advancements.
There is a hierarchy of study designs when it comes to the causal conclusions we may draw from a study. Correlational studies provide much less certainty that longitudinal studies, which look directly at how things change after a policy is implemented. In turn, longitudinal studies provide less certainty that randomised controlled trials. Furthermore, the best studies—regardless of design—tell us not only whether a policy works, but also how. If we can understand the causal pathways from policy to outcome, we are also in a better position to figure out if a policy is likely to work in our context. This is the final tip, then: policy-makers should try to understand how a policy works, and not just the mere fact that it has worked somewhere before.
Experimenting with policies
This answer began with the assumption that research on public policy can be outsourced to academics to some extent. While I stand by this assumption, this does not mean that the government should neglect monitoring the effects of the policies they do implement: of course, this does not have to be done in house. Academic researchers may be incentivised to help the government in its evaluative work. And if the findings of this research can be shared to benefit others, so much the better.