__
This post is written by Thea Snow and Adrian Brown of the Centre for Public Impact__.
- The problem: The evidence-based policy movement has lost its way.
- Why it matters: If a policy is based on evidence that is biased, poor quality, or which fails to take into account local conditions, the fact that it is evidence-based doesn’t make it rigorous at all.
- The solution: Evidence-informed, rather than evidence-based approaches to policy-making
The term “evidence-based policy” was made popular in the 1990s by the Blair Labour government in the UK and, over the ensuing years, “the ideals of evidence-based policy became a global movement.” Now, a couple of decades on, how do we feel about this?
We believe that the “evidence-based policy” movement has largely lost its way.
It feels hard to argue against the idea - promoted by New Labour - that we should “improve our use of evidence and research so that we better understand the problems we are trying to address.” However, we believe that the “evidence-based policy” movement has largely lost its way. In the pursuit of a more rigorous, research, and evidence-oriented approach, proponents have sought to simplify and codify the policymaking process. This process reflects a logic of legibility, which is largely unhelpful - particularly in complex social environments, which characterise much of the policymaking domain.
To be clear, we are not arguing against the value of good evidence in helping to shape effective policy. Rather, we are suggesting that the popularisation of evidence-based policymaking has created an environment where we are venerating evidence at all costs, rather than taking a critical approach. We seem to be forgetting that evidence should be used in service of good policy. Policymakers, when using research and evidence to shape policy, often appear to be forgetting to ask:
- Is this good quality research evidence?
- How is power and institutional bias shaping how we choose and use evidence?
- How might the insights from research and evidence need to be adapted to suit local conditions?We will briefly explore each of these in turn.
Quality of evidence
Not all evidence is good evidence. This seems to be too easily forgotten by the evidence-based policymaking movement. In the enthusiasm to embrace an evidence-based approach, the veracity of the evidence oftentimes appears to go unchecked. This can lead to questionable – and ill-informed – policymaking.
For example, in a previous blog, we have highlighted the way that outdated, statistically insignificant, and generally problematic evidence was used to strongly discourage prison visitation programs for young offenders all around the world. We explain,“this is a prime example of overreach, where a rather limited and outdated set of studies from a particular place is used to make a bold statement about what works in general now.”
There is good evidence, not-so-good evidence, and bad evidence (and everything in between). It is critical for anyone using evidence to interrogate its quality, and question whether it is scientifically sound, before relying on it to guide their decision-making.
Power and institutional bias
Evidence is not neutral or objective. The research and evidence used by policymakers reflects a particular set of (unspoken and generally unacknowledged) values, norms and biases which influence how studies are designed, which data are selected (and which are not), which data are seen to be valid (and which are not), and how data is interpreted and applied by policymakers.
These invisible biases can have profound consequences for how insights from research and evidence are interpreted and translated to practice. For example, while urban planning decisions are increasingly evidence-based, there are “significant gaps in gender-and sex-disaggregated data” and a paucity of evidence around the preferences of minority groups, resulting in cities which continue to work better for heterosexual, able-bodied, cisgender men than they do for women, girls, sexual and gender minorities, and people with disabilities.
In Australia, the framing of research and evidence from the standpoint of the dominant racial, social and cultural majority results in government policies — such as the Closing the Gap agenda — which fail to accommodate and value Indigenous cultural difference and which therefore don’t reflect the outcomes which are important to the communities that the policies are designed to support.
How might we avoid this? Firstly, policymakers — as well as those generating research and evidence — need to be attuned to “how [the evidence] is constituted, who validates knowledge, what knowledge systems are privileged and who gets to decide." As Catherine D’Ignazio and Lauren Klein explain,“doing good with data’ requires being deeply attuned to the things that fall outside of the dataset — and in particular to how datasets… often reflect the structures of power of the world they draw from.”
This is not easy, but there are practical steps that can be taken to help. Ensuring diversity in research and policy teams is an important step, as is a commitment to reflexivity.
Adapting evidence to its context
Another challenge for evidence-based policymaking is the quest for standardisation of “what works”, which crowds out space for nuance and local adaptation.
What works here does not necessarily work there due to the myriad of local factors not considered by the original studies. Conditions including demography, geography, local infrastructure and culture influence how a policy plays out in a given place. As this podcast explains, "What randomised controlled trials tell us about an intervention is what that actual intervention does in a particular population, in a particular context. It doesn’t mean it’s generalisable.”
Given this, while evidence can be used as a starting point for guiding practice, it requires local adaptation by experts on the ground. For example, while there is evidence available about “what works” in children’s social care, it is critical that the social workers using this evidence adapt it to meet the unique needs of their clients, in their particular context, at that moment in time.
The greatest risk with evidence-based approaches is that they imbue policymaking with a veneer of objectivity and credibility.
We need to be encouraging those using evidence — from policymakers to frontline workers — to thoughtfully take the evidence into account, but to combine it with their own experience and judgment. This intelligent combination of quantitative data, together with professional expertise and qualitative insights (sometimes called thick data), is most likely to result in policies being implemented in ways which support the outcomes that are most important to people.
From evidence-based to evidence-informed policymaking
Does research and evidence have a role to play in supporting better policymaking? Absolutely it does. But in our quest to inject more rigour into the policymaking process, we need to take care not to overstate what evidence is, or what it can do. The greatest risk with evidence-based approaches is that they imbue policymaking with a veneer of objectivity and credibility, which may be justified in some cases. But if a policy is based on evidence that is biased, poor quality, or which fails to take into account local conditions, the fact that it is evidence-based doesn’t make it rigorous at all.
Borrowing from Research in Practice, we believe that the time has come to shift from evidence-based, to evidence-informed policymaking “being ‘evidence-informed’ differs from being ‘evidence-based’ or ‘research-based’, not only because it explicitly involves drawing on multiple sources but also in that the intention is to actively disrupt traditional power relationships between different forms of knowledge, and so promote a more inclusive equalities-aware model of evidence use.”
This means thinking hard about how to derive insights from blended forms of evidence including quantitative and qualitative research, expertise, and people’s “baggy, winding stories”, which can be hard to make sense of in neat ways.
It means acknowledging that all evidence is necessarily plural and incomplete, and that critical thought and constant interrogation is necessary.It means recognising the way that power and racism influence research and evidence, and actively seeking to address this.It means making sure that evidence is viewed and used as the servant, not the master, of good policy. __— Thea Snow and Adrian Brown __
👋 You can create a post like this one! Share your thoughts with a community of public servants. Learn more.
(Picture credit: Unsplash)
Log in or sign up to continue the conversation