Being “evidence-based” has become an important part of policymaking. But what happens when the evidence tells a story that you didn’t want to hear? Evidence Action, an organisation which scales up evidence-based programs throughout the world, recently faced this question when it came face to face with disappointing evidence about one of its projects.
No Lean Season, a project based on research from Yale University, was designed to help mitigate the effects of seasonal poverty on agricultural communities in Bangladesh. Yet despite a successful trial, the program showed no impact in its next phase, and Evidence Action had to cut it short.
Here’s what they learned, and why their work shows the importance of thorough testing at scale, as well as listening to the data, even when it doesn’t tell a great story.
From RCT to Beta Incubator
Central to Evidence Action’s mission is only undertaking projects that are backed up by rigorous and substantial evidence.
This means that each program it takes on moves from a successful randomised control trial (RCT) to an extensive “beta incubator” phase, in which the program is implemented at scale, but evaluated closely for impact.
This is almost like a “probation” period, to make sure the program does what it’s meant to, and it may include continued A/B testing to help iron out which approaches are most successful at scale.
Kanika Bahl, CEO of Evidence Action, explains that she never takes for granted the notion that a program which has been successful in the research and RCT phase will work once it’s in the beta incubator phase. “Just because something comes from a very promising research base or the mechanism is well known, we never assume that the program is having the intended impact”, she says.
No Lean Season was intended to help people in rural Bangladesh, where the season between planting and harvesting can plummet families into hardship. The program offered a migration subsidy to workers who would benefit from travelling to an urban area where fluctuations in jobs and wages are much less severe.
The central questions Evidence Action had about the program were twofold. First, was the rate of migration increasing in the areas that were being treated? And secondly, was the household income and/or consumption rate increasing along with that migration uptick?
The data speaks
When program leaders sat down to analyse the impact of the No Lean Season incubator phase, the data yielded disappointing results. There had been no significant impact on migration and therefore no effect on household consumption and income.
The question that naturally followed was: What went wrong?
Evidence Action’s best hypothesis, Bahl said, was that there had been issues between the organisation and its local partners in calculating the number of loans that were given out. This led to the targets that were set at the beginning of the incubator phase being met too quickly, without actually increasing the number of seasonal migrants.
The local partner set out to survey 100,000 households and assess whether they were eligible for a migration loan from the program. Evidence Action then set targets for the amount of money to be given out, based on how many of these households would actually take up the loan and migrate.
In the end, the partner surveyed 160,000 households, but the amount of loans given out wasn’t higher than if they had only surveyed 100,000. This meant that the percentage of people from the group offered loans who actually migrated was no higher than the control group, because fewer loans were given out than anticipated.
While this is Evidence Action’s best guess about why impact was so low, Bahl reiterates that this is only a hypothesis.
On top of this issue, Evidence Action received allegations of financial impropriety by their local partner. While a thorough investigation into the allegations were unable to corroborate them, in the end it made sense to terminate the relationship with the local partner.
This meant that in 2018, No Lean Season was looking at unsuccessful results from their beta incubator, with no impact on migration or household income, as well as no local partner on the ground to help with fixing these implementation issues for the next year.
This is when Evidence Action took the difficult decision to shut down No Lean Season. In the end Kanika Bahl thinks that it was an indication of Evidence Action’s integrity and commitment to its mission.
“Because we were testing at scale, and well beyond you’d typically do in an NGO, we were able to detect that the program wasn’t working.”
If the organisation had relied on the RCT alone, she added, “we would have said, Gosh there’s this really strong evidence, we’re in great shape and we’d have continued the program, and continued feeling really good about pouring millions of dollars into a program which wasn’t having the intended impact at all.”
Part of the decision came down to Evidence Action’s experience with highly successful programs in other areas, including the Deworm the World initiative, which leveraged existing infrastructure to deworm children in their schools and Dispensers for Safe Water, which has brought clean drinking water to over 4 million people.
“We are reaching over 280 million people today with our other evidence programs,” Bahl said. This caused her to think, “Is this the best use of organisational time and attention? Is the best program we could be doing? Or the best way for us to achieve our mission of measurably improving the lives of hundreds of millions of people?”
Ultimately, Bahl sees this as less a story about implementation challenges and more one about how to make difficult and tactical decisions based on evidence. After all, scaling programs in the developing world is complex and difficult. It’s unrealistic to assume that every single project will be as successful as it hopes.
“It can be painful, but if you’re not making these tough calls, there’s a good chance you’re not actually be honest with yourself about what actually is the most effective way to be using development dollars and resources”.
– Megan Dent
(Picture credit: Flickr)