"The world is scattered with pilot projects trying to work holistically"

This piece is part of Apolitical’s spotlight series on scaling social impact, in partnership with the Bernard Van Leer Foundation. It also appears in our government innovation newsfeed.

“We’re not a poverty organisation. We think of ourselves as an evidence and scaling organisation,” said Karen Levy, Director of Global Innovation at Evidence Action.

Evidence Action is an international NGO that bridges the gap between proven interventions and delivery at scale. Or rather, huge scale: they deal in the hundreds of millions. Evidence Action formally launched in 2013, and Levy has been with them from the beginning. She helped bring two of their flagship programs into operation.

The first is Deworm the World. This intervention involves school-based, blanket drug administration to children in areas where worms are prevalent, with proven health and educational benefits. In 2016, almost 200 million children were treated across Africa and Asia.

The second is No Lean Season. This targets seasonal poverty by offering agricultural households a travel subsidy so they can send a man to a nearby city, where he can find work and send money home. It is currently being tested in Bangladesh and Indonesia.

As part of our special series on how to scale up social programs, and help millions or hundreds of millions of people rather a few thousand, we spoke to Levy about her work at Evidence Action.

Your interventions are strikingly simple and scalable. How do you arrive at them?

We look for things that are elegant, well-defined and honed. There is a tendency in the development industry to try and approach problems from a holistic perspective. But when you talk that way, it becomes very hard to find an entry point. Yes, everything is connected; yes, everything is complicated — but if you let that be the framework through which you start, you won’t get anywhere.

"We look for things that are elegant, well-defined and honed"

That’s why the world is scattered with pilot projects. Lovely pilot projects that are trying to deal with holistic issues, but are never going to get beyond 50 schools or 50 villages. If you look at the things that have achieved massive scale, they are well-defined interventions — or at least started that way.

I always tell people: don’t try and paint the masterpiece — do one layer, and do it well, then do another on top of that. Just creating or strengthening a platform to deliver something simply but well gives you the opportunity to build other stuff on top of it.

When you’re expanding a program, how do you analyse prospective sites?

For Deworm the World, we ranked and prioritised countries along certain criteria.

The first thing we looked at was worm prevalence: the need. And not only where there is a high prevalence, but what geographic pattern it exists in: are they deep isolated pockets, or is it the entire western third of a country? So the pattern and depth of prevalence was one factor.

Another factor was school attendance. This speaks to the scaling strategy. I remember in the early days of the program we were considering Madagascar, but at the time because of political unrest, school attendance was going down. That’s a problem in and of itself, but it was also a problem for our program.

"The world is scattered with pilot projects"

Then there were certain supply chain issues: only some countries are eligible for WHO drug donation programs. If not, we checked whether pharmaceutical companies were able to provide it on their own.

These were just some of the criteria, but if you think about what are the core ingredients that make a school-based deworming program work, we were systemically looking for them.

For the No Lean Season program, we started looking for areas where the underlying characteristics exist that we think contribute to why it works in Bangladesh. First we look for the existence of seasonality: are there large groups of people that experience seasonal privation?

That gives you a set of places. Then you look within that for places where there are growing and thriving cities nearby. Two examples of places we looked at that passed the first test but failed the second were Zambia and Malawi. They have an acute hunger season, but there aren’t surplus wage labour opportunities in towns and cities — so they have that nail, but our hammer isn’t going to work there.

And within that set of places, you’re looking for those where the gap between the impoverished and the nearby towns and cities is far, but not too far. If you have to buy a plane ticket, that’s not something that No Lean Season can solve. But equally if the fare costs just three dollars then perhaps a transport subsidy is not going to solve that problem. We’re looking for that bit of resistance that a subsidy can overcome.

Once you apply those rigorously, it’s a lot of places but it’s not everywhere in the world.

Do you adapt the interventions to new contexts?

That’s where the rubber meets the road in really figuring out the implementation model and the delivery platform. That’s very much context dependent — although national or regional.

Thinking about No Lean Season, we’ve been doing some work to see if it will work in Indonesia, a place that’s much more internally diverse than Bangladesh is. The core theory holds, but the travel patterns for migration tend to be different. The way people send money home tends to be different. And that’s where that external validity question starts: if you’re really building a different model, then you need to test it again.

Do you always design your interventions for adoption by government?

Governments are the vehicles of scale in society. Layering these programs on government structures is a huge opportunity, but it’s a skill to know how to engage with government and how their systems work.

"Often you need to design an intervention with government adoption in mind"

Often you need to design an intervention with government adoption in mind. A good example of this: theoretically, you could design a deworming program that was more cost-effective by having it roll out in waves across the country. Because then you could use the same people over and over again as it moves from one place to the next. That’s fine — but you would need to have ongoing procurement, ongoing disbursements of operational funds. There’s a reason why lots of government campaigns are annual — the systems are much better suited to that.

Working at the scale you do, how do check whether the impact is being maintained?

With No Lean Season we are currently doing an evaluation at scale. So the program is being delivered as it would be in steady state, but we are rigorously measuring impact. But we’re not going to do it forever, because you lose the ability to do that as you get bigger and bigger.

There are sometimes opportunities to do experimental designs as a program grows — as it grows into a new area, for example. You can introduce a program with random variation, and that’s a way to check in on it.

Also you have your theory of change: A leads to B, B leads to C, and so on. Well, you may not be able to do a rigorous evaluation of every step from A to G, but you can do bits of it — and you should.

You can also keep checking that the underlying assumptions are still being met. For example with No Lean Season, one of the requirements is that wages are different in the two places — are they still different?

What’s the endgame?

With Deworm the World, the endgame is getting it to the point where worm infection is no longer a public health problem. That's when the transmission rates are such that it is no longer a population level problem.

With No Lean Season, I would be looking at what happens with wages over time. If you get to a point where the seasonality of rural wages flattens, then that would be an indication for me that we should stop doing this program.

I built these programs and they’re like your children — it’s very hard to contemplate shutting them down. One of the tritest sayings in our industry is that we’re all trying to work ourselves out of a job — really? Come on. It takes courage to do that. There aren’t enough examples of it. And that’s only partly because these problems are hard to solve.

(Picture credit: Flickr/Evidence Action (Stephanie Skinner))

Tom Graham