This opinion piece was written by Philip Horgan, Head of Monitoring, Evaluation and Learning at the Open Data Institute (ODI), a not-for-profit working with companies and governments around the world to build an open, trustworthy data ecosystem. It also appears in our digital government newsfeed.
We were interested to read a recent article in Apolitical by Stefaan Verhulst from The GovLab, arguing that to turn the open data dream into reality we need more evidence. The article argues that in order to move on from an environment where there are many open data pilots to an ecosystem where such initiatives can scale, there needs to "more data on open data" and robust evidence of the impact it can have. Only then can these projects attract sufficient support from stakeholders and investors and begin delivering real benefits.
We, at the ODI, agree with this. Some insights from our recent work might shed light on why this is and what can be done about it.
Why do projects have difficulty in progressing past pilots?
There can be many reasons why projects have difficulty progressing past the pilot stage, but the lack of evidence of the benefit of publishing the results, is certainly one of them. We gained some useful insights into this from a recent study of OpenActive, our work in the physical activity sector.
One of the insights gained from the study was crystallising the notion of first wave and second wave data publishers. The first wave of data publishers are those organisations that “get the vision”, they publish their data openly because they buy into the idea of the work and see a greater good that can be achieved. They tend to be organisations involved in the pilot stages of a project (or wider open data initiative), who don’t necessarily need economic or social impact calculations to publish their data.
There is also a second wave of organisations that have suitable data to publish but for whom the vision is not enough. They need evidence of the benefit of publishing data they hold on their bottom line. For second wave organisations, decisions are made based on justified business cases for which economic evidence is often needed.
What does the evidence look like?
The evidence that a data publisher is looking for comes in the form of the value from the use of data and associated decision making. This is where the data value chain from ODI’s theory of change is useful. It illustrates that value is generated as data is stewarded, data services (such as analysis) are created, and decisions are made.
The ODI theory of change
This seems to be true whether it is the use of education achievement data at the Tanzania Education Open Data Dashboard, the scrutiny of public contract and procurement data in Slovakia (to pick two examples from the GovLab website), or from the use of “opportunity” data on physical activity in ODI’s OpenActive program. In all cases, it is the use of the data which generates value by enabling better decision making. Evidence needs to demonstrate how this use of that data in decision making brings value back to second wave organisations.
This brings us back to a well-run argument: it is not enough to rely on the idea that, if you publish, uses of the data will emerge. Rather, specific uses should be supported on which the required evidence can be generated.
Sometimes the pathways to use are not direct. In the Tanzania examples above, Aidan Eyakuze, Executive Director of Twaweza, reflects that “Open data needs to be open plus curated plus chewed plus digested to appeal to most people, including policymakers,” but “few in the media, however, have the knowledge and skills to digest Shule’s data offerings”.
Alternative pathways may be available: “Members of civil society organisations, for example, including Tanzania’s active parent-teacher organisations, can act as intermediaries, printing out information about school performance to share on a community notice board or at meetings”. Specifically supporting these pathways to use can help to generate the evidence of use.
We have found that for second wave organisations, the evidence needs to rely less on stories of intent or visionary descriptions of benefits, and more on quantifiable elements such as increases in efficiencies and returns on investment. In studies of ODI’s OpenActive initiative, potential data publishers repeatedly highlighted the need for return on investment calculations and evidence of the demand for data to enable their organisation to publish.
Back to the questions we started with
Self interest is a powerful motive for any organisation and “second wave” organisations need a robust business case to support decision making to publish data.
In the OpenActive initiative, we shared audience and reach figures of campaigns like Change4Life and This Girl Can with leisure operators, to help them see the value of opening up their data. This information has helped them and other activity providers create the business case needed to publish their data. We expect that evidence about the different ways in which the data can be used to influence and change customer behaviour will also encourage more organisations to get involved — either as data publishers or as data users.
For organisations supporting open data projects and initiatives, the lesson is to think about capturing evidence early, so the initiative can progress beyond the first wave of enthusiastic publishers. — Philip Horgan
(Picture credit: Open Data Institute)
Make sure to share your own thoughts with the author by leaving a comment below
Log in or sign up to continue the conversation