In the public sector innovation community, the question of how or whether to measure impact is guaranteed to get people fired up. Some innovation labs reject the very idea of evaluation, arguing that the value of innovations is often implicit or impossible to quantify.

But design and innovation work challenges many ingrained aspects of public sector culture — chief among them financial pressures, siloed departments and stubborn bureaucracies. Doesn’t the onus fall on those challenging the status quo to justify the importance of their work?

This evaluation question has come up repeatedly on Apolitical’s Innovation Lab Show & Tell call, a monthly video conference joined by labs from over 35 countries. On 9 April Jo’Anne Langham, who is currently researching a PhD in Design Evaluation at the University of Queensland in Australia, shared a new tool that allows innovators and designers to measure the impact of their work, which she calls the Experience Effectiveness (XE) Framework.

“What we really need to know is if innovation has led to change, and whether this change is delivering the right outcome"

“The public sector hierarchy is designed to be robust, provide stability, protect revenue and offer broad guidance, which is reinforced by the people and culture. It’s known for stability and risk-averse behaviour,” said Langham, a former senior civil servant in the Australian Tax Office. “Innovation is not something it does well.”

Stakeholders in an innovation project typically want to see data relating to return on investment: how much did the design process cost? Are we achieving more for less?

But, Langham argued, this approach misses the point. What we really need to know is if the innovation has led to change, and whether this change is delivering the right outcome.

To illustrate this, she used the example of building a new house. To measure its value, we need to look at the outcome delivered: does it meet expectations for children and adults alike? How does the experience of living in the new house compare to that of the previous home?

What many governments would measure, however, is the time spent building the new house and its cost-effectiveness.

As a result, innovators must make concessions by learning to speak “the language of government” which, according to Langham, often means a focus on hard financial outcomes. In order to paint of a picture of an innovation’s impact, labs must deliver data in both qualitative and quantitative forms. The former can be used to tell a story, while the latter distils complex information into easy-to-understand figures, such as the cost savings of an intervention.

“If we instil the importance of appropriate measurement, we will create evidence and support"

“If we instil the importance of appropriate measurement around outcomes, we will create evidence and support,” said Langham. “This ensures funding is prioritised in the right areas.”

The XE Framework can be conducted by an evaluator using experience mapping and interviews. They will ask innovators and designers questions such as: who are the actors? Was the design delivered as intended? Did it exacerbate existing problems or create new ones?

Once evaluation is complete, the framework gives labs an overall score for effectiveness with a breakdown of individual components, highlighting which areas need improvement. It also provides data on whether the innovation led to an incremental or radical change.

The first step for labs beginning to evaluate their work, Langham said, is investing in research, planning and information-gathering prior to the start of a project. “You need to be filming, photographing, and measuring all you do. Show your work and how it was costed. Collect data on everything.”

(Picture credit: Pixabay)

Jennifer Guay
jennifer.guay@apolitical.co