A pilot is not a prototype: How to test policy ideas before scaling

Opinion: Use-cases for prototyping policies

This article was written by Marc K. Hébert, the Director of the Innovation Office in the San Francisco Human Services Agency. A version of this article originally appeared in the June 2019 edition of Policy & Practice magazine by the American Public Human Services Agency. For more like this, see our government innovation newsfeed.

“I didn’t know it would take this long, or what the next steps were. I waited a long time and had to hurry to complete some forms before a deadline. What a hassle. I almost didn’t go through with it.”

Some variation of these words has been shared with me over the years from clients, contractors and employees of various government departments trying to get something done. I’ve even said something like this myself. There are many reasons why policies produce inefficient service experiences and ineffective outcomes.

Imagining change

Two of those reasons are how policies are implemented and evaluated. Let’s start with policy evaluation because it shapes implementation. This isn’t inherently bad. Taxpayers, clients, policymakers and administrators alike all have an interest in knowing whether and how policies are working—or not.

            • Want to write for us? Take a look at Apolitical’s guide for contributors 

Policy success metrics can also helpfully inform the design of services and programs. Unfortunately, policy evaluation can have negative follow-on effects where the teams who implement can feel pressure to prioritise measuring service outputs over how people experience policies (service delivery) and the intended results. This can negatively shape program implementation, harming the public and frontline employees.

What could be done to call attention to evaluation’s follow-on effects the next time policy is being created at any level of government? How might stewardship of public funds be balanced with allowing new evaluative criteria for policy to emerge during implementation rather than all of its success metrics being predetermined?

Prototyping invites the lived experiences of the public and frontline staff into the design process

One way is to apply service design to the policy process. This approach aims to bring about measurable change through iterative learning. It includes “prototyping”: building and testing a mock artefact, service or space with the people whose needs we’re trying to meet through policy. Each round of testing leads to insights and a revised prototype. Prototyping policy is a particular approach to change, using small scale testing prior to scaling up.

Pilots and Prototypes

In government, when speaking of change efforts, we may be more used to talking about “pilots.” Pilots are also about creating change. While there are many types, the way some are implemented create helpful contrasts with prototyping. It’s understandable to feel prototyping is piloting by another name. But lumping them together risks losing the emphasis on iterative testing and building, which not all pilots share. Moreover, there is space to insert prototyping within a variety of pilots, even those associated with state and federal mandates.

One type of pilot is to prepare for changes because of mandates. These tend to flow from research and consultation by a group of people at high levels who often try their best to identify the problems, risks, and solutions ahead of implementation. They create a project plan with success metrics, and ask operations teams to implement. Once the pilot is done, the project team reflects on what did(n’t) work and attempts to adapt the rollout within the mandate’s constraints. In my experience, teams often feel their ability to optimise the service experience for their clients are limited at this stage.

In prototyping, project plans and success metrics may also be developed by those in higher positions. But the commitment of prototyping is to the problem rather than seeing a predetermined solution through. Prototyping allows for room to revise success metrics or even whole approaches as the project is unfolding, based on new data about people’s needs when interacting with the prototype. Prototyping invites the lived experiences of the public and frontline staff into the design process. They can influence what success means for a particular policy, how that policy could be improved, and whether it should be scrapped.

When should you prototype?

Prototyping may not be appropriate or feasible for every policy. In general, complex problems with high degrees of uncertainty are good candidates as are instruments for service delivery (websites, paper forms, lobby way-finding, etc). The use cases for prototyping below can even be inserted into mandated roll outs.

● Officials are deciding on a new policy or changing an existing one. Consider a facilitated two- to four-hour prototyping session where participants do a “premortem” by imagining the policy is a shocking failure and an amazing success . They list out the factors contributing to each future scenario. Finally, the group describes what steps they can take today to reduce the likelihood of failure and increase the chances of success. If appropriate and with their permission, invite local operations teams (including technologists and frontline employees) as well as clients and advocates into the same room for this activity. Doing so may widen options for policy teams, help with implementation and improve overall effectiveness. It may even suggest that the policy should not move forward at all or at this time.

● A commission is being planned to research solutions to an important issue in order to provide recommendations. Include within the commission’s tasks prototyping possible improvements. Certain recommendations could be mocked up, simulated or tested in a new part of the city, state or country based on what’s working elsewhere. The endeavour doesn’t need to be massive or run for a long period of time. Alternatively, if there is funding and an existing higher fidelity prototype, then the commission could test it through randomised control trials (RCTs). There are pro bono experts willing to help human services agencies in the U.S. do this. Commissions could also consider adapting RCT practices used in other public sector contexts for their specific purposes.

● A large-scale effort is being planned to create a database to better align government departments or programs. There may also be a new website or intranet as part of this project. Freely available templates and instructions, made by and for the public sector, can be used to prototype and test how the site would look and feel. Doing so would help improve usability by employees and the public by testing low to high fidelity versions of webpages (wireframes). There are also checklists for accessibility standards that the contracting team, supported by their internal IT colleagues, can use.

● Your department is about to roll out a large pilot. This process could be thought of as a series of prototypes where the project team is allowed to convene with frontline employees and the public on a frequent basis to understand their experiences as the pilot is unfolding. The stakeholders could devise their own performance metrics or prioritise the existing ones to provide an alternative view of what success means to them.

● A paper form, letter or brochure is being changed. Before it’s sent to clients or employees, someone on your team could test it with a handful of potential users. The goal is to see if they understand what is being asked of them, their next steps in the process, and where to turn with questions. There may not be time and resources for multiple rounds of testing and revising. Just doing this once with five or more different people could lead to valuable insights. Direct feedback about what is and isn’t working often produces better materials.

Equity-Centered Design

While not always an integral component to prototyping, equity is something policies should help create. Our approach begins by talking about “equity,” with project stakeholders. We define equity as recognising that we insert ourselves into a historical context characterised by power, privilege, discrimination and trauma.

Equity is about making things usable for everyone

Our team strives to understand how these forces play out in specific projects. Equity serves as a touchstone throughout our process. We refer to it in an attempt to lessen our contributions to the structural causes of disadvantage in whatever we co-create with employees and the public. Though not perfect, it prompts us to reflect with others on the implications of our decisions as projects are planned and carried out.

Equity is about making things usable for everyone. For example, knowing most non-English speaking participants in one of our programs speak Cantonese shapes our design decisions when serving them.

Mocking up for a better world

A focus on equity allows us to recognise when our own agency falls short of online accessibility standards, requiring us to work across several teams to prioritise it. Integrating equity in our practice is also valuable when colleagues from other counties come to us for help with a specific problem, and we co-create a more equitable strategy than what they were initially thinking.

Every attempt at change in human services agencies is animated by our values and our vision of a better world. Prototyping isn’t different in that way. It’s humbling to do this work, to admit that our understanding of a problem does not always align with the daily realities of the clients on our website or frontline employees in our lobby. And that’s OK.

Because the sooner these understandings can be mocked up, shown to others and tested, the quicker we can take a different course of action if necessary and test again until we achieve the intended results of the policy. Even then, we can continue making it better through prototyping and service design. — Marc K. Hébert.

A version of this article first appeared in the June 2019 edition of Policy & Practice magazine by the American Public Human Services Agency. Marc K. Hébert is the Director of the Innovation Office in the San Francisco Human Services Agency. You can find more about their work on Medium.

(Photo credit: Death to the stock photo)


Leave a Reply

to leave a comment.
Master the skills you need for the public service.

Discover inspiring resources, tools and policies.