In a community centre in West Virginia, Beyoncé blares through the speakers, playing as the supporting act alongside trained educators using music to teach at-risk teens about sex, STDs and healthy relationships.

The Teaching Health Instead of Nagging Kids (THINK) program’s “Love Notes” method – a media-based skill-building workshop – is unusual for using popular music as a way to teach sex-ed. But it also stands out because, after undergoing rigorous academic evaluations, it is scientifically proven to work.

Without enough resources, proper oversight and continuous evaluations of the delivery of programs, putting evidence into practice can be a challenge, no matter the quality of information available. Here are four key points to bear in mind when implementing policy on the basis of evidence.

1. Decide on a framework, and stick to it

When implementing any program, Gary VanLandingahm, a professor at Florida State University School of Public Policy, said that policymakers should ask the questions: “What are we doing now? And why is this [new program] the way to go?”

Once policymakers have reviewed the entire process and have the motivation to see it through, implementation should be done one step at a time.

“Look at case studies and review step by step processes. Get the evidence, evaluate it, then design a pilot to run by stakeholders. Ask stakeholders questions to help establish or build on different stages of implementation,” said VanLandingahm.

Following implementation guides based on evidence analysis will determine if a program is effective, he added, while cutting corners or not following frameworks will lead to failure.

For example, rather than putting teens referred by the justice system into juvenile detention centres to learn better behaviour, the Washington State Functional Family Therapy (FFT) program garnered better results by providing intensive family therapy at home.

Half of the states that implemented the program, including Washington State, saw a vast improvement in behavioural outcomes from children in FFT while the other half of states using the program experienced recidivism.

VanLandingahm attributed the unsuccessful results to a failure by officials to follow technical frameworks and the FFT’s official protocols after reviewing the outcomes.

2. Constantly review progress

According to a research assessment from Pew-MacArthur Results First Initiative — an organisation that works with US states and counties to implement evidence-based policies — policymakers should establish a set of standards, build and maintain capacity for ongoing improvements, balance program responsibilities with local needs and conduct data-driven reviews in order to institute proper oversight.

This approach is a comprehensive step by step process to ensure that policies are effectively implemented and reviewed years later to assess if goals are being met.

The state of Maryland, for example, continuously evaluates programs by reviewing set indicators through bi-weekly agency data. Governors, agency heads and other staff members then meet to discuss performance and refine goals. Regularly reporting target benchmarks to policymakers can lead to better decision-making and show the value of evidence-based programs.

Policymakers should also ensure that monitoring and evaluation activities are built into budgets because those functions are most likely to be cut first if funds are reduced, according to the assessment. This can be achieved by working with budget offices and conducting a cost-benefit analysis to show how evidence-based programs can save on costs.

3. Be ready to adapt to the context

But it may not always be easy to thoroughly implement an evidence-based policy by following the guidelines. States, cities and communities often have different budgetary requirements and location- and population-specific demands.

Although budgetary constraints can be problematic, Adam Obley, a Clinical Epidemiologist at the Center for Evidence-based Policy, said that small tweaks to fit the location or manage resource constraints may be necessary.

For example, if a program model was based on a clinical trial that used nurses, but the area implementing the same model doesn’t have a big nursing staff, they might swap it for what they do have, such as physician assistants.

It’s critical that all partners coordinate and review framework manuals to make adaptations that fit with local conditions and cultural differences to ensure that quality isn’t compromised.

For example, the Oregon “Wraparound” monitoring service, overseen by the Oregon Department of Human Services, has ten basic principles that program implementation should follow, but allows for flexibility to ensure that community needs are considered in the program design.

Public servants should hold regular meetings to review practices and to get feedback from local organisations and citizens to assess the cultural fit of an evidence-based program.

“The evidence is where we begin, but we’re [policymakers] pragmatic enough that we recognise in day to day policymaking that your fidelity to the evidence can’t be perfect,” said Obley.

4. Ensure proper evaluation

VanLandingahm said often public servants rely on anecdotal stories to prove that a policy is working. Evaluating a program or policy at a set date will help determine what is or isn’t working.

There are three types of evaluations: policy content, policy implementation and policy impact, according to the American Centers for Disease Control and Prevention (CDC). Each type of evaluation can help assess the barriers, short and long term impacts, and the overall effectiveness. The process should begin with assessing if a policy or program is meeting its intended goal.

The US department of Health and Human Services considered THINK to be a successful program based on a set of performance metrics. All grantees are expected to collect performance data for reporting annually. For THINK, the performance metrics included the number of youth served, program dosage, implementation quality and grantees' progress on forming partnerships.

Requiring evaluations as a condition for funding for more high-risk projects that have a smaller evidence base can help to ensure that a project that isn’t achieving its goal is changed or adapted. Funding contingencies will also guarantee that pilot programs will be properly evaluated for their impact.

As more policymakers choose to use evidence in their decision-making, ensuring that there are enough resources, oversight and evaluation metrics in place will be paramount to the success of evidence-based policies. – Amelia Axelsen

(Picture credit: Pixabay)