How to plan an experiment

This article is written by Susan Broomhall, Behaviour Specialist at New South Wales Fire and Rescue. It builds on her interview in the last section, and runs through the process of designing a behavioural insights experiment — and winning buy-in for it.

Behavioural insights are used to understand how people behave and why they behave that way.  The application of these insights is often used in social policy, program design and development, and messaging.  This understanding of the insights is then predominantly used to enable behaviour change or behaviour modification — some form of ‘intervention’.

Despite debate about the legitimacy of behavioural science as a “science” (See: The “Is Psychology a Science?” Debate), and attempts to denigrate the behavioural sciences as “soft” rather than “hard”, (For example: How Hard is Hard Science, How Soft is Soft Science?) the fact remains that proper behavioural science undertakes the same concepts of research methodology as all the sciences — or at least it should.

Increasingly, governments and other decision-makers are coming to understand that behavioural science is not an abstract fantasy. Instead, it enables substantial, evidence-based, human-centred solutions development that can be applied to all sorts of “people” problems and is critical to outcomes success. Anything that involves humans should be designed with an understanding of how human behaviour applies to it.

While the use and value of behavioural science is gaining traction, for the general layperson, behavioural science and its application are not well understood.  Avoiding jargon, simplifying language, showing how it applies in the everyday, and bringing people along on the journey are all integral to encourage decision makers to embed the use of behavioural science as a standard first step in the decision making process.

To “prove” the value of behavioural science and get buy in, the insights developed need to be of quality, understandable, applicable, resonate with the issue being addressed, and be tested for efficacy.

Optimising our capacity to make the best use of our understanding of why and how people behave the way they do comes down to the strength of the evidence that has been collected to develop those insights.

What we’re talking about here is basic scientific research methodology.

So, how do you develop rigorous and robust behavioural science research methodology to elicit quality behavioural insight results that get people to take notice?

Understand what the issue or problem is that you are seeking to resolve or change.

If you’ve been asked to provide behavioural insights, ask “why? What are you trying to understand? What are you trying to change?”

Ask ‘why’ of everything even if you think you know the answer, verify and validate the direction by constantly asking ‘why’.

Unpick the issue from a psychological and social science perspective and consider and define what your ‘ideal’ or preferred behaviour would look like. Then design the path to your ideal via your research.

From your initial groundwork in understanding the issue, develop hypotheses to test.

A critical point in any research methodology is the formation and testing of hypotheses as this gives the research direction and focus.  Writing a hypothesis is a creative thinking exercise and the more considered your approach, the more valuable the results

Take a look at this resource to help you.

Don’t be afraid of developing a few hypotheses to test and, more importantly, don’t be afraid of proving your hypothesis wrong!  In science, there are no “right” and “wrong” results, just results.

Disproving your hypotheses allows you to revisit your methodology and thinking, which may allow you to eliminate the hypothesis as an option and thereby refocusing on other options to consider.

You can develop your hypotheses using literature reviews, empirical research, observational research, or simply asking people questions about what they think.

Science Buddies notes a simple process of “If ____ [I do this] ______, then ______ [this] will happen”.

Adhere to core scientific research principles.

Consider what your research methodology will look like, why you are collecting information, and how that information will develop an improved understanding of what you are analysing.

Make sure you have your basics covered – understand the difference between correlation and causation, ensure your sample size is robust, apply the appropriate control measures to compare results to, and make sure you document everything so that it’s as replicable as possible.

After having designed and implemented a community-based intervention program to improve smoke alarm penetration rates in households, we sought to test the efficacy of that intervention.

We were questioning “was the intervention any different to not doing anything at all?” Using fire incident data overlaid with lifestyle segmentation data, we developed a Fire Injury Risk Model that was used to develop a pilot program looking to reach households that had a higher propensity for fire incidents.

It was believed that having firefighters visit people in their homes and discuss the importance of smoke alarms, while also ensuring the home had a working smoke alarm, would increas, not only the smoke alarm penetration rate but also households understanding of the importance of maintaining a working smoke alarm.

Using a cluster randomized controlled trial to determine the effects of intervention of battery and hardwired smoke alarms in New South Wales, Australia: Home fire safety checks pilot program we surveyed three cohorts — one that had been visited by firefighters, one that had home safety material mailed to them, and one that had no intervention at all.

The research not only found that intervention resulted in the increased use of smoke alarms but additional insights to the issue and how best to address it.

Making an experiment replicable encourages others to look at your research and test your results which continues to generate conversation on your insights and only improves our collective capacity for understanding.

Use the skills and knowledge of properly qualified social scientists.

Avoid one-off opinion poll type questionnaires that have been popularised thanks to the growing availability of simple survey tools.  Access to simplified tools has led to an increase in pseudoscience speak commonly used in media and marketing which lead to undermining your science and capacity to showcase evidence-based results.

Governments across the globe, (such as  the Behavioural Economics Team of the Australian Government and the New South Wales Premier & Cabinet Behavioural Insights)  are embracing the value of behavioural science application and provide resources on their websites to assist and guide you.

When looking for agency assistance, look for those that specialise in behavioural economics such as The Behavioural Architects.

Customise your methodology according to the issue you are looking at.

Unless you are looking to replicate someone else’s experiment, generalising other people’s work and retrofitting results to your issue is more often ‘miss’ than ‘hit’.

Use literature reviews to see what others have done in the space then customise and design out your hypothesis according to your specific context.  Look to develop bespoke behavioural insights as these will be more effective in addressing your issue than more generalised, high-level findings.

Be creative rather than prescriptive in your research design. Remember, it’s important to understand that each behavioural science experiment may be different/unique/bespoke/customised to the context you are exploring.

For example, while unattended kitchen fires are a significant issue for many Western countries, messaging regarding prevention and response are often generalised (see, for example, How to Put Out Kitchen Fires)

In researching how kitchen fires impact different people in different ways (while working on “Keep Looking When Cooking”) it was found that messaging, for example,  advising covering a flaming pot with a lid led to a higher and more consequential fire injury in the elderly.

Researching this occurrence led to the understanding that the elderly are less agile in their movements, increasing their chances of fire exposure to the skin, which is thinner and frailer than younger people.

While the issue (of a kitchen fire) may be similar for each person, the messaging to the elderly regarding response should be different in that they need to move away from the fire and call for emergency assistance immediately.

Use as many different research methods as you can to triangulate your results. There are plenty to choose from – correlational research, data and content analysis, literature reviews, surveys, focus group testing, empirical research, to name a few.

Be collaborative and inclusive.

While you lead the design and development of robust research methodology, ensure you bring in different perspectives and skill sets to ensure you understand the concerns and issues in relation to all stakeholders.  This helps to address how the results will impact different people in different ways.

When working on our unattended kitchen fires research, the data and evidence clearly showed that, indeed, we had a problem with people not watching their stovetops.

However, looking broader and talking to those that interact with people in their homes uncovered information leading to the understanding that different cultures have different cooking practices.

Surprisingly, we found certain cultures use gas cylinders stored in laundrys to cook food at higher temperatures than are available on a stovetop, for example.  Clearly these insights led to a different message being promoted to those cultures regarding safe cooking practices.

Understand the difference between “reporting” results and “analysing” results.

Reading an excel spreadsheet and relaying what the numbers say is “reporting”.  Analysis is the careful consideration of various data and then synthesising that into information.  It’s the analysis of the results that create the insights – basic statistics are not insights.

Ensuring your research team has data analytics capacity is important to developing metrics to measure the effect of any intervention.  It’s data analysts that help with developing insights to your research.

Insights should lead to capacity to implement some form of intervention and change in direction to some issue.  Ensure you create datasets, not only to test your hypotheses but also, to track intervention efficacy.  How will you know that your insights are of any value if you cannot track and monitor the application of those insights?  Meaningful information and data analytics are critical.

Ensure your behavioural insights are understandable to the layperson.

Brilliant scientific results are useless if no one understands what you’re talking about or how to use them.

Synthesise your results so they are digestible to everyone in your stakeholder group and they clearly see what these mean and how these can be used.

Embed iterative evaluation into the process to ensure you are tracking as expected.  Once you’ve developed insights and proposed an intervention, tracking how the implementation of your insights is going will prove or disprove the efficacy of the intervention.

Revisit why something is or isn’t working as you’d thought.  Explore whether this relates to the insights or the application of those insights.

So, how do you optimise behavioural insights?  Show everyone how these insights improve their life.

Enjoy!

Key takeaways:

• Use literature reviews and other existing research, but your experiment must be designed according to your specific scenario and problem.
• Make sure your process is scientifically robust — but your results are communicable to the layperson
• Evaluate as you go so that you can iterate and improve