This article is written by Greg Kato, compliance director, at City and County of San Francisco - Office of the Treasurer & Tax Collector
San Francisco’s Office of the Treasurer and Tax Collector is responsible for collecting billions of dollars annually for San Francisco through various taxes and fees. While most taxpayers pay their taxes in full and on time, some taxpayers do not. These accounts become delinquent, requiring collection efforts. Collections can be time-consuming and costly.
- Want to write for us? Take a look at Apolitical's guide for contributors
Our Office values making evidence-based decisions to use limited resources effectively. We have recently seen success by adding random controlled testing of messages with behavioural insights to our collections. In the process, we have come to see ourselves as “choice architects” who use smart design to nudge constituents to do the right thing.
What is randomised controlled testing?
Random controlled testing, also known as A/B testing, is when you identify a target population; randomly separate it into two groups (control and treatment); run the test, and then compare the results.
With a sufficiently large experimental group you can infer causation by randomising who receives the treatment versus the control, which ensures that both groups are similar. This approach means that you will be controlling for differences between the two groups, even those that you haven’t thought of. In this manner you can infer that the treatment is the cause of the difference in results between the groups (if any), and the change was not simply correlation.
Behavioural insights in practice
Recent psychological research has shown that small changes can have outsized effects on responses. San Francisco partnered with the Behavioural Insights Team (BIT) as part of the What Works Cities Initiative of Bloomberg Philanthropies to use behavioural insights to improve taxpayer correspondence.
We used the so-called EAST framework developed by BIT, which organises behavioural insight concepts into categories to help policymakers deploy them more easily. EAST stands for Easy, Attractive, Social, and Timely Paying taxes on time.
By conducting random control trials, we have found a way to improve our practices using an evidence-based approach
Every business in San Francisco is required to pay an annual registration tax. We wondered if a behavioural intervention could make more businesses pay on time, which would save both significant time and money for our department. The target population was the roughly 100,000 registered businesses in San Francisco.
We began by randomly splitting the population in half by adding an identification number to each taxpayer using the random number generator in Excel. We then sorted by the identifier and split the group in half. We now had randomised the intervention with about 50,000 businesses in treatment and 50,000 in the control group.
We then sent the treatment group their tax notice in an envelope that had a red box printed on the outside that read “Tax Notice Avoid Penalties Renew by May 31.” Our intuition was that the red box and text would be Attractive and Timely in the EAST framework by attracting attention and emphasising the deadline.
The control group received our standard envelope without the red box and text.
We waited until after the deadline, then compared the two groups. In the treatment group, 282 more taxpayers had paid on time compared to the control group, an increase of about 0.73%. Due to the size of the samples (50,000 taxpayers) the difference was statistically significant because the chance that the difference was due to a difference between the two groups rather than random chance was more than 95%. Since we randomised the samples and they were sufficiently large, we can state definitively that the treatment, the envelopes, made a difference.
In the experiment, San Francisco increased timely payments by about $30,000 and reduced our downstream collections. It cost about $4,000 to print 50,000 special envelopes. The experiment generated more than it cost by a factor of over 7 to 1 — quite a return on investment!
Become a choice architect
San Francisco’s experience with using behavioural insights has been positive. We saw statistically significant changes in taxpayer behaviour with experiments that were very cheap to implement. While these approaches are not silver bullets, by conducting random control trials, we have found a way to improve our practices using an evidence-based approach, rather than hunches or guesses.
Thank you for taking the time to learn more about San Francisco’s experience with randomised controlled trials and behavioural insights. I encourage you to become a “choice architect” by incorporating evidence-based decision-making into their work. — Greg Kato
See more Behavioural experiments here.
(Picture credit: Unsplash)
Log in or sign up to continue the conversation