Search across all content
projectAnnie™ MOORE (Matching and Outcome Optimization for Refugee Empowerment) uses advanced machine learning and state-of-the-art integer optimization methods to recommend optimal placements of arriving refugees across hosting communities around the US.
This information was sourced from "Annie™ MOORE" by Observatory of Public Sector Innovation (OECD), https://oecd-opsi.org/innovations/annie/. Licensed under Creative Commons Attribution-ShareAlike 3.0 IGO (CC BY-SA 3.0 IGO).
There are over one million refugees around the world who, according to the UNHCR, are in urgent need of resettlement, i.e., permanent relocation to another county. In recent decades, the United States (US) has admitted the vast majority of resettled refugees. However, the US government faces a crucial task: to which communities should the refugees be placed when they arrive? The outcomes of refugees depend strongly on the first community into which they are placed. For decades, the US government has been collecting data on whether refugees were successful at obtaining employment, however, until now these data were not put to good use. Our pioneering software Annie™ MOORE (Matching and Outcome Optimization for Refugee Empowerment) uses advanced machine learning and state-of-the-art integer optimization methods to recommend optimal placements of arriving refugees across hosting communities around the US. HIAS, one of nine US refugee resettlement agencies, has been using Annie™ since May 2018. Annie’s objective is to suggest placements of refugees that would maximize their employment chances while simultaneously ensuring that the needs of the refugees (e.g., child support or language support) are met and the service capacities of hosting communities (e.g., housing or places in training programs) are not exceeded.
The primary beneficiaries of Annie™ are refugees resettled in the United States who now have access to better employment opportunities. The secondary beneficiaries of Annie™ are the residents of communities where the refugees are hosted. Annie™ ensures that communities only welcome refugees that are most likely to find employment there and whom they can help integrate. This reduces community tensions and builds goodwill toward refugees. The tertiary beneficiary is HIAS itself which have been able to streamline and their processes and, by extension, add value to the broader American refugee resettlement program. As more refugees are able to get into employment and become self-sufficient, the return on government funding for resettlement dramatically increases.
Annie™ only recommends the allocation of refugees across communities to HIAS. Therefore, HIAS staff have complete control over the final allocation and ultimate responsibility over where the refugees are placed. Nevertheless, we have found that HIAS staff have been extremely impressed with Annie’s ability to place refugees and rely strongly on Annie’s suggestions.
Annie™ is already perfectly adapted to the refugee resettlement context of the United States. Therefore, Annie™ can be adopted almost immediately by the other eight resettlement agencies contracted by the US government. The benefits of this adoption would spill over to all the other agencies: the richer and more complete data shared across agencies would allow Annie™ to more accurately predict employment likelihood for every agency. Finally, the interface and methods developed for Annie™ can also be easily translated into international contexts. For example, Annie™ could be used for the Syrian Vulnerable Persons Resettlement Schemes in the United Kingdom or for the allocation of recently arrived refugees in Sweden.
To understand how Annie™ works, one can consider two separate steps. First, they apply machine learning methods to find patterns in a large dataset covering all refugee placements by HIAS over the past decade. Their machine learning algorithm is able to pick out the characteristics of refugees that make them likely to get employment in particular communities. Therefore, the model is able to predict the likelihood of employment for newly arriving refugees in any of the communities where HIAS operates. In the second step, organisers decide which communities to allocate all refugee families to maximize overall employment. However, to do that, they need to keep track of different needs of refugees and the service capacities of the communities. In particular, they need to ensure that the total number of placed refugees does not exceed the government-approved annual quota. Moreover, the communities must be able to meet the needs of refugees that join them. For example, if all family members only speak Arabic, we ensure that the community has support staff who are Arabic speakers. There are many constraints of this kind and it is extremely difficult to keep track of them when allocation is done manually. The organisers' integer optimization methods, however, allow us to maximize employment which ensuring that all the needs and capacity constraints are met. Therefore, Annie™ is simultaneously able to improve employment outcomes and ensure that none of constraints are violated!
The inspiration for the research came from the National Resident Matching Program that matches tens of thousands of medical to their residency programs and from patient-organ donor matching programs around the world. These innovations were behind the Nobel Prize in Economics awarded to Al Roth and Lloyd Shapley in 2012.
Annie™ is the first-ever software that uses predictive and prescriptive analytics to optimize employment outcomes of resettled refugees. Annie™ is, therefore, a pioneering development in refugee resettlement. The project is innovative in three ways:
Annie™ was deployed by HIAS in May 2018. Previously, HIAS was using manual matching refugees to communities causing 1) information overload: failing to account for multiple attributes (language, nationality, wellness, family makeup) that may be key to successful integration; and 2) placement inefficiencies: case-by-case sequential assignments made with only a partial view of matching landscape are suboptimal. Annie™ analyzes support attributes and communities simultaneously to offer optimized placements through a user-friendly, interactive interface. Annie™ therefore completely eliminated HIAS's placement mismatch. Our backtesting suggests that Annie™ has been boosting employment outcomes by over 30 per cent. Our plans are to evaluate Annie™ with an RCT. The CEO of HIAS stated that Annie™ is at the forefront among the nine US resettlement agencies of using interactive, optimization-based technology to improve refugee integration outcomes, especially employment (see attached letter).
Annie™ was born out of a careful collaboration between academia, civil society, and the federal government. The refugee resettlement agency HIAS played a lead facilitative role in working with our five-member international academic team to develop both a Memorandum of Understanding (MOU) and a data-sharing agreement, as well as a willingness to house, deploy, and test Annie™ in order to improve the resettlement process. The MOU was approved by the leadership at the US Department of State (PRM).
US civil society organisation HIAS uses Annie™ weekly to resettle refugees. Talks are underway to expand to other US resettlement agencies; the Swedish government has also expressed interest. Refugees and host communities are key beneficiaries: Annie™ ensures that refugees are placed in communities where they are best suited to find employment, integrate, and become self-sufficient. Lastly, HIAS resettlement staff benefit by finding better community matches for more of their refugee clients.
During early testing in 2018-2019, the United States government cut refugee arrivals. Thus Annie™-recommended placements were low. Even so, our backtesting of placements with Annie™ indicates that we obtained around a 30% boost in the number of employed refugees. These gains were obtained using a counterfactual analysis of reallocating 2017 capacity by placing refugees in a way that maximizes the total expected number of employed refugees. While numbers are down, discussions are well underway to verify the performance of Annie™ using a randomized control trial. We can already verify that Annie™ has reduced--from around 20% to nearly zero--the number of families that are placed in communities that cannot provide proper support services. As Annie™ is ready for other resettlement contexts, and with greater numbers of refugee arrivals, we would only expect gains to increase as more capacity tends to result in better outcomes. Thus, the 30% employment gain is as a conservative baseline.
Challenges in developing Annie™ have included 1) an acute lack of resettlement agency budget, staff, time, and data; 2) a complex decision environment comprising host communities, refugees and the public-private, federal, non-profit, academic partnership; 3) volatile political climates and attitudes, domestic and abroad, generating crippling uncertainty in resettlement agency operations and planning, 4) lack of other relevant integration outcome measures such as education, physical and mental health, and English proficiency. Approval from the US Department of State through close relationships with HIAS has fortified our resolve to improve the robustness to external factors of the developed employment estimates. A potential future failure is the loss of graduate student support, principally for Narges Ahani – the leading architect of Annie™. We continue seeking financial avenues to support her continued work on this important initiative.
The support of key leadership at both the US Department of State (Barbara Day) and HIAS (Mark Hetfield and Mike Mitchell) were instrumental for the success of Annie™. Their collaboration provided the necessary infrastructure for successful data sharing agreements and understanding of the day-to-day operations which enhanced the prototyping of Annie™, leading to its eventual adoption. Karen Monken, HIAS's Arrivals Officer, uses and gives regular feedback on Annie™. Support from the US National Science Foundation (Operations Engineering) grant CMMI-1825348, UK Economic and Social Research Council grant ES/R007470/1, and Sweden's Jan Wallander and Tom Hedelius Foundation (Research Grant P2016-0126:1) and the Ragnar Söderberg Foundation (E8/13) have been vital to the continued development of Annie™. Finally, perseverance in uncertain times was certainly a factor in the success of Annie™, which was guided and motivated largely by the personal values of the research team.
There is great promise in replicating the successes of Annie™, most directly at other US resettlement agencies with nearly identical matching challenges. Annie™ could also improve placement in the Syrian Vulnerable Persons Resettlement Scheme operated by the British government. A recent report by the UK Independent Chief Inspector of Borders and Immigration recommended that the Home Office “improve the geographical matching process” of refugees in this Scheme. Annie™ has great potential for placement of asylum seekers. In Sweden, asylum seekers granted a residence permit need resettlement. While Sweden does not currently use sophisticated matching techniques, a recent report by the Swedish government recommends the adoption of carefully designed optimization and matching systems. Organisers generated interest after presenting Annie™ at the Swedish Ministry of Finance in 2019. Finally, related contexts such as matching in adoption and foster care may likewise benefit.
Organisers learned several important lessons while developing Annie™.
The hope is that by providing a semblance of order in the chaotic resettlement process, Annie™ can offer new hope and opportunity in the lives of many refugees. The placement recommendations that Annie™ produces integrate seamlessly with the domain expertise of resettlement staff, thereby helping to restore dignity to the vulnerable, marginalized, yet indomitable populations that Annie™ is serving.
Connect with 500,000+ public servants solving your hardest challenges.
Connect with 500,000+ public servants solving your hardest challenges.
Help public servants worldwide learn from your work, what worked, what flopped and what you'd do differently
Share your project
Log in or sign up to continue the conversation