• **Problem: **Choosing the right approach to evaluation can be difficult to navigate.
  • **Why it matters: **Which evaluation lens we choose to use will determine how we view a problem, who we choose to help us solve it and the kind of action we will take.
  • Solution: By learning the best ways to evaluate these initiatives, public servants can make better decisions and enhance their skills in evaluating, leading to improved results.

The problems public servants face today, in regards to complexity, turbulence and scale are different to what they used to be. Initiatives aimed at improving gender equality and equity may, for example, span several years and involve multifaceted strategies. They are also deeply rooted in social, cultural and systemic dynamics.

Some traditional evaluation methods are outdated and unable to capture the nuances of the current political landscape. When we think about evaluating change, or assessing impact, we increasingly see the need to understand the human stories beyond the numbers. Where more conventional approaches support improvement, summative testing and accountability, the approaches discussed in this article support the development of innovation and adaptation in dynamic environments, focussed predominantly on human experience.

Choosing an evaluation approach

When we talk about evaluation, it’s common to hear grand terms like 'gold standard'. However, the reality is usually very different. Choosing the best evaluation approach for your purposes is always dependent on context — what works in one place might not in another. Whether you are conducting the evaluation yourself, or contracting externally, it's important to understand that the best approach is one that fits with your specific goals, resources and cultural context.

The evaluation approach you choose will be the overarching lens through which a programme or intervention is assessed. The approach that you choose will influence how you work with stakeholders, how you assess outcomes and how likely your evaluation report is to meaningfully impact policy. It's a process that warrants consideration and an exploration of how different evaluation methods sit within wider society. The approach you use is also likely to impact the specific data collection methods you use to conduct your evaluation.

Public servants face unique and complex challenges, and evaluating initiatives aimed at fostering a fairer, more equitable society can be a daunting task. The evaluation approaches under discussion in this article aim to address these complexities. Selected for their adaptability, inclusivity, and emphasis on comprehending the mechanisms of change, these methodologies empower the person applying them to illustrate the impact of any given initiative.

Participatory Evaluation

Participatory approaches move from simply engaging with stakeholders, to them being active participants in the process. The core objectives of participatory evaluation are twofold: to nurture inclusivity and to engender transparency, with a particular emphasis on empowering marginalised communities, making sure every voice holds significance. By weaving these principles into your chosen evaluation approach, you not only enrich the depth of your insights but also ensure that your processes are characterised by a commitment to inclusivity, transparency and empowerment.

Please see our ‘Data Collection Methods’ article for more information on practical methods that support participatory approaches. If you’d like to learn more about participatory evaluation, we love this guidance provided by Health Nexus.

Realist Evaluation

Realist evaluation is an approach that goes beyond determining whether an initiative works or not. It delves into understanding what works for whom, in what contexts, in what respects and how.

This involves collecting more than just basic data on the results or how the programme was carried out. It's about really understanding the nuances that affect the programme and pinpointing exactly what is causing changes to happen and in what circumstances. It's a way to fully understand the complexity of a programme’s impact.

Realist evaluation equips the evaluator with the ability to understand the underlying mechanisms driving change or hindering progress. By dissecting these, alongside the contextual factors that influence a program's outcome, public servants can identify and address the root causes of inequalities and adverse power dynamics, and create policies that are more inclusive and that actively work to reduce disparities.

Some key characteristics of Realist Evaluation include:

  • **Context: **Realist evaluation starts with a thorough analysis of the social, cultural and environmental factors that influence programme implementation and outcomes. This step is crucial in understanding the landscape in which a policy will operate, thereby helping to foresee potential challenges and opportunities.
  • Mechanism: This aspect involves identifying the underlying processes, behaviours, or responses that are triggered by the programme. By understanding these mechanisms, public servants can design initiatives that leverage positive responses and mitigate negative ones, fostering environments where equity can thrive.
  • Outcome: Evaluators scrutinise both the intended and unintended consequences of a programme, offering a holistic view of its impact. This comprehensive understanding aids in refining policies to ensure they are achieving the desired outcomes and not inadvertently perpetuating inequalities.
  • Context-Mechanism-Outcome (CMO) configurations: These serve as detailed explanations that connect context, mechanisms and outcomes, providing a roadmap to understanding how and why a programme works. By leveraging these configurations, public servants can develop a deep understanding of the dynamics at play.

If you’d like to learn more, we love the guidance provided by Better Evaluation. We also like this case study example from ITAD.

Feminist Evaluation

At its core, Feminist evaluation centres on gender analysis, power dynamics and inclusivity. It considers the perspectives and experiences of marginalised groups, through a gender-sensitive lens, ensuring that the evaluation process itself is equitable. This approach aims to dissect existing gender inequalities that foster social injustice, thereby paving the way for policies that are grounded in equity.

Feminist evaluation acknowledges that evaluation is an inherently political activity, encouraging the evaluator to become an activist — notably by recognising the power they have in facilitating spaces for people to share experiences, knowledge sharing and leveraging this knowledge for social change.

What sets feminist evaluation apart is its emphasis on scrutinising not just the outcomes but also the processes and methodologies employed in evaluations. It advocates for a process that is fair and promotes equality from inception, thereby ensuring that the path to the end goal is as equitable as the goal itself.

Unlike other evaluation approaches, feminist evaluation doesn’t advocate a specific methodology. Instead, it’s often viewed as a way of looking at evaluation. To understand how feminist evaluation can be a powerful tool, let's break down some its key characteristics:

  • Participation: Similarly to participatory evaluation, feminist evaluation emphasises the involvement of marginalised and underrepresented groups in the evaluation process. It values their perspectives and experiences. In theory, this leads to policies that are not just for the people that are impacted, but are shaped by them too.
  • **Qualitative Data: **Quantitative data can tell us a lot, but it can't tell us everything. Feminist evaluation encourages us to listen to people's stories, to understand their experiences deeply, and to use this rich information to shape policies that are effective and empathetic.
  • Contextual Analysis: Feminist evaluation recognizes that social and cultural contexts matter. It urges us to consider the social and cultural backgrounds of the people affected by policies, ensuring that the strategies we develop are grounded in the real-world contexts of diverse groups.
  • **Acknowledging difference: **It promotes a nuanced understanding of women’s experiences, urging evaluators to steer clear of homogenising women as a category and to appreciate and value the diverse experiences and perspectives they bring to the table.
  • **Transformative Goals: **Feminist evaluation is not just about assessment; it is about fostering transformative change. It challenges and seeks to alter the structures and systems that perpetuate gender inequality, aiming for a future that is grounded in equity and justice.

The terms ‘Feminist’ and ‘Evaluation’ can sometimes lead to presumptions; you’d need to be a feminist to use this approach, it's only about women's issues or it's too focused on gender. Therefore, If you are considering using this approach, you might want to avoid using labels. Instead, explain clearly which parts of the Feminist evaluation approach you're using and why it works for the initiative you’re evaluating. This way, people can see the real benefits of the approach without getting caught up in the name.

If you’d like to learn more, we love the guidance provided by Actionaid. We also love this case study example, from Global Affairs Canada.

Developmental Evaluation**

Developmental Evaluation (DE) is not a progressive approach developed to evaluate complex programmes. Where other approaches focus on measuring outcomes, DE focuses on providing real-time feedback and making adaptations in response to changing circumstances.

This approach is particularly suited for innovation development, taking place in complex, dynamic, and rapidly changing environments and where the goals and objectives may not be clear from the outset. In today's fast-paced world, public servants face complex problems that demand flexibility and adaptability. This makes DE a valuable tool for public servants navigating new projects, programmes, products, organisational changes, policy reforms and system interventions.

Some key characteristics of DE include:

  • **Stakeholder engagement: **DE emphasises stakeholder involvement, ensuring diverse perspectives and voices are included in decision-making processes and ultimately more inclusive and equitable programmes. Together, stakeholders and evaluators define the goals and intended outcomes of, acknowledging that these may evolve, or adapt, with time. See our free stakeholder engagement coursefor more information.
  • **Embedded, not independent: **DE involves an embedded evaluator within the project team, departing from the traditional independent evaluator model. Unlike conventional evaluations, DE encourages rigorous and transparent reasoning from data to value judgments, including negative findings. In this sense, DE is a courageous approach — one that is dependent on the evaluator holding difficult and transparent conversations.
  • **Continuous data collection and feedback: **DE relies on continuous data collection, emphasising qualitative data and real-time feedback. Regular meetings and feedback sessions with stakeholders also help inform decision-making. These discussions are not limited to predetermined metrics but encompass the broader context and emerging insights.
  • **Real-time course correction: **As discussed, evaluators use the insights gained from DE to make adaptive decisions. This ability to make agile course corrections as issues arise, rather than waiting for the end of a program or policy cycle can prevent costly mistakes and improve program effectiveness. The iterative nature of DE is what sets it apart from more traditional evaluation approaches, but as it’s not what people are used to, it can also be uncomfortable for those commissioning it.

If you’d like to learn more about DE works in practice, we love this resource from the McConnell Foundation and this resource from USAID.

Public servants need to be ready to deal with problems as they arise and make the most out of each opportunity. Dedicating time to understanding monitoring and evaluation approaches means we can significantly improve our chances of identifying and mitigating risks, use our available resources more effectively, make changes when we need to, and confidently stop doing things when they don’t work.

Modern problems require modern solutions. The approaches described here are not just theoretical concepts, they’re practical toolkits that can be used to help public servants understand, adapt, and make informed and effective policy decisions. It’s important to note that these methods aren’t mutually exclusive, in fact, they can complement each other to provide a more comprehensive understanding of the complex issues public servants encounter in their daily work.

👋 We would love to know what you think! Leave us a comment below to share your thoughts.

(Image Credit: Unsplash)