Rethinking Evaluation— Is Relevance Still Relevant?

TWEET THIS

Meeting the bar for relevance might be irrelevant in today’s world of complexity
Tools like network analysis can help us reach more stakeholders, anticipate amplifiers of success
Must capture parameters outside linear project logic essential for success of an intervention

As evaluators we need to shed light on whether an intervention’s focus is on nodes in the network that matter, that can have large multiplier effects, or that are peripheral to the desired solution. That is a lot more than 'relevance'.

In last week's #WhatWorks post, I argued that it was perhaps time for us in the evaluation community to rethink our evaluation criteria. After nearly 15 years of applying relevance, effectiveness, efficiency, impact, and sustainability as our foundational evaluation criteria, is now the time to change or adapt?

The evaluation criterion “relevance” has troubled me for quite some time. In many development settings, a project is considered relevant when “the aid activity is suited to the priorities and policies of the target group, recipient and donor.” Of course, this is important. In plain language, it makes us question whether the intervention aimed to address real needs.

But that is exactly where the challenge lies: the needs of whom?

In an ideal world, the needs of the target population are aligned within the community, with the government’s priorities, and the policies of donors. In reality, such a theory makes a large number of assumptions. For instance, that the target community is homogenous, which it often is not. Nor are priorities at central and decentralized levels identical, be it for a real difference in needs or for political reasons.

In practice, evaluators often use policies of governments, donors, and aid agencies to assess whether an intervention is relevant in that context. More often than not, these policies are written in ways that can justify a whole slew of different activities. Hence, meeting the bar for relevance is not all that hard.

In addition, I would argue, this criterion might be irrelevant in today’s world of complexity.

Look at network analyses that map out situational problems and how they are interlinked. The TED Talk by Eric Berlow illustrates in less than 4 minutes how complexity theory and technology allows us to map and understand development challenges in completely new ways. Being a visual person, I am fascinated by the modeling capacity that technology now provides.

More importantly, techniques like these could change the process through which we seek and find solutions to development challenges. It provides us with an opportunity to live up to the values of a more inclusive world, where the voices and perspectives of a much broader group of people matters in defining goals, solutions, and pathways that will get us there. This modeling capacity could help bring together the views of a broader set of stakeholders, add perspectives to understanding a particular development challenge and interrelated factors and come up with different solutions than, say, a group of experts might see from the vantage point of their technical expertise.

And, an approach like this can help anticipate potential amplifiers of success, or what we used to call “killer assumptions” that are strong predictors of failure or diminished development outcomes. These assumptions are often embedded in project or policy design without recognizing them.

Impractical? Watch the video and look at the model the US military had developed for the situation in Afghanistan. Berlow maps all of these factors into an interactive model and then identifies nodes that have much larger ripple effects throughout the system than others.

What does all of this have to do with the simple evaluation criterion called relevance?

If we apply relevance to a more complex reality in the same way we have used up to now, with the policy context as the yardstick to assess relevance, any intervention will meet the criterion as long as it falls anywhere in the network of interrelated factors.

But that is not important for decision-makers! Instead, as evaluators we need to shed light on whether an intervention’s focus is on nodes in the network that matter, that can have large multiplier effects, or that are peripheral to the desired solution. That is a lot more than “relevance”.

Instead, I suggest that we fundamentally rethink the “relevance” criterion and replace it with something that helps assess whether:

Diverse perspectives were taken into account in identifying and implementing solutions, namely the networked analysis of the development challenge captures parameters that are outside a linear project logic that are essential for success or failure of the intervention;
Development interventions address key entry points – the significant nodes that are bottlenecks or opportunities for multiplier effects – in a networked analysis of the development challenge at hand; and
There are synergies across – or joining up of – a multitude of interventions aimed at the same development challenge.

Doable? Add your thoughts on what it would take.

The Rethinking Evaluation series is dedicated to unpacking and debating evaluation criteria by which we judge success and failure, and whether they are fit for the future. Stay tuned and contribute your views.