Rethinking Evaluation - What is Wrong with Development Effectiveness?

TWEET THIS

Focusing on intended results could circumvent examination of unintended consequences
Attributing change to a single actor or intervention ignores many forces that could be at play
Distributional effects need to be assessed if we are serious about boosting shared prosperity

If development planners were to use complexity models to understand the web of interrelated processes to identify their objectives, intended and possible unintended effects would become clearer, and possibly increase evaluability.

Effectiveness is central to international development and its evaluation. The OECD/DAC Glossary of Terms defines development effectiveness as “the extent to which a given development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance.”

By itself, the DAC definition embodies the accountability dimension of evaluation. Complemented with an evaluative question of “why” objectives were achieved (or not), one gets to learning about the experience of trying to achieve a particular objective in a particular context.

The term embodies the fundamental concept that development assistance is measured against the yardstick that it sets for itself, because it is the development partners who decide on the objectives they aim to pursue. This notion is very different from assessment tools like benchmarking (a comparison with an agreed standard) or competition, where success is defined in comparison with others.

When viewed among these options, effectiveness seems rather lenient, given that the development partners define what success looks like. Nonetheless some development practitioners argue that effectiveness is too tough, and too rigid to account for adaptation during the life of the intervention (read our earlier blog – Rethinking Evaluation: Agility and Responsiveness are key to success). Others, mostly evaluators, argue that it is the practitioners’ risk aversion that makes them shy away from effectiveness as a measure of accountability, and has incentivized behaviors to “game the system”. In such scenario, objectives are written to get a good rating at the end rather than as the intended results that development partners try to achieve. There are good points to each of these arguments.

But, from my perspective, there are additional reasons why the way we look at development effectiveness needs a facelift!

With our increasing understanding of and ability to work with complexity there will be different demands on project planners and evaluators, as discussed in an earlier blog about relevance. This might change the nature in which objectives are set, which will either make it more challenging to assess whether they were met, or demand an equally dynamic evaluation tool, or both. It raises questions about the differentiation between effectiveness and impacts – something many practitioners have struggled with – and might call for merging these two criteria.

In addition, the way effectiveness has been defined has kept attention focused on intended results. Most evaluations grapple with getting evidence to determine whether objectives were achieved and to measure an intervention’s contributions. Fewer evaluations are able to collect evidence on effects outside the immediate results chain and identify unintended consequences. If development planners were to use complexity models to understand the web of interrelated processes to identify their objectives, intended and possible unintended effects would become clearer, and possibly increase evaluability. And even if planners do not use such tools, evaluators should explore how they can become part of defining program theory and evidence collection.

At the same time, complexity models make it clearer that attributing change to a single actor or intervention ignores that many forces are at play. The question of attribution has been at the heart of many a debate about the rigor and validity of evidence and whether it could prove one policy or action was better than another. A better understanding of complexity might help join up interventions of different development partners, and suggests that (in the long-term) evaluations have to be undertaken from a systemic point of view rather than focused on a single development agency or intervention.

Likewise, distributional effects of interventions, whether explicit part of the intended outcomes or not, need to be assessed if we are serious about goals like “no-one left behind” (proclaimed by the global community through the SDGs), or boosting shared prosperity, as one of the goals of the World Bank Group. Too little attention is paid to the assumptions we make about interventions that are not targeted and supposedly have no distribution effects. If the analysis of intended and unintended effects is differentiated by different stakeholder groups (rather than “beneficiaries” as one homogenous category), we can get a better understanding of the actual effects or impacts of interventions.

In short, the criterion “effectiveness” needs a facelift, not just for the purpose of addressing counter-productive behaviors. The spotlight that we evaluators shine has incentivized certain behaviors of decision-makers, program planners and implementers. Let’s do so intentionally, rethinking evaluation criteria and methods that incentivize behaviors for better development outcomes.