TWEET THIS:

Evaluators must ask if DAC criteria are inclusive enough and respect under-privileged groups.
Evaluation must take into account new thinking that questions escalating consumption patterns.
Need to develop evaluation models that capture complexity to inform policy.

Have we reached a Copernican moment where we realize the 'earth isn’t flat', and our definitions and 'understanding of the world' need to be reset? Leaving aside jargon and methodological challenges, there are other good reasons to revisit the evaluation criteria we use.

Over the past 30 years, evaluation in the development field has gone through multiple cycles of questioning which method is better than another. But few in the development circles in which I have operated, have questioned the standard evaluation criteria that we use.

Many development institutions, including the World Bank, regional development banks, the UN, and bilateral aid agencies subscribe to what has come to be known as the DAC evaluation criteria. Specifically, there are five criteria – relevance, effectiveness, efficiency, impact, and sustainability; in short R/E/E/I/S – that underpin most evaluation systems in international development.  

Evaluation questions get framed around these criteria, and reports get written up using this language. But, many an evaluation struggles to implement these criteria in sincerity. Others are accused of using too much jargon as they report faithfully on these criteria. And often, the evaluations tend to leave readers with unanswered questions.

After nearly 15 years of adhering to the DAC evaluation criteria, is it time for a rethink? Have we reached a Copernican moment where we realize the “earth isn’t flat”, and our definitions and “understanding of the world” need to be reset? Leaving aside jargon and methodological challenges, there are other good reasons to revisit the evaluation criteria we use.

Values

As our societies develop, norms and values shift. While the evaluation criteria appear to be neutral and should be applied as such, they were formed by a set of values. The post-2015 agenda has declared its intention to be more inclusive, respecting under-privileged groups of people, which means we as evaluators need to reflect whether the criteria represent such diverse views. Being able to shape norms that are more inclusive of diversity rather than judge everyone through more limiting norms will be a necessity if 2030 is to become the world we want.

End Game

The adoption of the Sustainable Development Goals (SDGs) signal that we need to shift our understanding of development outcomes. Our development and economic models are premised on ever-increasing consumption. By contrast, the SDGs recognize that such consumption levels are unsustainable from an environmental, economic, and social point of view. This new commitment should lead to a paradigm shift around desirable development pathways that are not premised on escalating consumption patterns. Evaluation tools to unpack intrinsic impacts on consumption patterns will be needed to determine whether the world is evolving in desired ways.

Complexity

The world has become more complex, or rather: our ability to accept and understand complexity has increased. International development has relied on often linear and simplified logical frameworks or results chains that string inputs-activities-outputs-outcomes-impacts into a straight causal path. Development practitioners as much as evaluators know that development processes do not follow such linear assumptions. Instead, one action might cause a number of reactions that have effects in rather diverse ways. Hence,we need to develop evaluation models that capture the effects of complexity to inform policy-makers and practitioners about the actual effects of choices they make and actions they take (see excellent book on this topic by Jos Vaessen et al).

Technology 

The pace at which technology develops and influences lives has far-reaching effects on societies. Solutions to complex problems can be generated in un-thought of ways and often through unconventional networks of people. Information travels, is demanded, and influences large groups of people at a much faster and inter-connected pace than ever before. We are faced with an avalanche of data, a dearth of facts, and an ease of spreading (mis)information that has been unprecedented.  Evaluation can benefit from technology, be it to construct with greater ease models that reflect theories of change, help with data collection and processing, or sharing evaluation evidence with a much wider audience than before. But, it does so in an environment of multitudes of realities that may or may not lead to evidence-based decision-making, especially if a “post-fact” era were inevitable. 

Cost & Benefits 

Current considerations of efficiency, cost savings, or cost-benefit analyses are challenged to take long-term impacts into account. Something that appears efficient today, might have inadvertent devastating long-effects on natural resources or the social capital of communities. Likewise, the distribution of cost and benefits have been uneven, as witnessed by those who bear the brunt of eroded natural resources, or of development outcomes that benefit some groups in society and not others.

Do these issues really necessitate a Copernican shift in the evaluation field that would require questioning the established five evaluation criteria? Are the criteria so inflexible that they can’t be adapted as they are to address these challenges?  Does this even matter for anyone else, other than the nerdy evaluators and their jargon-filled reports?

I say yes to all three questions. And particularly so, in a world that lives by the mantra “what gets measured, gets done”.

The Rethinking Evaluation series is dedicated to unpacking and debating evaluation criteria by which we judge success and failure, and whether they are fit for the future. Stay tuned and contribute your views.

Read other #Whatworks posts in this series, Rethinking Evaluation:

Is Relevance Still Relevant?, Agility and Responsiveness are Key to Success, Efficiency, Efficiency, Efficiency, What is Wrong with Development Effectiveness?,  Assessing Design Quality, Impact: The Reason to Exist, and Sustaining a Focus on Sustainability