When I became Director of Evaluation at the Norwegian Development Cooperation Agency (Norad) in 2011, my natural instinct was to look for the evidence of impact of Norwegian aid. What did the money achieve that Norwegian citizens were investing in far-away countries? 

I sat down one afternoon and went through all of the evaluation reports published by our Evaluation Department over the previous year, and to my disappointment had to conclude that none of the evaluations and studies commissioned by the Eval­uation Department and finalized in 2011 could report sufficiently on results at the level of outcomes or impact.

The report showed well what money was being spent and what direct activities or services were being delivered. But critical questions about whether those services gave rise to real benefits for poor people and other target groups proved elusive.  As it turned out this was not a very popular insight. Why?

First, while it was clear to me that “no evidence of effect” does not mean “there was no effect” – equating these two very different statements was a misunderstanding common in the press, among colleagues and the general public.

This was not a harmless misunderstanding - there was substantiated fear that it would be used to support political voices that wanted to cut Norway’s generous aid. Others found my statement difficult to digest because it implied the necessity of seeing an effect of Norwegian Aid – when they felt measuring results was irrelevant, perceiving aid as merely a foreign policy tool.  And finally, it was simply human to resist criticism and scrutiny – and my statement indeed suggested that we needed to do a better job in ensuring the evaluability and monitoring of the programs funded. 

Out of this realization, we decided to carry out the Evaluation Department’s first ‘corporate evaluation’.  The resulting report titled ‘Can we demonstrate the difference that Norwegian aid makes? (1) was very successfully launched last week in Norway and presented at the World Bank Group this week, hosted by the Independent Evaluation Group (IEG). Vibecke Dixon, Senior Advisor and Manager of the evaluation and Ida Lindkvist, an advisor in the Evaluation Department, explained how the evaluation had been framed around a series of testable hypotheses: were the arrangements for planning results in grants adequately designed and specified; were staff adequately trained to manage for results in grant management; were policies and systems correctly implemented when grants were approved?

The study also checked to see if problems were arising in the way the evaluation department designed and managed evaluations: did it ensure evaluation designs placed an appropriate emphasis on measuring results; and were the consultants recruited sufficiently competent?

Organizational culture and incentives - the silver bullet?

Limitations were found in all of the above-mentioned dimensions, and concrete recommendations for improvements made.  The main take-away was the need for a clear commitment from senior management for results measurement, through role modeling, signaling, resources, and incentives. This finding is consistent with IEG’s evaluation of learning in World Bank lending (part I), which will be discussed at the Bank’s Board on May 12th. This report finds that tinkering with systems and organizational structures will not ensure that learning and knowledge sharing flourish in the organization’s operations unless measures are taken to transform the organizational culture and incentives.

The partner-led approach – an excuse for not reporting on outcomes or an opportunity to strengthen the systems?

One of the major differences between the comparator agencies in the report -DFID, the World Bank Group - and the Norwegian aid system is that while the former impose strict procedures on their borrowers or grant recipients, with potentially significant transaction costs, Norway has long followed a partner-led approach built mainly on trust.  

This approach could be strengthened by underpinning it with a clear theory of change that would drive any supported initiative or policy, and planning, evaluating and reporting on results without losing the recipient-responsibility principle. This then would be a lesson highly relevant to a World Bank Group that is moving towards a more partner-led approach.  Indeed, as IEGs Director General, Caroline Heider, pointed out during the panel-discussion ”There is no contradiction between a partner-led approach and a strong emphasis on outcomes – the client should be looking to get the most out of the money, not just the most money.”

IEG’s “no benefit of doubt” approach not well-understood

It’s interesting to note that the misunderstanding around the statement that there is ‘no evidence of effect’ would mean ‘evidence of no effect’ is also common across the World Bank Group. In order to incentivize results measurement and evaluation of World Bank projects, the Independent Evaluation Group and the Bank’s Operational Policy teams harmonized guidelines and introduced the concept of ‘no benefit of the doubt’ in 2006.

In practice this means that lack of evidence leads to a downgrade on the efficacy measurement of projects in the same way that an evidence-based lack of effect does. This discrepancy is perhaps the most important cause of a persistent disconnect between World Bank internal ratings and IEG ratings, and it causes disagreements as much as hurt feelings.

Better communications and more outreach to World Bank staff will be needed to enhance the understanding of the underlying concept and to get broader buy-in for the approach.

Whether it is Norwegian aid or World Bank lending, we have to know that what we do works, that it has no adverse effects and that it could not be achieved more efficiently through other means – people’s lives and well-being are at stake. Meaning well doesn’t make it well. Only knowing well will lead to improvements.