My blog in July on Value-for-Money generated a lot of feedback. Many thanks for the thoughtful comments and suggestions. The sense I got from the comments is that we need to unpack the value proposition of evaluation before we can contrast it with the money – cost and price – part of the equation.

Theory of Change

The value of evaluation lies in helping to improve development results. Our aim is that evaluation leads to better informed decisions and actions before and during the life-cycle of an intervention, which in turn results in greater benefits at a lower cost.

The Value of Independent Evaluation

The value of independent evaluation to an institution like the World Bank Group lies in its transparency, accountability, and learning. The importance of independence was discussed in a previous blog.

Specific evaluations add value by:

  • Producing information no-one else has compiled. That’s what we did in our PPP evaluation where work across The World Bank Group was so dispersed that it took our evaluation to get an overall sense of the portfolio. The value of this information to our client, the new Cross-Cutting Solutions Area for PPPs, can be estimated by the saved cost to them.
     
  • By stepping back from the daily routines of designing and implementing interventions, independent evaluation can generate new insights by shifting perspectives and combining evidence in ways that allow people to see improvements that they otherwise might have missed. For instance, our evaluation of sustainable finance of transport investments found that in spite of policy commitments from the highest levels, the share of projects with objectives to sustain transport investments had declined to 10 percent. If our recommendations are heeded, countries could attain savings and higher economic returns on their investments in the transport sector.
     
  • Evaluation evidence (or validating self-evaluation findings) on results and performance generates an understanding of what works and what needs to be fixed. Our validations of project completion reports provide information which then gets aggregated in the annual Results and Performance report. The value of our work would be to decision-makers and implementers to help them take corrective actions when interventions do not produce expected results. In other words to avoid or reduce losses.

Realizing the Value of an Evaluation

Value is not generated by simply producing information. It’s the changes in behavior, practices and policies that matter. That means we have to reach out and share lessons and knowledge in ways that get absorbed and influence behaviors. We are fortunate to have the resources to do that and have committed ourselves to revving up our work on this front. To know whether we succeed, we have an established system to track the implementation of recommendations over time. Our results framework also includes an external assessment of the actual use and impact of our evaluations.  

Estimating the Value-Added

But, our measures do not quantify the expected value improvements that should follow from evaluations. How would we make this work?  For example, in our evaluation The Big Business of Small Enterprises we found that the WBG spends $3 billion annually on targeted support to small and medium sized enterprises. But only 34 percent of the interventions state whether they address market failures and only 13 percent provide evidence to that effect. And while the overall outcome ratings of these projects is positive, we also found that few projects had quantified data and evidence to demonstrate the extent to which SME development had been helped and what wider economic effects this has had. Assuming that as a result of our evaluation, targeted SME interventions would have a clear line-of-sight on the market failures they will address and how this will affect the sector’s growth and multiplier effects, the value of the evaluation could be enormous.

Does it translate into a number, a dollar amount? Only if we start seeing data that helps quantify outcomes. And, let’s not forget the intangible value of more or better information.

Comments

Submitted by Anna Guerraggio on Tue, 09/16/2014 - 02:56

Permalink
Caroline, thank you once more for sharing your thoughts and stimulating the debate on this important issue. The difficulty of estimating the evaluations's value added lies in a part of this value being difficult to measure and intangible. This pertains to the change we are able to provoke not just through our evaluation reports and recommendations, but also through the dialogue we have in the course of the evaluation with our evaluands, initiating a discussion on the way our organizations "conduct their business". In the public sector where the market is less of an indicator of "how good our organizations are/our clients are satisfied", having such a debate and having staff thinking about existing processes is not usual practice. In the office where I work, three years after the completion of the evaluation, we ask the evaluands (through interviews and survey) not only about the extent to which recommendations have been implemented, but also what has been the evaluation's major value added for them. Often it goes indeed back to the quality of the evaluation process, and the engagement of the evaluation team throughout, which inspired them to do a better job and actively push for change.

Submitted by Caroline Heider on Thu, 09/18/2014 - 04:53

In reply to by Anna Guerraggio

Permalink
Anna, as always: very thoughtful comments and a great suggestion for expanding our follow-up studies beyond implementation of recommendations. Process does matter a lot, but so does the report itself and the value added it leaves behind. One of my former colleagues at the Asian Development Bank told me how she reread an evaluation as it was so useful to her thinking and work in the country for which she was the economist. Or others from previous jobs who in one case said "the best thing about the project was your evaluation" (sad really). But moving from this anecdotal approach to something more systematic, as you suggest, is a great step in the right direction.

Submitted by Ian goldman on Fri, 09/19/2014 - 00:54

Permalink
The challenge is to work out what difference the evaluations make to the Programmes. We are planning an evaluation of evaluations for 2016/17. However meanwhile we are using the back of envelope calculation - $5 billion of gov expenditure being evaluated, if we make 10% difference (conservative) that is $500m over 3 years and cost of the national evaluation system is around $5 million per year - so roi of minimum of 30:1. Be interesting to test that in the evaluation of evaluations.

Submitted by Caroline Heider on Fri, 09/19/2014 - 01:50

In reply to by Ian goldman

Permalink
Ian, it will be very interesting to see how you calculate the rate of return on evaluations. A couple of questions ahead of 2016/17. Before issuing a report, do you assess whether/what level of efficiency gains the recommendations might produce? Likewise, can you estimate at the time of making recommendations whether and how much they will contribute to better outcomes? And for that matter, are all the benefits of government programs quantifiable and hence the increase from better outcomes measurable? It would be great if you could share more details.

Submitted by Daniel on Tue, 09/30/2014 - 03:33

Permalink
You quite rightly say how the value of evaluation lies in helping to improve, not just evaluate, development results. I do sometimes wonder whether some donors have adequate capacity to design, commission and then translate the recommendations (assuming they are 'correct' and 'valid') in order to improve results. A price some evaluation departments perhaps pay for being deliberately positioned as 'independent' from operations. Tough trade-offs between principles and objectives. There appears to be an imbalance of effort and wit in managing evaluations’ theory of change. There is a lot of noise and words written about how to manage the action part of the theory (galvinised through "when will we ever learn", RCTs. J-PAL, participatory impact evaluation, the list goes on among those who peddle approach and method etc.). This has received "swooning responses" among many. As I mentioned on your other blog, I think this is because much of the 'value' of evaluation is judged by standards more relevant for academic research, not professional evaluation, as an end it itself – the person has a PhD, has been published and it got published. The academic credibility of the person and the method is important of course, but not as important as many are led to believe I think. Managing the change theory, on the other hand, has received relatively little attention and, in different, ways as challenging if not more so than delivering the evaluation: in helping users interpret, critically look at and take action that will...........deliver better dev results. The consequence? Evidence that limits itself to “management responses” and patchy evidence as to the consequences (intended and positive or vice versa) of the process and the product. There is also the issue of lagged effects. I led an impact evaluation that revealed significantly and really interesting, different interpretations among various folk (donors, managers, field staff and their clients) associated with the programme as to why and how it worked. Good and useful that the evaluation process raised such differences, through animated discussions on issues not discussed before. As Anna rightly points out, often the evaluation process can provide space and discretion to some to rate and comment on the performance of those paid to support them and the relevance. Such innate value is really important. We tried to do this, using Lawrence Salmen's approach called Beneficiary Assessments he developed in the late 1980's. It seems to have come back into vogue, albeit through other channels. As an aside, it is funny how some question the significance of findings 'beneficiary' assessments generate based on how statisticians treat the term. We tried to avoid passing judgement in all this as well as restrict our efforts to measuring indicators. Good, too, that with a change of 'leadership', some of the recommendations those still there wanted are now being followed up three years later.... Thanks again

Submitted by Caroline Heider on Tue, 09/30/2014 - 01:07

In reply to by Daniel

Permalink
Daniel, many thanks for your detailed and thoughtful comment. Let me limit my response to just a few things. Does independence have the trade-off with learning and follow-up? Maybe in some situations, but in my experience independence ensures that sometimes unwanted truths are surfaced that go against existing policies and are necessary to come up with new directions. Independent evaluation can and has to step out of the comfort zone and help others who otherwise have to follow the company line do so. But, as part of the process we also discuss the recommendations to ensure good understanding of what is needed. Does it lead to academic accolades rather than action? In our case less so, as we have a pretty strong system of follow-up to management responses where we rate progress against commitments. We have also piloted some follow-up studies and are looking into other ways to determine what difference our evaluations have made. An important step in that process was articulating our own objectives, aligning the choice of our evaluations and now ensuring that they speak to these objectives including using processes that shed light and further understanding and discussion.

Submitted by Daniel on Tue, 09/30/2014 - 05:59

Permalink
Dear Caroline, You quite rightly say how the value of evaluation lies in helping to improve, not just evaluate, development results. I do sometimes wonder whether some donors have adequate capacity to design, commission and then translate the recommendations (assuming they are 'correct' and 'valid') in order to improve results. A price some evaluation departments perhaps pay for being deliberately positioned as 'independent' from operations. Tough trade-offs between principles and objectives. There appears to be an imbalance of effort and wit in managing evaluations’ theory of change. There is a lot of noise and words written about how to manage the action part of the theory (galvinised through "when will we ever learn", RCTs. J-PAL, participatory impact evaluation, the list goes on among those who peddle approach and method etc.). This has received "swooning responses" among many. As I mentioned on your other blog, I think this is because much of the 'value' of evaluation is judged by standards more relevant for academic research, not professional evaluation, as an end it itself – the person has a PhD, has been published and it got published. The academic credibility of the person and the method is important of course, but not as important as many are led to believe I think. Managing the change theory, on the other hand, has received relatively little attention and, in different, ways as challenging if not more so than delivering the evaluation: in helping users interpret, critically look at and take action that will...........deliver better dev results. The consequence? Evidence that limits itself to “management responses” and patchy evidence as to the consequences (intended and positive or vice versa) of the process and the product. There is also the issue of lagged effects. I led an impact evaluation that revealed significantly and really interesting, different interpretations among various folk (donors, managers, field staff and their clients) associated with the programme as to why and how it worked. Good and useful that the evaluation process raised such differences, through animated discussions on issues not discussed before. As Anna rightly points out, often the evaluation process can provide space and discretion to some to rate and comment on the performance of those paid to support them and the relevance. Such innate value is really important. We tried to do this, using Lawrence Salmen's approach called Beneficiary Assessments he developed in the late 1980's. It seems to have come back into vogue, albeit through other channels. As an aside, it is funny how some question the significance of findings 'beneficiary' assessments generate based on how statisticians treat the term. We tried to avoid passing judgement in all this as well as restrict our efforts to measuring indicators. Good, too, that with a change of 'leadership', some of the recommendations those still there wanted are now being followed up three years later.... Thanks again

Add new comment