In search of champions of learning from self-evaluations

Forty-five years ago, World Bank President, Robert McNamara mandated self-evaluation for Bank projects. And we have been going at it ever since.

Self-evaluations are now undertaken by all of the Bank Group institutions - World Bank, IFC, and MIGA-and are targeted at project, program, and country levels. The stated purpose of the self-evaluation regime is to foster organizational learning, inform management action, and ensure accountability for results and performance. In turn, IEG's validations (or desk reviews) of certain outputs from the self-evaluation system feed into corporate "scorecards", "dashboards", and other systems. They also provide foundation-level material for our larger scale evaluations.

That, broadly, is how things are set up to work - at least on paper. But we're interested in finding out more about how well the theory is reflected in reality. So we've decided, now that spring is in the air, it's time to check out the ROSES (Report on Self-Evaluation Systems)! And we're looking for your help as we attempt to dig out good practice and lessons on how self-evaluation might best support learning and effectiveness.

The ROSES will cover most WBG self-evaluation systems, looking at:

  • the integrity of the self-evaluation architecture, and whether accountability, performance management, and learning objectives are being achieved; and, at
     
  • behaviors, incentives, and organizational norms and practices that shape how self-evaluation information is produced and used.


One of the questions we plan to address is "how well these systems serve their learning purposes?" Over the years the Bank Group has seen a number of initiatives-"learning from failure", the "Solutions Bank", "Knowledge Talks", and "Knowledge Silo Breakers" - designed to encourage and maximize learning; however, relatively little attention has been paid to the use we make of project level evaluations, of which there are hundreds every year. Staff surveyed for our evaluation on How the Bank Learns reported that they only infrequently consulted project completion reports. And while all self-evaluation templates have mandatory sections for teams to record "lessons learned", our reviews of these reports see near-identical lessons recorded year after year. Arguably, if lessons were being learnt, we would be bending the downward trend in overall portfolio performance documented in IEG's Results and Performance 2013.

Achieving better results through self-evaluation

Achieving better results through self-evaluation

So the question we're asking you to help us with is how might the Bank Group best tap into the learning value of self-evaluations? Specifically, we're looking for instances and examples of how and where self-evaluation has been systematically used for learning purposes.

There is no restriction on the type of learning we would like to hear about, but we're particularly interested in hearing from colleagues about any type of Bank, IFC, or MIGA self-evaluation. Perhaps your team launched a study of completion or supervision reports to inform design of future projects. Or you started a seminar series to disseminate self-evaluations. Or a trust fund partner or client country worked with the Bank Group to promote operational learning in the context of self-evaluation of a project or program. Or maybe you have experience of some other self-evaluation system that worked for learning.

There must be good news stories out there, so please respond and share your experience. For our part, we plan to study and follow up on examples of good practice to help fertilize the ROSES!

You can respond below and share your experience with the whole community, or you can email us at ieg@worldbank.org. Looking forward to hearing from you soon!

 

Comments

Submitted by Alexander Kremer on Tue, 03/31/2015 - 21:29

Permalink
We persist with the assumption that development outcomes would be better if only we had better knowledge and lesson-learning. This assumption that lesson-learning is the key sustains the evaluation industry, and IEG is therefore the institution least likely to question it objectively. The assumption that lesson-learning is the key has recently been reinforced by the pressure on management and staff to show loyalty to the President's model of the Knowledge Bank and the Science of Delivery. Lesson-learning might not be the key. IEG's graph above, "Achieving Better Results Through Self-Evaluation" is not an axiom. It is a hypothesis. Consider for a moment the alternative hypothesis that the key to better project outcomes might actually be what nearly all TTLs say it is: more time and more resources and incentives that reward a team which backs out when things look bad. Before launching the umpteenth investigation into knowledge management, it is necessary for IEG to answer the logically prior question: when projects fail, was it knowledge, incentives or resources that caused the failure?

Submitted by Rasmus Heltberg on Mon, 04/06/2015 - 22:14

Permalink
Dear Alexander: Thanks for your thoughtful comments. It is just these kinds of on-the-ground perspectives we are hoping to hear and include in our analysis. We are right in the middle of our research and one of the things we have heard in TTL interviews is that these self-evaluations can provoke reflection about the project process and help TTLs integrate how to deal with all the pieces and people involved in projects, meaning that there is some tacit learning going on, but also that this learning does not consistently get channeled in a way that makes it available beyond the TTL’s experience. So, in the case you provide here, how would you know when to back out of a project, if you are not learning? And in starting a new project, wouldn't you then have a lesson learned of, "Don't bother with that approach, we tried it in X project, and it didn't work?" Our evaluaiton is indeed thinking through the value of the Lessons Learned as they are currently done in self-evaluations, and getting ideas from people like you of alternatives and how to get lessons to go beyond the individual TTL. In addition, our study is looking at the accountability, performance management, and learning aspects of self-evaluation, how they affect each other and the best balance between them.

Add new comment

By submitting this form, you accept the Mollom privacy policy.