Well done We should document the lesson learn as well as to avoid the same mistakes in future in our development project and programs. You provided a good example of chess players who are regularly improving their selves for new games. Thanks for sharing the valuable and meaningful article on Linkedin

Back in 1995, twenty years ago, in the second edition of the journal Evaluation, Christopher Pollitt warned of the dangers of what was then called New Public Sector Management in a paper called Justification by Works or by Faith?: Evaluating the New Public Management. It is as relevant today as it was a warning then. But this goes back even earlier. Back in 1989 the policy academic James Wilson developed a matrix that showed how few public sector activities could be held 'accountable' for what they do purely because of the nature of their activities that prevent a straight line being drawn between independently observable action and independently observable result. The matrix formed the basis of a fascinating paper by Robert Gregory in the mid 90s (ACCOUNTABILITY, RESPONSIBILITY AND CORRUPTION: MANAGING THE 'PUBLIC PRODUCTION PROCESS), where the word 'corruption' is about how the accountability focus 'corrupted' the work of the New Zealand Police force. Around the same time a major figure in the European Union policy field was recruited by the New Zealand government to assess it's own very purist version of New Public Sector Management. Asked to comment on his findings he said that in demanding greater accountability from managers .... the new system of public management could, paradoxically, weaken responsibility. "I really believe that the hard edge of contractualism [ie high levels of accountability] in New Zealand can unwittingly diminish from that sense of responsibility. About ten years ago a review of the relative failure of Results Based Management in four major UN agencies came more or less to the same conclusion. The formal response from those agencies was revealing. It basically rejected the criticism that the idea was fundamentally flawed and essentially blamed staff for not taking it seriously enough. Which as my old colleague Jerry Winston used to say is like telling people jumping off buildings that if they only flapped their arms harder they would be able to fly. None of this of course undermines your work, but merely says that a lot of very skilled and knowledgable people predicted this would happen (and of course a lot of people - who turned out to be more influential - argued against this). So while your report just adds to the literature on the topic it may be timely. Somebody once told me that the difference between being 'right' and 'righteous' is often more a matter of 'time' and 'timing'.

The accountability process should, through obligation, include recommandations analisys and authentification by a limited panel and therefore more fluid in practice. Case of operational research concept. Specifically, it is to systematize and lighten the learning process coupled with accountability.

Thank very much for the document and insightful information. This challenge is very common across a number of development institutions. As M&E practitioner, I have often wondered why after investing so much resources (time, money and energy) into conducting project assessments, the findings and lessons are the findings and lessons that could have helped to enhance the effectiveness of implementation and impact of the intervention are.overlooked and neglected by implementation teams. I believe, there is need to take a careful look at the definition and drivers of what is considered as a successful project or program. Surprisingly, from my experience project teams tends to focus on "getting things done and ticking the box" rather than always reflecting on whether it could have been done differently with better results. There seems to be too much focus on contractual obligation and delivering according to the initial agreement. There is need to focus on "changing the mindset" of project teams from compliance and accountability to focusing on learning and achievement of development results.

Thank you for this thought provoking blog. It rings true that the pendulum has swung far towards compliance in the Bank's operations rather than learning. But based on my operational experience, I would submit that there is a huge amount of operational learning--other than through ICRs-- that feeds into the design of the programs and with much faster feedback loops: team learning and informal and formal transfer of knowledge from past to new TTs, constant TTL-team learning, manager-to-team guidance, OPCS clinics, to name a few. I suspect that some of the self-evaluation knowledge feeds into these learning processes though it may not be so obvious. So maybe we need to ask the question: what is operational learning and how best we can promote it in the new Bank for greater effectiveness?

There are many in Cambodia who do not understand why the World Bank decided to resume funding when the original issues - explained very clearly by their own Inspection Panel - have not been addressed. This is one of the reasons why impunity remains such a problem affecting all aspects of a country's development and human rights. https://www.cambodiadaily.com/news/world-bank-will-resume-funding-to-ca… http://www.worldbank.org/en/news/press-release/2011/03/08/world-bank-bo…

Nice job Rasmus and Caroline....I could not agree with you more that ´more agile, purposeful learning-oriented evaluations´ are needed..Each project that I have evaluated could have benefitted greatly from a real-time M&E system built into the project design. By the time the MTE or Final evaluations are done, it is too late to change course and people have moved on to new things, and of course, accountability is easily swept under a bulging rug that is bulging with mistakes. Timely assessments of the results, adaptive learning and course corrections throughout the implementation are some key ingredients for creating a learning culture that we should be discussing in new project formulations.

Dear Joe Great to hear from you. There is a vast missed potential for M&E to really inform projects and add value during implementation. Would better M&E create the learning culture, or do we need organizations to adopt a more learning-oriented culture to enable them to better seize the opportunities afforded by M&E?

The creation of a more holistic organisational learning cultural frame seems to be critical for taking the next steps in policy and program improvement. While self evaluation seems more than less important for reorienting a project that is off track, a deeper, strategic understanding and will may be required to measure performance in a way that formulates what's needed to improve effectiveness.

Rex: Yes, M&E systems need to measure the right things and teams and their managers need to have the incentives and opportunities and organizational culture to use M&E data and information and take action accordingly. In the World Bank, formal restructuring of projects involving changes to the objective is sometimes very time consuming because some client governments require approval at the highest level, President or Parliament. This makes teams avoid restructuring.

I like the message and tone of this posting: Learning from Evaluation. It shows that the authors are keenly aware of the failure of the Bank's management in charge of operations to implement effectively what is the new goal of the institution -that the WB should function as a knowledge bank. If the learning from the institution's own experience cannot be disseminated effectively among its staff and the findings reflected in more efficacious projects or programs designs, monitoring and supervision, then there is something clearly faulty with its working processes and incentives. I used to think that the World Bank faced an unique set of operational problems. But the reality is that even institutions operating in competitive markets face similar problems because they too have to work with multiple stakeholders to achieve their short and longer term objectives. Short termism is a major problem afflicting most organizations but solutions need to be found. Bank staff are not uniquely responsible for a project's good or poor performance. A variety of other players are involved. Also, if there is a nine year lag between project approval and self evaluation (ICR) preparation, it becomes much more complicated to pin blame since the characters have changed and even the country context is different. So pursuing accountability at the cost of learning may not serve the best interest of anyone, unless of course there is fraud or malfeasance which is a different topic. But the good news is that one can always learn important programmatic lessons about the project even after a long delay. IEG staff are evaluating a large portfolio of projects and interventions at any point of time. Collectively, they provide valuable programmatic lessons which are applicable in a wide variety of contexts. Their work and findings also need to be integrated better with the work operational staff are currently engaged in. It is easy to state that the institution needs a cultural change which is undoubtedly true to some extent. But, hopefully the managers in IEG will outline a sustainable and practical plan to educate operational staff on the lessons . This may require not only IEG staff spending more time in educational seminars for staff and managers , but actually closer involvement in reviewing project briefs and ensuring that programmatic lessons have been taken into consideration. There should be no conflict of interest from this participation as nine plus years would have elapsed before the project is evaluated and they might not be still in evaluation.

The balance between accountability and learning from self evaluations is a problem for every organisation. However, what I think is the the evaluators should design new approaches for presenting their evaluation findings that could have a direct influence on learning. We are in a fast and multitasking world where there are too many demands on the players and to allot time for learning is the challenge. However, I think learning would be facilitated where the manner in which the findings are presented does not necessitate too much devotion of time from the part of the learner to decipher what is new and what is old. Therefore, we need a practical way of depicting the findings to facilitate learning by doing.

Dear Caroline and Rasmus, Many thanks for sharing a very thought provoking blog. I think you highlighted fundamental issue faced by several organisations in the sector. I also picked up few lessons I have been trying to address the same issue and struggling to ensure that we consistently use the learning for improving programme design. We also disseminate learning's from the evaluation studies by producing evaluation summaries. which is a two page document focused on recommendations and learning. Thank you once again for your great work. Regards EH

Thanks, glad you liked it! As others have also commented, the issues we discuss here affect evaluation uptake in many organizations.