institutional learning

On My Mind: "When Will We Ever Learn?"

mpadrino — Tue, 16 Jan 2018 18:18:11 +0000

On My Mind: "When Will We Ever Learn?" mpadrino Tue, 01/16/2018 - 13:18

Comments

Thanks for sharing this post…

Thanks for sharing this post. Learning should be stimulated at great institutions such as World Bank where theories can be developed but what is difficult is the practicability of learning. What involvement should the field has in learning??

Thank you very much for…

Thank you very much for sharing your thoughts on this. I strongly believe that evaluators (not only for World Bank projects) even will smaller country-based evaluations "keeps repeating recommendations" and in fact, including a repeat on evaluation designs. The comment on "...we don't see anything new" is somewhat half-truth. Many evaluation design is focused on the DAC framework of efficiency, effectiveness, relevance, impact and sustainability and, with that we focus our recommendations within such framework. As a result, we often repeat the ideas on how to manage projects or how to deal with project issues in the covered communities. That is not a problem at all. The problem with learning from the lessons and experiences (even if they were similar or the same ideas) is on how we integrate such ideas in the re-entry plans at the community level. The participation of the people (the covered communities) in the entire evaluation process has been given less importance because the evaluators has the propensity to formulate and ensure that re-entry plans for themselves or donors to undertake. The people in the covered communities are often listeners rather than the doers. I have witnessed several feedbacking and backstopping sessions/meetings but they are intended for the managers and supervisors or team funding institutions rather than the covered communities to act. This is molding "dependent mentalities" rather than empowerment. Perhaps, the emphasis should be on the real participation of the ultimate beneficiaries of the project no matter how long it would take the evaluators to realize this process. This is not a solution as well, but it is a point of reflection to look into the design on how we put learning and change into the hands of the people we work with. Again thanks for sharing.

Add new comment

Self-Evaluation and Learning - Where is the Reset Button?

mpadrino — Tue, 06 Sep 2016 05:22:35 +0000

Self-Evaluation and Learning - Where is the Reset Button? mpadrino Tue, 09/06/2016 - 01:22

Add new comment

Learning from Evaluation: How can we Stay at the Top of the Game?

mpadrino — Tue, 02 Aug 2016 09:37:51 +0000

Learning from Evaluation: How can we Stay at the Top of the Game? mpadrino Tue, 08/02/2016 - 05:37

Comments

Well done We should document…

Well done We should document the lesson learn as well as to avoid the same mistakes in future in our development project and programs. You provided a good example of chess players who are regularly improving their selves for new games. Thanks for sharing the valuable and meaningful article on Linkedin

Thanks for your kind words…

Thanks for your kind words Mr Khan! To document and learn these lessons we need to identify them first and that can be impeded if the organizational culture focuses on accountability and reporting excessively and at the cost of fostering learning. Many organizations, public and private alike, struggle with this problem. At IEG we are very interested in how to change the organizational culture in a more learning-oriented direction and believe that there are many bright spots, including around impact evaluation and some of the retrospectives. These are voluntary unlike the mandated self-evaluations which follow prescribed formats geared to uniform reporting. This is all discussed in the "Behind the Mirror" report. Glad you like the analogy to chess, my favorite sport. I must confess that I rarely spend time analyzing my own past games though.........

Back in 1995, twenty years…

Back in 1995, twenty years ago, in the second edition of the journal Evaluation, Christopher Pollitt warned of the dangers of what was then called New Public Sector Management in a paper called Justification by Works or by Faith?: Evaluating the New Public Management. It is as relevant today as it was a warning then. But this goes back even earlier. Back in 1989 the policy academic James Wilson developed a matrix that showed how few public sector activities could be held 'accountable' for what they do purely because of the nature of their activities that prevent a straight line being drawn between independently observable action and independently observable result. The matrix formed the basis of a fascinating paper by Robert Gregory in the mid 90s (ACCOUNTABILITY, RESPONSIBILITY AND CORRUPTION: MANAGING THE 'PUBLIC PRODUCTION PROCESS), where the word 'corruption' is about how the accountability focus 'corrupted' the work of the New Zealand Police force. Around the same time a major figure in the European Union policy field was recruited by the New Zealand government to assess it's own very purist version of New Public Sector Management. Asked to comment on his findings he said that in demanding greater accountability from managers .... the new system of public management could, paradoxically, weaken responsibility. "I really believe that the hard edge of contractualism [ie high levels of accountability] in New Zealand can unwittingly diminish from that sense of responsibility. About ten years ago a review of the relative failure of Results Based Management in four major UN agencies came more or less to the same conclusion. The formal response from those agencies was revealing. It basically rejected the criticism that the idea was fundamentally flawed and essentially blamed staff for not taking it seriously enough. Which as my old colleague Jerry Winston used to say is like telling people jumping off buildings that if they only flapped their arms harder they would be able to fly. None of this of course undermines your work, but merely says that a lot of very skilled and knowledgable people predicted this would happen (and of course a lot of people - who turned out to be more influential - argued against this). So while your report just adds to the literature on the topic it may be timely. Somebody once told me that the difference between being 'right' and 'righteous' is often more a matter of 'time' and 'timing'.

Thanks for your thoughtful…

Thanks for your thoughtful comments and references! I was not aware of this literature and the broader debate about new public management. Your comment reminds me of Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.” (In other words, when we use some indicator to evaluate performance and it is the same as the target being optimized by those who are being measured, it is no longer a reliable measure of performance.) I think you are probably right to imply that learning this lesson has taken long. In fact, I am not sure it has yet been internalized. I should say though that our report has been well-received by staff and management in the World Bank Group, who appreciate its analysis of behavior, incentives and motivations--the very drivers of the tension between learning and accountability.

The accountability process…

The accountability process should, through obligation, include recommandations analisys and authentification by a limited panel and therefore more fluid in practice. Case of operational research concept. Specifically, it is to systematize and lighten the learning process coupled with accountability.

Thanks for your comment. I…

Thanks for your comment. I agree that making the learning process lighter, simpler, more intuitive would be helpful. But I am not sure that more mandates and oversight and accountability would help with this. In fact, the report cited in this blog post documents a tension between accountability and learning which has to be resolved through, for example, more agile, purposeful learning-oriented evaluation and stronger managerial signals that results, learning and self-evaluation matter.

Thank very much for the…

Thank very much for the document and insightful information. This challenge is very common across a number of development institutions. As M&E practitioner, I have often wondered why after investing so much resources (time, money and energy) into conducting project assessments, the findings and lessons are the findings and lessons that could have helped to enhance the effectiveness of implementation and impact of the intervention are.overlooked and neglected by implementation teams. I believe, there is need to take a careful look at the definition and drivers of what is considered as a successful project or program. Surprisingly, from my experience project teams tends to focus on "getting things done and ticking the box" rather than always reflecting on whether it could have been done differently with better results. There seems to be too much focus on contractual obligation and delivering according to the initial agreement. There is need to focus on "changing the mindset" of project teams from compliance and accountability to focusing on learning and achievement of development results.

Couldn't agree more! We need…

Couldn't agree more! We need to away from "check the box" approaches to reflection and course correction based on insightful use of M&E data, identification of issues and challenges, and candid discussion within teams. Doing so will require a change in mindset of how M&E and project implementation are approached from a compliance focus to a learning and results mindset. As you state, the challenge is common across a number of agencies. I will be curious to learn of examples of how other institutions have tackled this challenge...................?!

Thank you for this thought…

Thank you for this thought provoking blog. It rings true that the pendulum has swung far towards compliance in the Bank's operations rather than learning. But based on my operational experience, I would submit that there is a huge amount of operational learning--other than through ICRs-- that feeds into the design of the programs and with much faster feedback loops: team learning and informal and formal transfer of knowledge from past to new TTs, constant TTL-team learning, manager-to-team guidance, OPCS clinics, to name a few. I suspect that some of the self-evaluation knowledge feeds into these learning processes though it may not be so obvious. So maybe we need to ask the question: what is operational learning and how best we can promote it in the new Bank for greater effectiveness?

Dear Zeljko Very true that…

Dear Zeljko Very true that there is a lot of operational learning and knowledge sharing outside the formal self-evaluation systems. Of course there is. As you know, IEG evaluated this in the two reports on Learning and Results in World Bank Operations: How the Bank Learns (https://ieg.worldbankgroup.org/evaluations/learning-and-results) and found, among other things, that the prevailing oral culture of informal knowledge sharing has its downsides: knowledge is easily lost when people move or leave; knowledge transfer depends on who you know; in a big global decentralized institutions, exchanging knowledge face to face is not always feasible, and so on. The question is therefore also how the formal self-evaluation systems can help identify and document relevant knowledge.

There are many in Cambodia…

There are many in Cambodia who do not understand why the World Bank decided to resume funding when the original issues - explained very clearly by their own Inspection Panel - have not been addressed. This is one of the reasons why impunity remains such a problem affecting all aspects of a country's development and human rights. https://www.cambodiadaily.com/news/world-bank-will-resume-funding-to-ca… http://www.worldbank.org/en/news/press-release/2011/03/08/world-bank-bo…

Thanks for the comment! The…

Thanks for the comment! The World Bank Group has multiple mechanisms of accountability: us here at IEG; the audit function; the Inspection Panel; and more. I am not in a position to comment on the Inspection Panel report that you reference, or on Management's response to it. The Bank's office in Cambodia would be better placed.

Nice job Rasmus and Caroline…

Nice job Rasmus and Caroline....I could not agree with you more that ´more agile, purposeful learning-oriented evaluations´ are needed..Each project that I have evaluated could have benefitted greatly from a real-time M&E system built into the project design. By the time the MTE or Final evaluations are done, it is too late to change course and people have moved on to new things, and of course, accountability is easily swept under a bulging rug that is bulging with mistakes. Timely assessments of the results, adaptive learning and course corrections throughout the implementation are some key ingredients for creating a learning culture that we should be discussing in new project formulations.

Dear Joe Great to hear from…

Dear Joe Great to hear from you. There is a vast missed potential for M&E to really inform projects and add value during implementation. Would better M&E create the learning culture, or do we need organizations to adopt a more learning-oriented culture to enable them to better seize the opportunities afforded by M&E?

The creation of a more…

The creation of a more holistic organisational learning cultural frame seems to be critical for taking the next steps in policy and program improvement. While self evaluation seems more than less important for reorienting a project that is off track, a deeper, strategic understanding and will may be required to measure performance in a way that formulates what's needed to improve effectiveness.

Rex: Yes, M&E systems need…

Rex: Yes, M&E systems need to measure the right things and teams and their managers need to have the incentives and opportunities and organizational culture to use M&E data and information and take action accordingly. In the World Bank, formal restructuring of projects involving changes to the objective is sometimes very time consuming because some client governments require approval at the highest level, President or Parliament. This makes teams avoid restructuring.

I like the message and tone…

I like the message and tone of this posting: Learning from Evaluation. It shows that the authors are keenly aware of the failure of the Bank's management in charge of operations to implement effectively what is the new goal of the institution -that the WB should function as a knowledge bank. If the learning from the institution's own experience cannot be disseminated effectively among its staff and the findings reflected in more efficacious projects or programs designs, monitoring and supervision, then there is something clearly faulty with its working processes and incentives. I used to think that the World Bank faced an unique set of operational problems. But the reality is that even institutions operating in competitive markets face similar problems because they too have to work with multiple stakeholders to achieve their short and longer term objectives. Short termism is a major problem afflicting most organizations but solutions need to be found. Bank staff are not uniquely responsible for a project's good or poor performance. A variety of other players are involved. Also, if there is a nine year lag between project approval and self evaluation (ICR) preparation, it becomes much more complicated to pin blame since the characters have changed and even the country context is different. So pursuing accountability at the cost of learning may not serve the best interest of anyone, unless of course there is fraud or malfeasance which is a different topic. But the good news is that one can always learn important programmatic lessons about the project even after a long delay. IEG staff are evaluating a large portfolio of projects and interventions at any point of time. Collectively, they provide valuable programmatic lessons which are applicable in a wide variety of contexts. Their work and findings also need to be integrated better with the work operational staff are currently engaged in. It is easy to state that the institution needs a cultural change which is undoubtedly true to some extent. But, hopefully the managers in IEG will outline a sustainable and practical plan to educate operational staff on the lessons . This may require not only IEG staff spending more time in educational seminars for staff and managers , but actually closer involvement in reviewing project briefs and ensuring that programmatic lessons have been taken into consideration. There should be no conflict of interest from this participation as nine plus years would have elapsed before the project is evaluated and they might not be still in evaluation.

Thank you for the kind words…

Thank you for the kind words about our posting and about IEG's larger work! Readers of this blog know that we make a lot of effort to promote learning and uptake from IEG's work, while preserving our independence and accountability functions. This blog post is really about use of and learning from self-evaluations done by or for operational units -- that is, the portions of the WBG's evaluation architecture not led from IEG. As you indicate, these systems are not really meant and designed to pin blame on individuals. They are intended to serve: broad organizational reporting and accountability; project and portfolio performance monitoring; and organizational learning. The "Behind the Mirror" evaluation finds that the reporting purpose is much better served than the performance monitoring and learning purposes. This because of the prevailing "compliance mindset": fulfilling reporting requirements but not learning from the process. I find it interesting that so many people are commenting that other organizations face similar challenges and I am keen to hear if anyone has experience with driving organization-wide behavior change in a context that is similar to what we talk about here.

The balance between…

The balance between accountability and learning from self evaluations is a problem for every organisation. However, what I think is the the evaluators should design new approaches for presenting their evaluation findings that could have a direct influence on learning. We are in a fast and multitasking world where there are too many demands on the players and to allot time for learning is the challenge. However, I think learning would be facilitated where the manner in which the findings are presented does not necessitate too much devotion of time from the part of the learner to decipher what is new and what is old. Therefore, we need a practical way of depicting the findings to facilitate learning by doing.

Thanks for your comment! The…

Thanks for your comment! The templates for the WBG's self-evaluations actually try to do this with sections on "lessons learned" that are meant to distill what's new into short bite-sized pieces of information. Unfortunately, reality is that lessons have a justified reputation of being quite obvious and generic and therefore of low value. We found that in World Bank completion reports, lessons are often written in very general terms, without specific recommendations on how to do things differently in the future. But this being said, I also think that it is too easy for practitioners to blame non-learning on external factors such as being busy or information not being packaged in just-the-right way. People in professional jobs also need to take responsibility for their own continued learning and professional growth. Evaluators can and should package and present information in attractive ways, but there has to be a demand for the information.

Dear Caroline and Rasmus,…

Dear Caroline and Rasmus, Many thanks for sharing a very thought provoking blog. I think you highlighted fundamental issue faced by several organisations in the sector. I also picked up few lessons I have been trying to address the same issue and struggling to ensure that we consistently use the learning for improving programme design. We also disseminate learning's from the evaluation studies by producing evaluation summaries. which is a two page document focused on recommendations and learning. Thank you once again for your great work. Regards EH

Thanks, glad you liked it!…

Thanks, glad you liked it! As others have also commented, the issues we discuss here affect evaluation uptake in many organizations.

Add new comment

The Role of Mentoring in Growing the Next Generation of Evaluators

mpadrino — Tue, 26 Jul 2016 09:32:24 +0000

The Role of Mentoring in Growing the Next Generation of Evaluators mpadrino Tue, 07/26/2016 - 05:32

Comments

Agreed. This article also…

Agreed. This article also highlights the importance of recognition through awards for outstanding performance. Feedback and encouragement through Awards for excellence and achievement can play a vital role in promoting exemplary practice, fostering continuous improvement and engendering an adaptive management culture. If we are interested in promoting exemplary practice, we should all be encouraging the establishment of awards for excellence!

Glenys, certainly it is…

Glenys, certainly it is important to reward people for things they do in outstanding ways, as much as acknowledging all the many good contributions people make on a day-to-day basis. What we did in the mentoring program was driven much more by the importance of and demand for mentoring than for any award--the program is the reward in itself. This does not diminish that I am immensely proud of my team for having won the award and happy for them that they and their hard work has been recognized well beyond IEG.

Recognition of prior…

Recognition of prior learning (RPL), prior learning assessment (PLA), or prior learning assessment and recognition (PLAR), describes a process used by organizations, training institutions, colleges and universities around the world to evaluate skills and knowledge acquired outside the classroom for the purpose of recognizing competence against a given set of standards or learning objectives. RPL is practiced in any country employing vocational education and training processes as a means of training individuals in competencies required in the workplace. It is also used in some private and public sector organizations for the purpose of recruitment, performance management, career and succession planning. Methods of assessing prior learning are varied and include: evaluation of prior experience gained through volunteer work, previous paid or unpaid employment, standardized exams or observation of actual workplace behavior. The essential element of RPL is that it is an assessment of evidence provided by an individual to support their claim for competence against a given set of standards or learning objectives. RPL is often confused with Credit Transfer or assessments conducted in order to recognize advanced standing or for assigning academic credit. The essential difference between the two is that RPL considers evidence of competence that may be drawn from any aspect of an applicants professional or personal life. Credit Transfer and advanced standing deal primarily with an evaluation of academic performance as it relates to a particular field of study and whether or not advanced standing may be granted towards the gaining of additional qualifications. World Examination Services shall be please to offer its services if anyone require,visit www.wes-edu.org for futher info or call 00923009243419 regards

This initiative is amazing…

This initiative is amazing. This is an opportunity for young Professional to grow up quickly in terms of professionalism. By the way, how someone working outside the world bank group can benefit from this mentoring program? Thanks in advance for your reply.

Sylli, thanks for your…

Sylli, thanks for your comment and appreciation. It is actually not only young people who are partnering with a mentor. We haven't opened it up to outsiders, but I believe some of the regional and global evaluation associations are trying to do something similar. Check it out with them.

This all sounds wonderful…

This all sounds wonderful. Much to learn from. Congratulations!

Thank you, Krishna.

This experience is really…

This experience is really informative. How did you dealt with the workload and work program of the mentors and mentees? In other words, hwo did ypu get the mentoring time for your staff? In similar organizations, mentoring is overlooked because there is no institutional incentives to engage in, especially the recognition of time devoted to mentoring and learning. I would like to hear from you. Congratulations.

Debazou, thanks for your…

Debazou, thanks for your comment (nice to "see" you online). At the institutional level, the answer is that we program 2 weeks of staff time for training. This can be used for formal training but also more informal training like mentoring. But, my impression is that our staff are so keen on supporting each other that they make time for mentoring relationships (on both sides), whether formally charged to training budget or not.

Hi, Debazou, We encourage…

Hi, Debazou, We encourage mentoring participants to try to spend about 1 hour per month together as a rough guide to make it manageable (if they can do more, that's fantastic). Even those who are able to spend less time report real benefits in our most recent mentoring program evaluation, such as: 91% were motivated to improve performance at work from the mentoring engagement 83% learned new knowledge and skills useful in their work from the mentoring relationship 100% were satisfied with the mentoring matches 67% achieved more than half of their stated learning goals/objectives One comment sums it up with "What is important is mutual motivation..." We also encourage staff to record their mentoring participation as part of their professional development. The institution is also including mentoring in the developmental actions (recommendations to be a mentee or a mentor) in the annual talent review process. So we are working on the incentives, but the real credit goes to the staff motivation to support each other. regards, Kristin

Congratulations to you,the…

Congratulations to you,the team on the award and commendable that the Independent Evaluation Group responded positively to a gap that employees had identified.It is true that Mentoring is a powerful tool towards advancing professionalization.Formally, I have been privileged through my career to be part of mentoring program as a mentee to Trainee Accountants. Organisations nowadays face a myriad of challenges ; in managing different generations coupled with the 4th Industrial Revolution where Klaus Schwab states that talent more than capital will be a more critical factor of production.Companies are as a result under pressure to attract, develop and retain high caliber of talent .Millennials being the fastest growing generation in workplace , responding to their complex needs is not only progressive but adaptive to the changing environment .Independent Evaluation Group is ahead of the curve because it has already developed and is implementing the Mentoring program which includes other tools apart from mentoring. According to Jenna Atkinson a corporate training consultant; Millennials will need to be equipped to assume leadership positions sooner than other generations have had to.Problem would be that the crop in pipeline has much stronger technical skills and lack soft skills therefore urging companies to be more proactive rather than reactive to a minimize future shortage of suitable candidates to fill leadership vacancies. The Independent Evaluation Group through its program is managing this risk in that its program pairs employees from stakeholders,employees on varying grades and levels along with Directors of Organisation in display of their commitment. In a survey conducted by Development Dimensions International ; it found that for every $1000 invested in employees soft skills training, employers earn a return of $4000.Furthermore Atkinson states that : > Millennials want to make an impact in Organisation immediately and that engaging them in training early on is likely to increase retention percentage as well as loyalty to Companies Independent Evaluation Group model is exemplary because new recruits undergo new staff learning roadmap. > Employee training should achieve personal and professional goals while offering opportunities for growth , IEG has addressed this challenge by incorporating coaching component as well as TGIF ( Technology , Good Practice and Innovation ) > Atkinson argues that Millennials want short bursts of training which in the main have a shorter,consistent span,while it is educational must be entertaining .Again,IEG understood this well and ensured that its TGIF learning series is weekly broken into one hour sessions and include social media workshop as well as innovative ideas development . Independent Evaluation Group certainly looks poised for future management and the 4th Industrial Revolution employees whose innovative ideas ,analytical skills ,adaptive managerial competence will be key to leading workforce through the changing environment or disputation.Secondly contribute towards achieving goal 8 of Decent Work and Economic Growth goal in the SDGs ( Sustainable Development Goal ). Continue breaking new ground and all the best for you and the team .

Congratulation on the…

Congratulation on the success of the initiative and thank you for sharing your experience and lessons learned. I am curious to know how the employees raised mentoring as the priority five years ago? Have you conducted a survey at the time? How are the staff assessed those who are mentoring in terms of their capability and those who request mentoring in terms of determining the area of improvement? I assume this initiative is expected to continue and continuously improve?

Hi, Hala, The need for…

Hi, Hala, The need for mentoring emerged from consulting with staff and from receiving feedback from them in multiple forms. One form was a survey that was designed knowing already that there was a need and desire for mentoring from staff and that we needed to further understand their specific career, organizational, and specialization learning needs. since then the program has been adjusted based on program evaluations and ongoing participant feedback. We do not assess staff on mentoring capability but instead work with the idea that the basic training, tools, and resources we provide to all participants, combined with their willingness to engage and intent to contribute will prepare them enough for success. We are continuing the program and will evolve it to support the ongoing staff needs.

Add new comment

[Getting to] Informed Solutions from the Solutions Bank

admin2 — Fri, 24 Jun 2016 05:21:38 +0000

[Getting to] Informed Solutions from the Solutions Bank admin2 Fri, 06/24/2016 - 01:21

IEG LIVE: [How] Does the World Bank Group Learn...from its Operations? Lessons from the Fifth Discipline

admin2 — Fri, 24 Jun 2016 05:08:11 +0000

IEG LIVE: [How] Does the World Bank Group Learn...from its Operations? Lessons from the Fifth Discipline admin2 Fri, 06/24/2016 - 01:08

Institutionalizing Evaluation: What is the Theory of Change?

ieg_admin — Tue, 29 Mar 2016 14:41:31 +0000

Institutionalizing Evaluation: What is the Theory of Change? ieg_admin Tue, 03/29/2016 - 10:41

Comments

I have truly no comment to make, I want to just appreciate . Thank you .

That's nice. Thank you!

Well put. Institutionalising performance monitoring, evaluation and reporting is an important step in improving (and/or optimising) the achievement of objectives and desired long-term outcomes. Tasmania (Australia) has made significant progress in developing and institutionalising an evidence-based adaptive management approach to its national parks and reserves. For more information and examples of evaluation reports, go to www.parks.tas.gov.au/monitoring.

Excellent news! Thanks for sharing.

I believe that indicators are key parameters to evaluate value for money development program effectiveness performance.

I completely agree with you. Nice piece of well written thoughts. The role evaluation plays in determining relevance and appraising projects' realities is undeniable. However, we must watch that, we don't in any way ploiticize the institutionalization process by focusing on what sells us better, rather than reality of a social good. Very educative and interesting post. Thank you, Ma.

Thank you for the nice feedback, and for the point of keeping the eye on evidence-based decision-making for the greater social good. An important reason for doing evaluation work!

It´s an excellent and useful article

Thank You!

Appreciate your views and approach. Honest independent evaluation tells exactly where does the project stand and in which direction it needs to change its course and move on to reach its destination.

Exactly!

I like your ideas, I will keep them. When reading you, I thought on evaluation as an expression of the institutional culture and independent evaluation as a big piece.

Thank you Rosemary. Appreciated!

This is a very good article. My comment is that for projects to achieve intended objectives, evaluations will need to be demystified. Many a team member see evaluation as a foreign activity and imagine that it should be carried out by people outside the organisation. If evaluations are well institutionalised, reporting performance every so often, e.g. on quarterly basis or even on monthly basis, people get used to preparing evaluation reports. This brings about the intended change in mindset and improves performance.

Good point, James, but our evaluations show that even when there are regular performance reports, they do not lead to the kind of ownership you talk about.

Add new comment

Facing off: Accountability and Learning - the Next Big Dichotomy in Evaluation?

ieg_admin — Tue, 22 Mar 2016 14:27:00 +0000

Facing off: Accountability and Learning - the Next Big Dichotomy in Evaluation? ieg_admin Tue, 03/22/2016 - 10:27

Comments

This is a useful post and a real question. The issue is who uses the evaluation, for what, and how the evaluation can contribute. All too often discussion in Parliament is punitive, blaming the executive for their failures. We need to shift from blaming for mistakes, to blaming for not learning from mistakes. Evaluations contributing to fear will seed the failure of evaluation. Departments are frightened that evaluations will lead to blame. So we have to find ways to make the culture one of how to improve not to blame. Our second evaluation of impact of the reception year of schooling was s classic case. The evaluation said no impact in poor schools in poorly performing provinces. The deputy minister who presented it congratulated the department for the degree of rollout, and then said but impact is not what it should be and we need to improve quality. There followed a lively and constructive debate. If we can engender that approach we can get accountability and learning.

Thank you for the example Ian! You are entirely right "We need to shift from blaming for mistakes, to blaming for not learning from mistakes." Let's all work towards that!

A new paradigm war would be unfortunate at a time when the evaluation community should be pulling together to implement the Global Evaluation Agenda. It would also be unnecessary since we should learn from experience. Just as the methodological divide was bridged when both sides acknowledged that no single method is equipped to answer all evaluative questions it is time to agree on a simple proposition: neither a pure accountability model nor a pure learning model can fit all situations. Thus a properly conceived accountability driven function makes authority responsible for results over which the organisation can exercise control (performance) while organizational learning focuses on continuous improvements in the design of partnership arrangements and internal management protocols so as to enhance effectiveness.. In most cases an admixture between the two functions is desirable and in all cases a judicious balance between self evaluation and independent evaluation should be struck to create the right enabling environment for effective governance. Thus accountability for learning and learning to be accountable converge - two sides of the same coin. This evokes the need for combining accounting with auditing in order to meet sound financial management standards.

Many thanks, Bob. I agree.

I am entirely supportive of the assertion that "evaluation profession should aim for accountability and for learning". In the same vain, I really like the question of who use the evaluation inputs. We should focus on lessons derived from evaluation exercise and what happen then? To ensure that evaluations' main findings are the drivers of better designed and implemented operations we need to solve the issue of countries' ownership on that. That means evaluations findings are disseminated and used for decision making. But, how to make these findings less sophisticated for a great majority of countries experiencing weak capacity on this matter that is reserved to small circle of practitioners? That's the big problem in my sense.

Begnadehi, thank you for your reflections. Yes, taking lessons back to audiences in countries is always difficult for centrally located evaluation offices like IEG. At times, we organize local outreach activities, like the one in Zambia that the WB country director blogged about. But, we need to find more ways to stimulate discussions of evaluation findings in ways that are effective and efficient.

Thank you, Caroline, for raising this issue. Indeed, it should not be ether-or, but how accountability contributes to learning. And, as Ian and Bob have added, accountability for learning from evaluation. I like to promote facilitated self-evaluation using the image of an evaluator holding up a mirror to help the evaluand reflect realistically on how things are going. Rather than using a magnifying glass to examine and judge from an external, 'objective,' 'gotcha' perspective, lacking empathy and the promotion of improvement. The difference between performing on an autopsy on a dead body to determine how it died (pure accountability), and a doctor conducting a medical exam providing the facts to help the individual know what to do to improve his or her health (to promote learning). Indeed in my experience I've seen groups resist evaluation, thinking it only serves the former (accountability) function, not realizing that evaluation can also make significant contributions to learning for improvement.

Jim, good points, though I would not put "objective" in the same category as "gotcha". Objectivity and impartiality are hallmarks of evaluation and drive its commitment to evidence-based assessments. These attributes set it aside from opinion-based assessments, where the expert's personal view is sought rather than triangulated evidence brought together from various sources. Also, I am not entirely sure about the example: isn't an autopsy done to understand why the person died, which would lead to learning for others?

Caroline thank you for your post: it is an interesting dichotomy especially the way you framed it, it seems to me, from within organizations implementing and by commissioners of evaluations. I would extend a question which is, what if we expanded our notion of accountability to accountability to the country nationals who we have pledged to assist? If we facilitated their evaluation of how well our joint investments through projects have furthered their self sufficiency during and long after closeout, what can we learn? How well we valuing their voices to shape the project they are to self-sustain now for long after our projects end? How would development be done differently? Warmly,, Jindra of Valuing Voices

Thank you Jindra. Yes, that is the direction of travel I expect evaluation -- as a profession and a practice -- to evolve towards.

Excellent point raised Caroline, I think when we talk about evaluation for accountability and learning the focus should be on the future rather than looking back (may lead to blaming). It should be transformed into culture of organization to except and respect to the fact that the evaluation will lead to improvement in Programme and foster the performance by looking back the journey and learning of it.

Thank you, Abdul, for the suggestion. "Looking back" is the main thing that we do in evaluation but with a strong commitment to understand and interpret what has happened in the past through the lens of the future. For instance, if there is a new policy, what can we learn from the past (from looking back) to help us make sure we make the right policy choices and implement it well.

Dear Caroline, thank you very much - as usual - for having spearheaded such an important debate on this page. I think that there are two distinct, and both important, reflections to make here. The first reflection pertains, as colleagues have already well put, on an empathic approach to evaluation, one that would naturally blur the line between accountability and learning. The second reflection is related to ‘accountability’ and ‘learning’ from evaluation being too often used/considered as extrinsic incentives to change. Recent research on incentives to lifestyle has stressed that Eudamonia-rewards (intrinsic incentives), which relate to a sense of meaning and purpose, are however more powerful than Hedonic-rewards (extrinsic incentives). So: how can evaluation not only promote learning but provide intrinsic incentives to change, while preserving its independence? If intrinsic incentives are about autonomy (being ourselves), competence (feeling we can make a difference) and relatedness (knowing that we can help and we will be helped), this to me is a call for evaluation – as at least conceived in large international organizations – to be more participatory. This would not only mean a more extensive use of focus groups as methodological tool, but also increased attention devoted to the follow-up of evaluation, so that the report and the knowledge there generated are indeed used to foster change among managers and staff at all levels in the organization.

Dear Anna, many thanks for your thoughtful comment. Looks like your research is going very well! Hope you send me an update (by email) some time. I agree with you in many ways, but also have experience that evaluations that aim to address systemic issues that undermine doing meaningful jobs, one can make a big difference. Once a program manager can see the evaluation is not personal -- not of him or herself -- but of the underlying system that hinders him/her to be successful, the mindset can shift. If then evaluation can demonstrate that it actually can be effective in inspiring these systemic changes, it becomes a powerful and accepted tool.

I am concerned that the term "accountability," as it is used in management practice, carries the connotation of blame. This may make it difficult for accountability and learning to be used together as productive aims of evaluation. In hierarchical organization, power structures sometimes insure that blame is place at the lowest possible level. I am interested in how others think about these factors.

Carey, this is exactly the point here. If we confuse accountability and blame, the institution is stymied. And that does not just relate to evaluation, albeit we might be at the forefront of the discussion of the issue. The blame culture, however, is not (necessarily) caused by evaluation.

This has been quite an interesting issue. Not entirely new within the evaluation sector. However, my take will be that the seeming dichotomy can be solved by ab initio defining what the evaluation is out to achieve. If accountability and learning, then the approach to the evaluation and the results should be seen in that light: How accountable have the resources and results been and what learning can we derive from the outcomes of the evaluation conducted. I therefore agree that it should not be an either or

Thanks, John.

Caroline, This is a really important point. To be an organisation that learns, we need to experiment and take calculated risks. The IT industry has made rapid progress by creating a culture where rapid experimentation (and multiple failures) is seen as the key to success. Evaluation needs to be part of this cycle, both to identify what projects met their goals (accountability) and to learn (why some things worked and why others didn't). This depends heavily on organisational culture - whether the management reaction makes individuals feel threatened by a negative evaluation, rather than it being an opportunity to learn and move to a more productive path.

Robert, I couldn't agree more. In addition, Tuesday's blog of the World Economic Forum speaks about psychological safe space that is needed for people to learn, whether from evaluation or otherwise. A culture of blame, one that does not accept failure as an option, is at the root of this problem. https://www.weforum.org/agenda/2016/04/team-psychological-danger-work-performance?utm_content=buffer358e0&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer

Add new comment

A New Year's Resolution for the World Bank? To be - or become - a learning organization

ieg_admin — Tue, 12 Jan 2016 19:46:44 +0000

A New Year's Resolution for the World Bank? To be - or become - a learning organization ieg_admin Tue, 01/12/2016 - 14:46

Comments

World Bank is known for its champion for fighting poverty where it emerge globally. The Organization does this swiftly and without red tapes. What my experience been as a scholar is the manner in which learning organizations are hierarchical and with unnecessary procedure's which is buearacratical and parochial red tape that no one would wish World bank to go same way. In fact I fully agree with Senge that World bank need be people centred not systems. The systems get enriched from the Knowledge of human beings not system to instruct humans what to do. World Bank being a leader and I mean good leadership need create opportunities like the famous leaders President Obama for example strove hard to create many opportunities globally rather than rushing for solving problems which by the time of his election to president were many. By World bank wanting to be learning Organization, it means they are ruthlessly rushing for quick fixes and sit there feeling self sufficient as an Organization and will let go all other would world nurtured innovation across and from entire Globe.

Many thanks for this blog, Monika. The development community tends to be relatively naive about learning, as if it is something one can do on command. The best learning is done without focusing explicitly on it, as Peter Senge points out. It's good to have this overview of the current state of thinking, drawing upon the World Development Report 2015 and Peter Senge's perspectives. Thinking through how evaluation can contribute to learning remains a major challenge, given the informal nature of learning...

I remember when Peter Senge came to the Bank in the mid-90s, and said the exact opposite. He may have learned, but a combination of recruitment practices, reliance on STCs (of all ages), and 'busyness' has limited progress. This New Year's resolution is like 'lose five pounds',

An excellent article and sound directions Monica. Thanks you for this insightful work. Alaa Garad, Chair and Co-Founder, Organizational Learning Conference MENA Chairman SOL Egypt