Back to cover

A Bridge to the Future

Chapter 5 | The Piers Supporting Evaluation

Several programs operate in the background but serve as the backbone of IEG’s evaluation practice. IEG invests in the development of cutting-edge evaluation methods through its Methods Advisory Function, in ensuring the strength of Bank Group accountability through the MAR, and in staff learning through the IEG Academy.

Methods Advisory Function

IEG’s Methods Advisory Function is critical to enabling teams to use cutting-edge and rigorous evaluation methods. Indeed, a meta-evaluation to assess the credibility of IEG’s evaluation studies found that over the past five years, clarity of evaluation design has improved and more innovative methods are being used. In FY21, the Methods Advisory team held workshops on adapting evaluations to restrictions imposed by the pandemic.

The Methods Advisory Function also released the Evaluation of International Development Interventions: An Overview of Approaches and Methods, a key resource for evaluators to learn about approaches and methods.

Image
Algorithms and Text Analysis Inform Migration Lessions At the request of World Bank management, IEG developed a summary of lessons, drawing from relevant Project Performance Assessment Reports and thematic and corporate evaluations completed over the past 10 years, for a technical briefing on economic and voluntary migration. We prepared the brief using a combination of machine learning algorithms and qualitative text analysis, which management acknowledged as an innovative method that could provide a model for portfolio analysis of cross-cutting projects. The algorithm identified relevant segments of text and sorted these into topics, and the qualitative text analysis then refined these topics and identified cross-cutting lessons on performance. The text described four interconnected lesson themes that consistently arose across the evaluations: (i) targeting of appropriate groups, locations, and time periods; (ii) the ongoing development of client, World Bank, and private sector capacity; (iii) the availability of granular analysis; and (iv) coordination based on comparative advantage. The brief sparked interest beyond the original audience because it discusses the performance of projects in the context of migration. This provides a broader lens to target success factors in projects focusing on social protection, rural development, and food emergencies. Stephen Porter extendedterm consultant, Office of the Director-General Kristin Strohecker lead knowledge management officer, Knowledge and Communications Harsh Anuj data scientist, Methods Advisory Function Yingjia Liu consultant, Knowledge and Communications

Management Action Record

The Management Action Record (MAR) reports on action catalyzed by evaluations and is a key element of the Bank Group’s accountability framework. Early in FY21, the Committee on Development Effectiveness endorsed a reform of the MAR. Working together, IEG and Bank Group management adjusted the system to focus on strategic recommendations with both the clear potential to enhance the development effectiveness and the strong commitment of Bank Group senior management. IEG is committed to a more selective approach to making recommendations that is driven by the evidence and where each recommendation presents a compelling case for enhancing development effectiveness. Bank Group management will commit to taking actions that advance the institution toward IEG recommendations’ intended outcomes. The reformed MAR enables better and more meaningful tracking, dialogue, and self-assessment of Bank Group management’s implementation of IEG recommendations.

Table 5.1. Management Action Record Before and After Reform

The Prereform MAR System

The New MAR System (starting FY21)

IEG evaluations contained on average five recommendations linked to evaluations’ findings.

IEG evaluations contain fewer recommendations, and those recommendations align more closely with the Bank Group’s strategic directions and have the clear potential to enhance the Bank Group’s development effectiveness.

Some recommendations focused on specific actions, and some contained subrecommendations or multiple parts.

Recommendations are outcome oriented and rarely contain subrecommendations.

Note: MAR = Management Action Reform.

Image
Implementation Completion Report Training as a Bridge between Evaluation and Operations Implementation Completion Reports and their review are important for maintaining accountability at the Bank Group. Although the project teams and IEG staff follow harmonized guidelines for ratings, these guidelines can be interpreted differently given the differences in experience and institutional placement of those making the ratings. In working with staff in the Energy and Water Global Practices (GPs), we received feedback that led us to pilot training workshops customized to the counterparts’ own projects. Other teams have successfully implemented similar approaches. The trainings used case-based material from sector project experiences, which allowed participants to immediately connect the training to their work. We encouraged sharing of expertise and experiences, which helped create a bridge of understanding between GP staff with field experience and the evaluators experienced in applying evaluation methodology to the harmonized guidelines. GP management acknowledged the importance of the tailored approach. At IEG, we can continue to build relationships and understanding between evaluators and operations staff to improve performance. Ramachandra Jammi senior evaluation officer, Infrastructure and Sustainable Development Micro Unit

IEG Academy

Image

The IEG Academy focuses on building and reinforcing the staff skills needed to carry out IEG’s mandate successfully. The FY21 Learning Days delivered 16 short, interactive learning sessions over four dedicated mornings to over 110 IEG staff and consultants in a fully virtual format. While addressing select learning needs of staff in all roles, the program focused on evaluation methods topics (such as Machine Learning for Economists, in collaboration with the IEG methods team), writing, and training on key soft skills such as unconscious bias and emotional intelligence.

In addition to the synchronous Learning Days, IEG Academy delivered learning through these offerings and modalities in FY21:

  • Virtual classes and brown-bag lunches (49)
  • On-the-job tools and coaching
  • A pilot using virtual reality for staff to practice difficult conversations about selected workplace scenarios in a safe space
  • An award-winning mentoring program, which supported pairs matched in FY20 who elected to continue working together remotely
  • Onboarding guidance, resources, and training to orient new hires

Image
Supporting the New Knowledge Framework Since the 1990s, the World Bank has sought to improve its approach to knowledge but has never quite reached its potential. In 2020, two IEG staff members were able to provide data and evidence to inform a new take on this longstanding concern. When a working group comprising experts from across the institution collaborated to draft a new knowledge framework for the World Bank, IEG provided insights from past evaluations on weaknesses in previous approaches to knowledge. IEG’s findings on knowledge, learning, knowledge flow, data for development, and knowledge management provided a unique source of quotable evidence to make the case for reform. As the new knowledge framework moves toward implementation, IEG will continue to independently evaluate related operations. IEG has tentative plans for evaluations of learning from operations and operational knowledge feedback loops (FY24) and knowledge flows and collaboration (FY25). Brenda Barbour manager, Knowledge and Communications Rasmus Heltberg lead evaluation officer, Office of the Director-General