The World Bank Group’s success rests on its ability to help its clients overcome challenges to achieve the development outcomes they desire. The unprecedented health and economic crisis prompted by the coronavirus pandemic (COVID-19) has put into sharper focus the importance of well-functioning and flexible country engagements to the Bank Group’s success. In this context, the Bank Group’s range of financing and knowledge instruments are called on to propose solutions to the crisis’s multidimensional nature and to help country clients’ transition to recovery. These solutions must have a long-term vision that taps into the Bank Group’s global knowledge while being adapted to the specific context of each country. Responding effectively to these types of crises and other development challenges demands a strong country-level outcome orientation, defined as the organization’s ability to generate feedback on what works, what does not, and why; use this feedback to engage clients and adapt country programs; and ultimately bolster the Bank Group’s contribution to country development outcomes. A country-level results system that adequately supports adaptive decision-making is a key element for achieving this outcome orientation.

The Bank Group uses the country engagement cycle to organize how it manages and measures its contribution to country outcomes. The country engagement cycle begins when the Bank Group conducts a Systematic Country Diagnostic (SCD), which it uses to define priorities and objectives jointly with its clients in Country Partnership Frameworks (CPFs). The World Bank’s country team adapts the CPF throughout the country engagement cycle and documents these adaptations in the Performance and Learning Review (PLR). The country team assesses the results of the country engagement in Completion and Learning Reviews (CLRs), which is intended to help determine the Bank Group’s future position in the country. This process is shown in the central part of figure 1.1. It is through the country engagement cycle that country teams organize, manage, and adapt their interventions and client dialogues for best results. At the country level, the Bank Group’s value lies in its capacity to deploy, combine, or sequence a wide range of instruments from each of its component institutions to help client countries eradicate poverty, increase shared prosperity, and achieve the Sustainable Development Goals. The Bank Group’s contribution to country outcomes comprises all these instruments plus informal advice, policy dialogues, stakeholder convening, country teams’ networks and know-how, and the day-to-day interactions between Bank Group staff and clients. As such, country engagement outcomes are meant to be more than the sum of their project inputs. It is at the country level that the clearest picture of the Bank Group’s development impact should emerge and coalesce.

The Independent Evaluation Group (IEG) conducted this evaluation to help ensure that country-level results systems adequately support the outcome orientation of country engagements. IEG did not conduct this evaluation to assess the project-level results system or the Bank Group’s achievement of country-level outcomes, which other IEG instruments, such as the Country Program Evaluations and the Results and Performance of the World Bank Group (RAP), do. This evaluation is designed to enhance the Bank Group’s results accountability, informed risk taking, and organizational learning. The evaluation provides insight into whether country-level results systems work as intended, which is to provide useful, timely, and credible information on the Bank Group’s performance and contribution to country outcomes. The country-level results system is meant to serve two purposes: (i) upholding results accountability to clients, the Board, and senior management, and (ii) informing country teams’ decision-making, learning, and adapting. There can be tensions between these dual purposes, and this evaluation examines how country teams navigate these tensions. The country-level results system is made up of tools (such as the CPF results framework and the CLR, and its validation by IEG in the Completion and Learning Report Review [CLRR]), the data and evidence that compose these tools, and the rules, norms, routines, behaviors, and organizational processes that shape how the system works in practice (Højlund 2014; Leeuw and Furubo 2008; Rist and Stame 2006). The country-level results system sits between project-level reporting—which is embodied by results and performance information for investments, guarantees, advisory services, and discrete lending operations—and Bank Group–wide aggregate reporting, which is embodied by Corporate Scorecards and International Development Association (IDA) results frameworks. Both project- and corporate-level reporting are outside the scope of this evaluation.

Figure 1.1. Conceptual Framework

Image

Source: Independent Evaluation Group.

Note: The country engagement is meant to work as follows: First the World Bank Group sets out to diagnose the biggest development constraints faced by their clients through a Systematic Country Diagnostic (SCD). Second, on the basis of the SCD findings, the Bank Group circumscribes specific contributions it will make to country development outcomes, at the intersection of the Bank Group’s comparative advantage, what others are doing, and the clients’ demands and priorities. The Country Partnership Framework (CPF) establishes these priorities, makes explicit the Bank Group’s objectives and the theory of change that links the Bank Group’s activities to the desired outcomes, and translates them into a results framework. Third, through a Performance and Learning Review (PLR), the Bank Group takes stock of progress and decides on needed changes and course correction. Fourth, at the end of the cycle, the Bank Group undertakes a Completion and Learning Review (CLR), a self-assessment of whether the intended objectives were achieved and of the lessons emerging from the cycle. The CLR is then validated by the Independent Evaluation Group.

Share Tweet Facebook WhatsApp Linkedin Email

Figure 1.1. Conceptual Framework

Source: Independent Evaluation Group.

Note: The country engagement is meant to work as follows: First the World Bank Group sets out to diagnose the biggest development constraints faced by their clients through a Systematic Country Diagnostic (SCD). Second, on the basis of the SCD findings, the Bank Group circumscribes specific contributions it will make to country development outcomes, at the intersection of the Bank Group’s comparative advantage, what others are doing, and the clients’ demands and priorities. The Country Partnership Framework (CPF) establishes these priorities, makes explicit the Bank Group’s objectives and the theory of change that links the Bank Group’s activities to the desired outcomes, and translates them into a results framework. Third, through a Performance and Learning Review (PLR), the Bank Group takes stock of progress and decides on needed changes and course correction. Fourth, at the end of the cycle, the Bank Group undertakes a Completion and Learning Review (CLR), a self-assessment of whether the intended objectives were achieved and of the lessons emerging from the cycle. The CLR is then validated by the Independent Evaluation Group.

The evaluation focuses on how country teams carry out four key practices of outcome orientation. These four practices include how country teams (i) aim for outcomes in the CPF and CPF results framework; (ii) monitor and evaluate the Bank Group’s contribution to country outcomes and learn from this evaluative evidence; (iii) adapt country engagements; and (iv) engage clients to measure and manage outcomes. The evaluation deliberately focused on country teams as the “frontline” agents tasked with making the country engagement model work in practice. A country team refers to the core staff working on strategy and operations for a Bank Group country engagement. This team includes the Country Management Unit (CMU), particularly the country director, country manager or resident representative, operations manager, country officers, country program coordinators, and key sector staff, including practice managers and program leaders who drive sectoral strategic direction. The country team also includes International Finance Corporation (IFC) staff, particularly the IFC country manager and IFC country officers.

Methods

The evaluation’s conceptual framework focused on the organizational conditions that enable country teams to manage for outcomes. The scientific literature identifies three broad categories through which country teams can use well-functioning results systems to inform adaptive decisions and make strategic course corrections (for example, ADB 2012; Dahler-Larsen 2011; Leeuw and Furubo 2008; Højlund 2014; OECD 2019; OECD and World Bank 2014; and World Bank 2016a, 2017d.) These categories include individual motivation, team capabilities, and management practices, all of which are featured at the bottom of figure 1.1. Individual motivation and incentives matter significantly. In international development, the staff’s intrinsic motivation to achieve results must be reinforced by extrinsic incentives, such as “carrots, sticks, and sermons” (Bemelmans-Videc, Rist, and Vedung 1998; Levine and Savedoff 2006). The staff’s autonomy to use outcome information and make decisions to adapt interventions and portfolios is also essential (Honig 2019; Ramalingam, Wild, and Buffarid 2019). Staff incentives and autonomy are largely a consequence of management practices, which are influenced by organizational rules, signals received from the authorizing environment, and managers’ leadership. Team capabilities, including staff skills, methods, knowledge, learning support, and networks, are also central to building outcome-oriented behaviors (OECD 2019; Vähämäki and Verger 2019).

The evaluation carefully selected a sample of 39 country engagements to generate evidence from the Bank Group’s diversity of clients and portfolios. These were enough cases to enable comparisons within categories and to identify challenges or opportunities that cut across country contexts and engagements. Among the 39 countries, the evaluation collected evidence from interviews for 10 countries, from desk reviews for 15 countries, and from both for 14 countries. IEG also planned to visit six countries to interview additional clients and partners, but two of these trips were canceled because of the COVID-19 outbreak. The evaluation’s detailed methodology is presented in appendix A.

The evaluation team undertook three types of data collection and analysis activities. These activities included the following:

• In-depth interviews: First, the evaluation team completed in-depth interviews with country teams. These interviews were essential for understanding how country teams interact with country-level results systems, what challenges they face in so doing, how teams make adaptation decisions, and how incentives and management signals influence staff behavior. IEG interviewed clients and partners in four countries, supplemented by interviews with Bank Group management in Regions, Operations Policy and Country Services, IFC, and the Multilateral Investment Guarantee Agency (MIGA). In total, the evaluation team conducted 185 interviews. Interviews focused on the sampled countries but included the interviewed staff’s experiences from other countries as well.
• Country product reviews: Second, the evaluation team conducted a structured review of 65 country products for 29 countries to understand what is being reported externally in the country-level results system. The team used a classification that was developed for the RAP 2020 to identify how country teams frame and measure objectives in results frameworks. IEG also analyzed PLRs and CLRs to identify types of country engagement adaptations and what evidence and justifications country teams use for these adaptations in formal documents. To help do this, IEG adapted a typology of portfolio-level adaptive management practices developed by the Global Learning for Adaptive Management initiative.
• Literature reviews: Third, the evaluation team conducted literature reviews to identify commonalities and idiosyncrasies of the Bank Group’s country-level results system and to identify ways to improve on it. The team reviewed 16 evaluation reports from IEG and the World Bank and 80 external reports from other development partners on results-based management. IEG used the external reports to understand adaptive management practices carried out by governments, the private sector, and philanthropic groups over a range of organizational contexts.

The report presents evidence and findings on how the Bank Group aims for, captures, and manages country-level outcomes. Chapter 2 reviews how country teams set objectives, define development pathways, and generally aim for outcomes in their country engagements. Chapter 2 finds that the Bank Group’s country-level outcome model is sound and country teams use this model well, with some room to be more selective and work better across Bank Group institutions. Chapter 3 examines how country teams measure the Bank Group’s influence on countries’ development outcomes. It finds that the country-level results system does not properly represent the Bank Group’s influence because of the system’s overreliance on results frameworks, which are unfit for this purpose. Chapter 4 reviews how country teams practice adaptive management and whether country-level results systems support them in making course corrections. It finds that country teams practice adaptive management in several ways: they closely monitor the health of country portfolios, they take action to address delivery issues, and they navigate changes in country contexts. However, country teams barely use country-level results systems to inform their adaptation decisions. Chapter 5 lays out a way forward for country programs to improve their outcome orientation. It makes specific suggestions for reforming the country-level results system so that it is better fit for this purpose. The report’s appendixes present the evaluation team’s underlying analyses.