Back to cover

The World Bank Group Outcome Orientation at the Country Level

Overview

Highlights

This evaluation assesses whether the World Bank Group’s country-level results system adequately supports the organization’s monitoring, evaluation, and learning needs to effectively manage country engagements. The evaluation does not assess the project-level results system.

The model of how the Bank Group aims for outcomes in its client countries is sound, and, for the most part, teams practice this well.

However, the country-level results system does not capture the Bank Group’s contribution to country outcomes well, because it relies on results frameworks premised on metrics, attribution, and time-boundedness that do not fit the nature of country programs well.

Country teams practice adaptive management, but the country-level results system does not effectively support them in doing so. Instead they rely on tacit knowledge, professional experience, and professional networks when making adaptive decisions. This has some disadvantages.

This evaluation provides a new vision of how the country-level results system’s accountability principles, tools, and incentives could function.

The World Bank Group’s success rests on its ability to help its clients overcome challenges to achieve the development outcomes they desire. This demands a strong outcome orientation at the country level, defined as the organization’s ability to generate feedback on what works, what does not, and why, use this feedback to engage clients and adapt country programs, and ultimately bolster its contribution to country development outcomes. The Bank Group uses the country engagement cycle and its underlying results system to support the organization’s monitoring, evaluation, and learning (MEL) needs to effectively manage country engagement. As part of this cycle, country teams are expected to aim to make meaningful contributions to country development outcomes, to capture those contributions in a results system, and to use information from the results system to inform adaptive management.

This learning-focused evaluation aims to help country teams and management strengthen the Bank Group’s country-level outcome orientation. It does this by focusing on how country teams carry out four key practices of outcome orientation. These practices include the following: (i) aim for outcomes in the Country Partnership Framework (CPF) and CPF results framework; (ii) monitor and evaluate the Bank Group’s contribution to country outcomes and learn from this evaluative evidence; (iii) adapt country engagements; and (iv) engage clients to measure and manage outcomes. The evaluation does not assess the project-level results system. The evaluation is based on a sample of 39 country engagements selected to represent the diversity of Bank Group country clients and portfolios, and draws on a combination of content analysis of country documents, interviews with over 185 Bank Group staff and clients, and a review of practices and literature on evaluation systems at the Bank Group and other organizations.

Aiming for Outcomes

The evaluation finds that the model of how the Bank Group aims for outcomes at the country level is sound and, for the most part, country teams practice this model well. The Bank Group’s country strategies set out to influence country outcomes and frame their objectives in terms of outcomes, while facing difficulties in being selective because of internal competition for program space and heterogeneous client demand. The Bank Group pursues its objectives through both direct pathways (such as the outputs and services provided by projects) and indirect pathways (including institutional development, capacity building, knowledge transfer, demonstration effects, and market creation). About one-third of CPF objectives coded for this evaluation explicitly aimed at building institutional capacity within a sector. However, these indirect pathways are not well articulated in country documents, and the relatively short CPF cycles do not match the long-term vision needed to sustain indirect pathways. There has been progress in integrating International Finance Corporation (IFC) and Multilateral Investment Guarantee Agency teams into the CPF design, but CPFs do not serve IFC’s strategic needs well. This motivated IFC to launch its own internal country strategies and other country products. Early experience with these IFC country strategies shows that they offer potential for increasing IFC’s focus on country outcomes, but close alignment with the World Bank and Bank Group country engagement is a condition of success and to avoid risks of duplicated efforts.

Capturing Outcomes

The Bank Group falls short of capturing its contribution to country outcomes because of the limitations of the country-level results system. The results system relies on the use of CPF results frameworks as its primary tool for assessment predicated on three principles: metrics, attribution, and time-boundedness. These three principles present critical challenges when applied to country programs.

Metrics. CPF results frameworks are built predominantly on quantitative indicators, but these indicators do not capture the CPF objectives being pursued. Newly collected data show that about half of the time these indicators are framed below the level of the objectives they seek to track. Some outcomes, such as improved governance, are more intangible and inherently more difficult to measure. Even for outcomes where meaningful metrics exist, data gaps in countries pose serious challenges.

Attribution. Teams are reluctant to—and are advised not to—include indicators that are not directly attributable to Bank Group activities, making it difficult to capture the Bank Group contribution to country outcomes and outcomes pursued through indirect pathways.

Time bound. The country-level results system is time bound and does not capture the medium- to long-term effects of Bank Group interventions. It also largely captures results from operations launched in the previous strategy period, rather than the new forward-looking strategy.

Most indicators come from projects because teams can have confidence that the data will exist and will be attributable to the Bank Group’s activities, and they can collect the data without additional evidence-gathering expenses.

There are direct and opportunity costs associated with compiling, processing, controlling quality, and reporting on indicators that are not used.

The Bank Group’s country evaluation product, the Completion and Learning Review (CLR), provides a partial picture of Bank Group contributions. The picture is partial because of its overemphasis on those results that can be measured and on results from lending projects. The CLR rarely captures complementarities across instruments or institutions and so is not able to establish whether the Bank Group’s contribution to country outcomes amounts to more than the sum of its parts. Because of their timing and limited content, the reviews are little used within the institution. Staff almost invariably argue that the accountability elements of the CLR lack any actual bite, in part because they are rarely discussed or used by senior management or the Board to create incentives and make decisions.

Although some country teams carry out other evaluative exercises, they are rare and overshadowed by the results system’s emphasis on indicators and targets. There are many examples of activities that seek to assess the outcomes of interventions embedded in sector work, but these are not well captured in country reports. The Bank Group’s monitoring and evaluation staff are consumed by corporate reporting requirements with little time to support country teams’ outcome measurement.

Managing for Outcomes

The country-level results system does not provide adequate support to country teams engaging in adaptive management. Country teams practice adaptive management by closely monitoring the health of the country portfolios focused on disbursements, addressing delivery issues, and navigating changes in country contexts. However, teams rarely use the country-level results system to make adaptive management decisions because they do not find the information to be useful for showing whether the country is progressing or not, and because the information is not up to date and has to be gathered manually. Instead, country teams rely on tacit knowledge, professional experience, and advice from professional networks when making adaptive decisions. This reliance has some disadvantages.

Country strategies’ midterm review instrument, the Performance and Learning Review (PLR), is used largely to document decisions that were already made and adjust the results frameworks. More than 80 percent of CPF indicators reviewed in this evaluation were revised or dropped at the PLR stage. However, the extent to which teams use the PLR for collective reflection and adaptive management depends on whether country team leaders make it a priority and choose to go beyond the requirements of the system.

Clients do not engage much in the Bank Group’s country-level results system, and the World Bank seldom helps countries develop their own monitoring, evaluation, and results management systems. Moreover, there is little coordination of results management activities among development partners.

Toward a Stronger Outcome Orientation

This evaluation provides a new vision of how the country-level results system’s accountability principles, tools, and incentives could function. It will be important not to repeat the mistakes of the past, which have doubled down on attribution and added more performance measures and indicators that have not been useful. In renewing the system, trade-offs will inevitably emerge between external reporting versus internal use, standardization versus customization, centralized versus decentralized models, and comprehensive versus selective approaches.

The current country-level results system relies on specific principles of accountability that do not fit well with the way the Bank Group pursues outcomes at the country level. The system is premised on a narrow view of accountability that equates it with counting results that can be translated into metrics and ratings, yet staff do not find that the system substantially drives their incentives.

A renewed country-level results system could conceive accountability differently, based on evidence of achievement and failures and descriptions of learning and adaptation. It could acknowledge that the Bank Group can influence but not control country outcomes, recognizing that country teams cannot decide all targets and objectives at design but must adapt during implementation, and realizing that capturing contributions to country outcomes and assessing cumulative effects from multiple interventions requires dedicated evaluation inquiries, not just measurement of indicators.

The country-level results system is too reliant on results frameworks and could benefit from introducing new approaches such as MEL plans. Under these plans, approaches would change in the following ways:

  • Monitoring approaches would focus on selectively tracking critical country outcomes to assess progress on objectives and to support adaptive management.
  • Evaluation approaches would focus on the Bank Group’s contribution to country outcomes, scale back ratings, and emphasize selectivity over coverage.
  • Learning approaches would focus on creating safe spaces for teams to engage and discuss evaluative evidence.
  • Changes in the results system would need to be supplemented by changes in signals and incentives to center on results achievement and management rather than overemphasizing approvals, commitments, and output delivery. Incentives could better match staff’s intrinsic motivations.

This evaluation also makes the following concrete recommendation:

Recommendation

The Bank Group should reform the country-level results system to ensure that it accurately captures the Bank Group’s contribution to country outcomes and usefully informs decision-making on country engagements. This would entail keeping certain practices, discontinuing others, and introducing new ones.

What to keep. The country engagement should continue to articulate clear outcome-level objectives and lay out the various pathways that will be pursued to achieve them. Periodic reviews to take stock of progress, achievements, and failures should be maintained, as well as an end-of-cycle review of evaluative evidence and learning informed by this evidence.

What to discontinue. The current requirements of developing a CPF results framework with a complete set of measurable indicators and targets at the design stage, making adjustments at midterm, and rating country programs on how far achieved metrics departed from initial targets should be discontinued.

What to introduce. Instead of the country results framework, adopt a new approach and enable country teams to develop a MEL plan at the CPF design stage that can be revised throughout the country cycle. A monitoring plan would describe how the country team will monitor performance, changes in key country outcomes that the Bank Group seeks to influence, and relevant contextual risks and conditions. An evaluation plan would define the criteria for selecting and using evaluations and would synthesize existing evaluative evidence. A learning plan would identify knowledge gaps to be filled and plan activities for reflection.