Highlights

There is no evidence of sustained improvements in country programs: development outcome ratings have increased, but these gains may not hold, and World Bank Group performance ratings have been flat. Between FY13 and FY20, the percentage of countries with moderately satisfactory or above development outcomes ratings rose from 68 percent to 78 percent. Meanwhile, countries with Bank Group performance rated either good or superior fluctuated between 59 percent and 63 percent.

The stagnation in Bank Group performance is concerning because the Bank Group directly manages it. The Bank Group plays a substantial role in its own performance as it makes choices about in which areas, and how, to engage with and support clients based on its own diagnostics, analytic work, and tools to support implementation.

There is 21 percentage point gap between development outcomes in countries classified as fragile and conflict-affected situations and non–fragile and conflict-affected situations (55 percent and 76 percent rated as moderately satisfactory or above, respectively), but the Bank Group performance is approximately the same in both groups (61 percent and 64 percent, respectively).

Results frameworks in Country Partnership Frameworks have persistent shortcomings that affect their ability to support implementation. Evidence on issues with results frameworks is relevant to the new Scorecard because management is considering ways to cascade the indicators into results frameworks across Country Partnership Frameworks (World Bank 2024b).

Country programs do not always deliver Bank Group collaboration that seeks to offer more complete development solutions to clients. In countries with two Completion and Learning Review Validations between FY13 and FY23, only 40 percent of countries showed collaboration in more than one sector, and collaboration has materialized consistently only in 26 percent of countries.

Relevance, risk identification and mitigation, and support to implementation are the main factors that have strong influences on Bank Group performance in country programs. For example, in 25 out of the 37 countries where Bank Group performance improved or declined, relevance was aligned with the shift in rating.

Adaptive management is relevant for country engagement and needs to be further incentivized. The analysis of Country Opinion Surveys shows that adaptive management factors are rated at a statistically significant lower level for country programs with lower development outcomes and Bank Group performance ratings.

This chapter examines the performance trends of Bank Group country programs in Completion and Learning Review Validations (CLRVs) between FY13 and FY24. We use the term Completion and Learning Review Validation to refer to all IEG validations of self-assessments of country program performance.1 In addition to rating trends, CLRVs delve into the challenges encountered during country program design and support to implementation, exploring their association with the Bank Group performance rating. For each of these analyses, we do not sample but use the population of CLRVs according to our selection criteria (appendix A). To further understand Bank Group performance from a country perspective, we incorporate analysis based on the COS conducted between FY12 and FY23. Typically, client countries will have completed two CLRVs and three rounds of COS within this time frame. Box 5.1 describes the key ratings from CLRVs and the COS that inform this chapter. The analysis also identifies levers within the Bank Group’s control that could enhance ratings. Online dashboards can be accessed and enable interested readers to undertake their own breakdowns of the data (appendix B).

Trends

Development outcomes have risen, but this trend may not hold. In this chapter, “development outcomes” means the share of CLRVs where the development outcome rating is moderately satisfactory or above. Figure 5.1 shows an increase in development outcomes, rising from 68 percent of country programs with a rating of moderately satisfactory or above in FY13 to 78 percent of projects with the same rating in FY20, above the corporate target of 70 percent.2 Although the share of countries with development outcome ratings remains above 75 percent, experience suggests that ratings for the most recent years fall as new CLRVs are validated. For example, all ratings for FY17–21 have fallen since they were first reported in RAP 2022. Consequently, it remains unclear whether the upward trend in recent years will persist because of a challenging global environment characterized by multiple concurrent crises.

Figure 5.1. Country Program Ratings, FY13–23

Image

Share Tweet Facebook WhatsApp Linkedin Email

Figure 5.1. Country Program Ratings, FY13–23

Source: Independent Evaluation Group.

Note: The data are reported with the smoothing approach adopted since World Bank (2020b), in which a CLRV is included in each of the fiscal years covered by the CLRV. A Country Partnership Framework must have closed and its CLR must have been completed before the Independent Evaluation Group produces a CLRV, which often leads to substantial lags. Given the sparse coverage for recent years, we have stopped the analysis of ratings at FY20. The six Organisation of Eastern Caribbean States countries were reviewed together but graphed individually. The total number of included CLRVs is 209. The dashboard that enables further review of country program ratings is available. CLR = Completion and Learning Review; CLRV = Completion and Learning Review Validation; good+ = good or superior; MS+ = moderately satisfactory or higher.

Bank Group performance has remained stagnant, with low ratings observed in approximately 40 percent of country programs. In this chapter, “Bank Group performance” refers to the share of CLRVs where the World Bank Group performance rating is good or superior. Low ratings refer to country programs with Bank Group performance rated fair and poor. Between FY13 and FY20, overall Bank Group performance fluctuated between 59 percent and 63 percent (figure 5.1), consistently below the corporate target of 75 percent.3 Given that this rating captures factors managed by the Bank Group, one would anticipate they could be addressed. Notably, Bank Group performance is more susceptible to change than development outcomes: more countries shift their Bank Group performance ratings (46 percent) than their development outcome ratings (32 percent) in two consecutive Country Partnership Framework (CPF) cycles. Moreover, there is approximately the same number of countries with improving or declining Bank Group performance ratings.

Country program ratings vary significantly among Regions (figure 5.2). The Europe and Central Asia Region exhibits the highest ratings in both development outcomes and Bank Group performance. Conversely, South Asia has high development outcomes (85 percent) but the lowest Bank Group performance, with only 44 percent of countries rated good or superior across the FY13–23 period.

Figure 5.2. Country Program Ratings by Region, FY13–23

Image

Share Tweet Facebook WhatsApp Linkedin Email

Figure 5.2. Country Program Ratings by Region, FY13–23

Source: Independent Evaluation Group.

Note: The data are reported with the smoothing approach adopted since World Bank (2020b), in which a CLRV is included in each of the fiscal years covered by the CLRV. CLRV = Completion and Learning Review Validation; good+ = good or superior; MS+ = moderately satisfactory or higher.

Development outcomes are more strongly associated with country characteristics than Bank Group performance (figure 5.3). There is a 21 percentage point gap between development outcomes in FCS countries (55 percent) and the outcomes in non-FCS countries (76 percent), but Bank Group performance is approximately the same in both groups (figure 5.3). Development outcomes rise alongside income levels, increasing from 59 percent in low-income countries to 88 percent in high-income countries. However, the situation reverses for Bank Group performance, with low-income countries outperforming high-income countries by 23 percentage points. The current evidence base does not allow us to explain these patterns in ratings in depth, although it indicates that some types of risks linked to country characteristics are likely more difficult to mitigate in achieving contributions to development effectiveness. For example, as discussed in chapter 2, operations in FCS encounter more country-level contextual obstacles and institutional capacity challenges.

Figure 5.3. Country Program Ratings in the Country’s Latest Completion and Learning Review Validation

Image

Share Tweet Facebook WhatsApp Linkedin Email

Figure 5.3. Country Program Ratings in the Country’s Latest Completion and Learning Review Validation

Source: Independent Evaluation Group.

Note: Income level and lending group are assigned based on the ending fiscal year of the latest CLRV, whereas FCS status considers all fiscal years of the latest CLRV. CLRV = Completion and Learning Review Validation; FCS = fragile and conflict-affected situations; good+ = good or superior; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; MS+ = moderately satisfactory or higher.

Challenges

Factors with a Strong Influence on World Bank Group Performance

The stagnation in Bank Group performance is concerning because it directly relates to core factors under the Bank Group’s control. The country engagement guidance provides a framework and process for the Bank Group to make choices about in which areas, and how, to engage with and support clients (World Bank 2021c). These choices are based on a range of core diagnostics and tools to support implementation.4 The stagnation of Bank Group performance—despite the wide use of these tools—deserves careful attention. On the basis of a qualitative analysis of 162 CLRVs (the 2 most recent CLRVs for each of the 81 countries with multiple Completion and Learning Reviews validated by IEG in FY13–24), we have identified the four main factors defined in Completion and Learning Review and CLRV methodology that strongly influence,5 with a high degree of confidence, the Bank Group performance rating: relevance of country program, risk identification, risk mitigation, and support to implementation (figure 5.4).

Relevance has a substantial influence on Bank Group performance, both positive and negative. Relevance reflects the “tailoring” of country programs and includes selectivity and framing of CPF objectives, choice of instruments and interventions, adaptiveness, and realism of program design. Using this definition of relevance, 64 percent of CLRVs pointed to good or superior country program relevance (figure 5.4). Moreover, in most countries where Bank Group performance improved or declined, relevance was aligned with the shift in one direction or the other. This alignment was observed in 25 out of the 37 countries with shifts in Bank Group performance. Examples of common relevance challenges identified in CLRVs include the following:

• Lack of selectivity. In addition to this analysis, IEG has consistently identified lack of selectivity in country programs as a key factor negatively affecting outcomes (World Bank 2015e). For example, the Tanzania FY18–22 program had 15 CPF objectives, including all Systematic Country Diagnostic recommendations plus additional government priorities, despite acknowledging capacity limitations.
• Inadequacy in the selection of instruments. Previously, an IEG synthesis of resource-rich countries found that instruments such as risk sharing, guarantees, and credit information did not receive the attention necessary when the credit risk seemed to have been a bigger constraint than the availability of finance (World Bank 2015f). In Bhutan, although the CPF for FY15–19 planned for a combination of lending instruments, the exclusive use of development policy financing during implementation may have compromised results.

Figure 5.4. World Bank Group Performance at the Country Program Level

Image

Share Tweet Facebook WhatsApp Linkedin Email

Figure 5.4. World Bank Group Performance at the Country Program Level

Source: Independent Evaluation Group.

Note: The first bar represents the World Bank Group performance rating as validated by the Independent Evaluation Group in the CLRV. The remaining bars were coded by the Results and Performance of the World Bank Group 2024 team, as elaborated in appendix A. A total of 162 CLRVs are represented, from all 81 countries with at least 2 CLRVs reviewed by the Independent Evaluation Group in FY13–24. Factors are considered to have a strong influence on Bank Group performance if they display a difference significant at the 5 percent confidence level in a t test of equality of means between the subgroups of CLRVs with Bank Group performance rating of good or superior and fair or poor. The dashboard that enables further review of factors linked to country program performance is available. CLRV = Completion and Learning Review Validation.

• Uneven or wavering government ownership of the country program. IEG evidence from operations confirms the importance of developing ownership across multiple interest groups (World Bank 2023a). In Tajikistan in FY15–18, an external shock took the government’s attention away from the ambitious reform agenda. In Mauritius in FY07–15, the initial commitment to the reform effort wavered in response to a newly elected government.

Most country programs did a good job of identifying risks, although sufficient risk mitigation did not always follow (figure 5.4). A review of CLRVs found that 71 percent described good or superior risk identification, whereas 55 percent described good or superior risk mitigation. When country programs were able to adapt to unidentified risks or challenges, they received more positive ratings. Both risk identification and mitigation played a role in countries with changes in Bank Group performance ratings (respectively, in 23 and 28 out of the 37 countries). Further analysis of the materialization of risks in country programs could offer valuable insights into opportunities to enhance performance. Among the reviewed CLRVs, the most common pitfalls regarding risk are as follows:

• Missed or underestimated political risks. With the benefit of hindsight, the Peru FY17–21 CLRV pointed to underestimated political risks, rated moderate both in the original CPF and in the Performance and Learning Review. The CLRV highlights this as a misreading of the political turmoil that was to follow, which had a significant impact on the country program implementation.
• Missed or underestimated capacity risks. In Kosovo in FY17–22, risks associated with institutional capacity constraints were not sufficiently acknowledged during program design. The risks were partly addressed through a portfolio improvement plan, which was underpinned by a thorough analysis of implementation bottlenecks.

Despite the limited depth of discussion of support to implementation in CLRVs, that support is a key factor influencing Bank Group performance ratings. Not all aspects of support to implementation are evenly discussed in CLRVs, which tend to focus on portfolio performance, advisory services and analytics (ASA) delivery, and safeguards and fiduciary issues. Support to implementation was positively assessed in most countries (67 percent) where Bank Group performance improved, with challenges relating to safeguards and fiduciary addressed, as well as marked improvement in the delivery of ASA. Common challenges in supporting implementation include the following:

• Insufficient attention to safeguards and fiduciary issues. During implementation, the Mozambique FY17–21 program faced challenges related to compliance with safeguards, with many difficulties stemming from project implementation units’ low capacity to assess and mitigate environmental and social risks.
• Scattered or poorly delivered ASA. In Costa Rica in FY12–15, the ASA program was not strategic enough nor directly connected to the program, with many large analytic reports not followed up with program-related actions.
• Intermittent staff presence. The lack of a World Bank office constrained the scope and effectiveness of the Botswana FY09–13 and Djibouti FY09–13 programs. Similarly, the absence of a country manager and high staff turnover were challenges for the Mauritania FY07–12 program.

Quality of Results Frameworks and World Bank Group Collaboration: Other Important Factors

The quality of results frameworks, collaboration with development partners, and collaboration within the Bank Group approach are important challenges, even though they have limited influence on Bank Group performance. Our qualitative analysis of 162 CLRVs codified Bank Group performance on three additional factors: quality of results framework, development partner collaboration, and the One World Bank Group approach (figure 5.4). While these factors exhibit a similar distribution across CLRVs—irrespective of whether the Bank Group performance rating is fair or poor or good or superior—they remain critical. These factors are included as criteria within the country engagement guidance’s methodology for assessing country programs. Furthermore, there is recognition in the literature of their role in enhancing project outcomes (World Bank 2015d) and mobilizing additional development resources (Eriksson 2001).

Results frameworks have shortcomings that affect their ability to support implementation. The evidence indicates that a weak results framework is “a key determinant of unsatisfactory outcome performance at the country program level” (World Bank 2015d, 1). Nevertheless, 83 percent of CLRVs reported major inadequacies in results frameworks, which disconnects from the overall Bank performance rating, where 36 percent of country programs were rated weak. Moreover, there has been no substantial improvement; out of 81 countries, 16 improved their results frameworks to a good rating in their most recent CLRV, while 9 declined to fair or below. Frequent shortcomings are noted with the intervention logic and the chosen indicators. For example, indicators may focus on inputs or outputs rather than outcomes, or they may be overly reliant on indicators of operations that fail to capture the full extent of the country program objectives and do not properly account for the contributions of ASA, IFC, MIGA, policy dialogues, or the Bank Group’s convening role. Although Performance and Learning Reviews often adjust the results framework, many weaknesses remain unresolved. Previous IEG reports have raised these concerns and also found that these practices generate incentives not aligned with an outcome orientation at the country level (World Bank 2020d, 2022c).

Insufficient collaboration with development partners still affects some country programs. Duplication of efforts and redundancies between donor programs occur only in 11 percent of country programs. Nevertheless, the Bank Group has historically acknowledged “the weaknesses of uncoordinated aid” (World Bank 1984, 57) and long understood the barriers that impede effective donor coordination (Eriksson 2001). Thus, it is reasonable to strive for effective collaboration between the Bank Group and development partners in all countries.

The intent to undertake Bank Group collaboration and offer more complete development solutions to clients is a feature of country programs. The intent to exploit synergies between Bank Group institutions to respond to client demands through enhanced collaboration at the country level has been sought over almost three decades. In 1996, the first joint country assistance strategies were defined; in 2013, the One World Bank Group strategy was released; the cascade approach was introduced in the 2017 IFC 3.0 strategy; and the Maximizing Finance for Development strategy was introduced in 2018. Currently, joint country representation has been introduced as part of the Better Bank initiatives to help enhance collaboration. Four types of Bank Group collaboration were identified within sectors in CLRVs: parallel and complementary, joint projects, sequenced interventions, and separate but coordinated work. In the reviewed CLRVs, an intent for collaboration among Bank Group institutions in one of these four types was identified in 90 percent of country programs in at least one CPF period. When this intent is realized, IEG evaluations have found that Bank Group collaboration can improve the performance of sectors in country programs because they are better able to address client needs with the tools and expertise of all three institutions at their disposal (World Bank 2016d, 2020a).

Bank Group collaboration has increased in country programs in a limited number of sectors and inconsistently over CPF periods. A review of CLRVs showed that for CPFs starting before FY16, less than half of country programs demonstrated collaboration in at least one sector. Since FY16, 28 out of 33 country programs undertook collaboration on interventions. Of these, just under two-thirds (18 out of 28) demonstrated collaboration with more than one sector. This collaboration mainly occurs in five sectors: the energy, financial, infrastructure, investment, and agriculture sectors account for just over three-quarters of the instances of collaboration. Moreover, collaboration as One World Bank Group has yet to arise consistently in countries across two CPF periods. Across the FY13–23 period, collaboration has materialized consistently only in 21 out of the 81 countries that had two Completion and Learning Reviews validated by IEG. The different forms of Bank Group collaboration, their instruments, and the sectors are discussed in more detail in appendix J.

Collaboration can occur even without joint financing. Collaboration occurs even when IFC and MIGA have small portfolios relative to the World Bank’s—that is, financing is not always necessary. Figure 5.5 shows that outside the countries with the lowest share of MIGA and IFC commitments, more than half have examples of collaboration in Bank Group activities. A review of CLRVs found that 30 percent of the examples discussed involved joint financing. The remaining 70 percent of collaboration examples involved financing from one Bank Group entity and advisory services (for example, advisory services from IFC or ASA from the World Bank) from another or involved purely ASA. This finding suggests that countries can undertake Bank Group collaboration even when commitments from IFC and MIGA are low. IEG evidence suggests that collaboration can be enhanced through an alignment of objectives between Bank Group institutions and a clear understanding of priorities (box 5.2).

Figure 5.5. World Bank Group Collaboration at the Country Program Level

Image

Share Tweet Facebook WhatsApp Linkedin Email

Figure 5.5. World Bank Group Collaboration at the Country Program Level

Source: Independent Evaluation Group.

Note: The data are based on the review of 162 CLRVs by the Results and Performance of the World Bank Group 2024 team (all 81 countries with at least 2 CLRVs reviewed by the Independent Evaluation Group in FY13–24). The first bar has all CLRVs, while the remaining bars split country programs according to the relative size of IFC and MIGA programs as quartiles of the ration between IFC and MIGA commitments and total World Bank Group commitments. Country programs that were exclusively the International Development Association and the International Bank for Reconstruction and Development programs are marked as not applicable. CLRV = Completion and Learning Review Validation; IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency; Q = quartile.

Levers

Extensive adaptive management practices can be found in CLRVs that often positively describe links to performance. IEG had previously identified several types of adaptive management used during country programs (adapted from World Bank 2020d), with some examples found in CLRVs for country programs closed between FY20 and FY23:

• Changes in practices or portfolio composition as context responsiveness. This is the most frequent adaptation (identified in 84 percent of the recent country programs). The North Macedonia FY19–23 Performance and Learning Review introduced a new objective to strengthen the program’s focus on supporting private sector competitiveness, innovation, and resilience; activated the Contingency Emergency Response Component; and approved the Emergency COVID-19 Response Project to support efforts to meet the challenges of the pandemic.
• Attention of staff and management to specific projects and actions to resolve problems during implementation. This was the second most frequent adaptation (identified in 72 percent of the recent country programs). As a response to project delays in the Kosovo FY17–22 program, the World Bank, for example, increased attention to problem projects and restructured complex ones, enhanced fiduciary support through hands-on guidance and training, and strengthened portfolio monitoring and review meetings with the client.
• Resource allocation shift during implementation in response to client dialogue. The third most cited adaptation was identified in 70 percent of the recent country programs. The Peru FY17–21 program shifted from a relatively small portfolio focused on IPF to an ample program based on development policy financing to support reforms for economic recovery and to respond to the changing client demands after the COVID-19 outbreak.
• Results reporting and organizational learning. The fourth most cited adaptation was identified in 38 percent of the recent country programs. In the Uruguay FY16–20 program, the Bank Group adjusted two objectives to sharpen its focus and better reflect the government’s support for the climate action agenda.

Client country perceptions indicated that adaptive practices are associated with Bank Group performance and development outcome ratings. Respondents to the COS provided perceptions that are linked to adaptive practices such as staff accessibility, responsiveness to country needs, and flexibility as circumstances change. When country programs had lower Bank Group performance and development outcomes, COS respondents averaged less favorable perceptions of these adaptive practices (table 5.1). In addition, adaptive processes are consistently identified as important to CPF design and implementation (World Bank 2020d, 2021c). This would suggest that focusing on adaptive practices to better meet country needs can also help improve ratings.

Teams can improve Bank Group performance by enhancing the way they use adaptive management, yet it is not incentivized in the country engagement guidance. According to The World Bank Group Outcome Orientation at the Country Level, Bank Group country teams practice adaptive management, but the country-level results system does not effectively support them in doing so (World Bank 2020d). Instead of using the tools in the country engagement model (for example, Performance and Learning Review and the CPF results framework), tacit knowledge, professional experience, and professional networks are relied on when making adaptive decisions. In the current country engagement guidance, adaptive practices are distributed across several factors considered in rating Bank Group performance. Moreover, the guidance does not specify beneficial approaches or provide explicit incentives for improving the focus on adaptive processes. In light of this evidence, defining incentives and describing important types of adaptive management could enhance Bank Group performance.

Source: Independent Evaluation Group, based on Country Opinion Survey data collected between 2012 and 2022.

Note: All scores are measured with the Likert scale: 1 = to no degree at all; 10 = to a very significant degree. Survey questions are reported as averages based on country years that match the 162 CLRVs, coded by the Results and Performance of the World Bank Group 2024 team, split by their Bank Group performance rating. The table reports the means, the difference between groups, and the one-sided p value in a t test of equality of means between the subgroups of Bank Group performance in the CLRV rating of good or superior and fair or poor. Rows are ordered by the gap between groups. The results from splitting country programs by their development outcome ratings, rather than Bank Group performance ratings, are similar. Significance level: * = 10 percent; ** = 5 percent; *** = 1 percent. CLRV = Completion and Learning Review Validation.