Management of the World Bank welcomes the Independent Evaluation Group (IEG) report Results and Performance of the World Bank Group 2023 and thanks the team for incorporating comments provided. This is the 13th annual Results and Performance of the World Bank Group (RAP) report, and management welcomes the overall positive findings of performance at the project level and the various steps taken to address operational disruptions during the COVID-19 pandemic. The report’s findings provide valuable insights for both project preparation and adaptive management during implementation. Management also welcomes the institution-specific suggestions for the future.

World Bank Management Comments

Overall Comments

Management is pleased with the overall findings of the report, including the improvements across all three elements of the RAP analysis—ratings of outcomes, Bank performance, and monitoring and evaluation (M&E). The average project outcome ratings are now at their highest since fiscal year (FY) 2012. This is noteworthy as the overall size of World Bank commitments is bigger than at any time in the past—the World Bank delivered record financial commitments in FY23: 66 percent over the precrisis average during FY14–19. This suggests that the incremental delivery has happened in parallel with staff attention to quality of projects under implementation. Bank performance ratings for development policy financing (DPF) projects, entailing both design and implementation, have improved since FY21 and are at the highest since FY12. At the same time, investment project financing (IPF) projects and Programs-for-Results projects have maintained their ratings between FY21 and FY22, including for quality at entry and quality of supervision. Efficacy ratings of intended development outcomes have shown statistically significant improvements during recent years. M&E quality ratings have shown consistent improvements, including notably in countries affected by fragility, conflict, and violence (FCV). While year-on-year changes are important, it is of note that the overall trend in recent years across most indicators used in the RAP has been positive and steady.

Management welcomes the report’s recognition of the effectiveness of proactive measures undertaken in response to the COVID-19 pandemic. These measures included project restructuring and adaptive implementation, which helped improve project performance—the report notes that “restructurings increased from an average of 1.9 per project in the prepandemic cohort to 2.6 in the RAP 2023 cohort” (32). The report’s validation of the benefits of timely project restructuring is particularly useful as teams consider measures to continually focus on outcomes: “projects that made course corrections earlier in the project cycle had a higher likelihood of achieving their intended development outcomes” (xii). Management also acknowledges the role played by clients to maintain project performance and results. Steps such as expedited decision-making and streamlined government procedures contributed to the quality of implementation by implementing agencies and project implementation units and underpinned timely project restructuring.

Management continues to be attentive to addressing project implementation challenges during this period of ongoing multiple crises. While being appreciative of the report’s conclusion “that overall project performance did not suffer is a testament to the resilience and adaptability of project teams” (92), management is watchful of any potential downturn in performance as projects with extended exposure times to the pandemic and other compounding shocks are included in subsequent RAP reports.

Other Comments

Management is pleased with the report’s acknowledgment of proactive measures taken to improve M&E ratings, which are now at their highest since FY12. Refining M&E methodologies, revising indicators, adjusting targets through restructuring, and gathering supplemental evidence on project achievements helped temper protractive challenges related to the inadequacy of indicators, unrealistic targets, lack of data collection methodology, and attribution issues. Together, these helped projects ascend from moderate to substantial efficacy ratings and achieve intended outcomes—enabling course corrections bases and providing improved evidence on project achievements. Management will continue strengthening M&E data quality and systems, including through periodic deep-dive reviews of Implementation Status and Results Reports and Implementation Completion and Results Reports.

Management notes the disaggregated analysis of the extent of outcome orientation in results frameworks and the progress to date. Based on an analysis of 4,808 indicators in the RAP 2023 cohort of projects, the report found that 40 percent of the project development objective indicators measured outcomes, 46 percent measured intermediate outcomes, and only 12 percent measured outputs. The report points to the scope for further improving outcome orientation for the intermediate results indicators, which management will learn from. The analysis did not find a significant association between a project’s indicator level (that is, outcome or output) and its efficacy ratings and explains that this might be because “other lower-level indicators demonstrate that projects completed intended activities and that these activities would plausibly contribute to the achievement of intended development outcomes, as outlined by the project’s theory of change” (37). Management notes this as evidence that tools such as the theory of change are playing a role in strengthening the lines of sight between project contributions and intended development outcomes. As part of the work on the new Corporate Scorecard, management is committed to strengthening the results architecture of the World Bank. The outcome-oriented focus of the scorecard will be reflected in the approach to be used for developing project-level indicators. Management is also examining the appropriate way of aligning incentives, capacity, and institutional systems with the outcome-orientation approach of the new scorecard.

Future Directions

Management concurs with the report’s suggestion to continually strengthen client capacity to identify and mitigate risks during project preparation, especially the risk of low implementing agency capacity. This suggestion is particularly salient with the continued expansion of support to clients facing greater uncertainties. The Systematic Operations Risk-Rating Tool calibration exercise has enabled teams to identify and mitigate residual risks and helped management focus on high and substantial risk projects. As part of the evolution’s workstream related to operational effectiveness and efficiency, management is looking at ways to strengthen country capacity and institutions and systems. It is also applying a risk-based approach to project preparation and implementation to focus attention that is proportional to risks and where it is most needed, such as low-capacity environments. The nature of implementation agency capacity risks varies significantly across Regions and sectors and between FCV and non-FCV contexts. Where the risk is acute, management is committed to addressing underlying factors through close monitoring, capacity building, commissioning third-party expertise to supplement systems, and use of technology. Experience also shows that relying only on ex ante risk assessment may not be sufficient, as counterpart capacity to implement World Bank–financed operations tends to be weaker in early phases of the project cycle. Updating of risk assessment would therefore need to be part of early course correction and adaptive management.

Management concurs with the report’s suggestion to continue improving M&E as an adaptation and accountability tool. This agenda is advancing further with work on the new Corporate Scorecard, which will help manage and course correct with evidence and report results at scale. As part of the scorecard’s implementation plan, management is planning to improve data quality, impact measurement, and investments in World Bank skills and M&E client capacity on data quality and data management. The scorecard introduces results narratives as a core component under each outcome area. Applying rigorous methodologies such as process tracing and contribution analysis, the results narratives will help articulate World Bank’s contributions to the enabling institutional and policy environment. The expanded use of impact evaluations in the planned Global Challenge Programs is another measure that will strengthen impact measurement. All these measures taken together will improve the results data quality and support midcourse corrections based on evidence, in line with the outcome-orientation agenda.

International Finance Corporation Management Comments

International Finance Corporation (IFC) management welcomes the flagship IEG report RAP 2023. Deep dives on (i) the evolution and relationship of project development outcomes to project performance ratings and (ii) the influence of the COVID-19 pandemic on project performance are particularly helpful. IFC would appreciate a deeper analysis in future to better understand whether the documented correlation among IFC work quality, additionality, and development outcomes extends to a causal relationship and how development outcomes are affected.

Management notes that COVID-19–related effects have started to materialize and to affect performance. The pandemic contributed to a challenging operating environment for both investment and advisory projects, resulting in depressed and changing patterns of demand, reduced access to capital, rising bankruptcies, and persistent uncertainty. Multiple exogenous shocks further exacerbated pandemic-induced economic downturns. IFC welcomes IEG’s assessment of factors that influence the implementation and performance of projects within this context, especially the report’s observations that the performance of investment projects in African and fragile and conflict-affected situations (FCS) countries was challenged mostly by (i) adverse economic factors, (ii) high business risks, and (iii) limited technical expertise and track record of sponsors and clients. As delivery ramps up in these challenging contexts, management is committed to strengthening IFC’s project preparation and M&E capabilities, while recognizing that significant factors remain outside its control.

Management acknowledges the continued weak outcome ratings for IFC investments in Africa, in countries classified as FCS, in countries eligible for International Development Association (IDA), and in World Bank countries and notes that adverse macroeconomic factors and high business risks are key drivers of performance for this group. After last year’s RAP 2022 Board of Executive Directors discussion, IFC undertook an internal review, and many of the drivers identified by IEG in its deep dive resonate with the review’s findings. The IFC deep dive also showed that projects in Africa and IDA-FCS contexts were particularly affected by the challenging economic and operating environment.1 In IFC’s analysis, external factors and risks that underlay weak development outcome ratings included (i) project- or industry-specific factors (fall in prices, weak demand for services or products, sector-specific regulatory and licensing challenges); (ii) unfavorable external macrofactors (economic slowdown, conflict, political turmoil, local currency depreciation, and infectious disease outbreaks); and (iii) sponsor or management issues (lack of management attention, turnover, inexperience, and weak relationship with a sponsor). Manufacturing and infrastructure projects were mostly affected by project or industry factors, while financial institutions were vulnerable to macroeconomic and sponsor issues. FCS projects were exposed to multiple factors that increased the severity of their impact.

While acknowledging that more work needs to be done to support development outcomes, management would also like to highlight IFC’s sustained progress in enhancing delivery in African, IDA, and FCS countries to meet strategic objectives in these priority markets. In FY23, IFC invested $10.4 billion across 41 countries in Africa, the highest ever annual commitment in the continent. Long-term finance reached $6.9 billion ($3.8 billion of own account and $3.1 billion core mobilization), and short-term finance and trade and supply chain in the region amounted to $3.5 billion. For the 17th Replenishment of IDA (IDA17) and FCS, the project count reached 41 percent of IFC’s total (surpassing the IFC Corporate Scorecard target of 39 percent), illustrating an increase in delivery of projects with high expected development outcomes. Short-term finance commitments reached $7.5 billion in IDA17 and FCS countries (68 percent of total short-term finance) and $3 billion in IDA17 low-income countries and FCS countries (27 percent of total short-term finance) in FY23. Moreover, IFC management has taken deliberate actions over the past years to bolster successful outcomes in African, IDA, and FCS countries. These include the merger of upstream and advisory teams in the regions to better align efforts to create the conditions necessary for private sector investment through client and project preparation work; adding experienced, senior resources in the field; and increasing environmental and social capacity in country offices. IFC has also established dedicated programs such as the Africa Fragility Initiative, focused on developing and implementing investment, upstream, and advisory programs in 32 fragile countries in Africa, and a Joint IFC-UNHCR Initiative to enable private sector solutions in the forced displacement context.

Management notes the report’s comments around IFC additionality and highlights two points. First, the report’s comparison of anticipated and realized additionality for select projects in challenging markets relies on nomenclature that is IEG’s interpretation of the categories defined in IFC’s 2018 Revised Additionality Framework. The framework was not applied to investment projects in the RAP 2023 cohort, which predates the framework’s rollout. Second, management acknowledges that IFC is comparatively less successful in realizing nonfinancial additionality than financial additionality and agrees that IFC needs to be more deliberate about nonfinancial additionality. To this end, IFC is focused on providing industry expertise, capacity-building advisory, and better monitoring of delivery of additionality.

Management appreciates IEG’s analytics and the finding on outcome types noting a high level of outcome orientation in project objectives but disagrees with the assertion of deficient tracking of project-level outcomes. Though the previous M&E system—Development Outcome Tracking System—was retired in FY20 and replaced by the Anticipated Impact Measurement and Monitoring (AIMM) system since FY18, results tracking for all active investment projects in the RAP cohort takes place and is ongoing (both systems were active until FY20). While most pre-AIMM vintage projects are not AIMM assessed during portfolio monitoring, development impact indicators of all investment projects were monitored and assessed by one of these systems. Deficiencies of the Development Outcome Tracking System in capturing claims have been addressed in the AIMM system. Further, IFC is stepping up analytical work in assessing ex post outcomes for specific projects and programs. This will complement the ongoing ex ante AIMM analysis and yield deeper insights into development outcomes and drivers of project success. To this end, IFC is hiring new staff with expertise in conducting impact evaluations.

Management notes that, in line with the report’s Future Directions section, IFC has already mainstreamed AIMM into its investment operations, strengthening the measurement and tracking of intended development outcomes of investment projects. RAP 2023 covers projects that predate AIMM. The subsequent adoption and full implementation of the AIMM system has already helped ensure increased use of standardized indicators and their regular monitoring, with an ongoing effort to establish a new data platform for data tracking and reporting, in line with the renewed vision and mission for the World Bank Group. The AIMM framework enables IFC to not only connect financing with quantifiable development outcomes but to also communicate the impact goals to the Board, stakeholders, and clients. As part of the ongoing enhancements to the AIMM framework, IFC is launching the AIMM Navigator—a new tool designed to create a more seamless impact rating and data management process, centralize IFC’s development impact and additionality data, and bring more efficiency to the impact data collection and reporting process.

The report observes that IFC investment projects have no formal procedures for modifying their original development objectives, indicators, and targets to adapt to changing market conditions. Management appreciates IEG noting this weakness in IFC’s approach and, in collaboration with IEG, will explore ways to address it.

Multilateral Investment Guarantee Agency Management Comments

The Multilateral Investment Guarantee Agency (MIGA) welcomes IEG’s RAP 2023 report and finds it useful and important. MIGA thanks IEG for the productive engagement during the report’s preparation. In particular, MIGA finds the RAP 2023 valuable as the first systematic reporting of World Bank project performance during the COVID-19 pandemic.2 We also note that this RAP updates and enhances the outcome orientation analysis of World Bank projects covered in the RAP two years ago. The RAP 2023 also applied innovative use of machine learning for the International Bank for Reconstruction and Development, the IDA, and IFC assessments. MIGA is hopeful that the possible synergies and efficiency gains of this approach can be applied to the analysis of MIGA projects going forward, and MIGA stands ready to work with IEG to operationalize this enhancement, if this would be helpful. MIGA also observes that the scope of the RAP 2023 did not include an assessment of the effectiveness of collaboration across the World Bank—a significant aspect of the work pertaining to the “Evolution of the World Bank” and the “new playbook.” While the subject was touched on in the previous RAP, reflecting in part a focus on more country-level evaluation evidence, MIGA would find it useful if IEG were to have more systematic and regular coverage of the One World Bank approach in future RAPs.

MIGA welcomes RAP 2023’s confirmation that MIGA maintained its historically high development outcome success rate of 72 percent. Also, MIGA welcomes IEG’s observation that the development outcome success ratings of projects in FCS and non-FCS countries remain the same. This finding is important given projects in FCS countries are riskier and more challenging. Conversely, RAP 2023 reported projects in IDA and Blend countries are less successful than in non-IDA countries. Pandemic-related challenges and related higher country risk environments are headwinds for MIGA, as evidenced by IEG’s observation that over a quarter of the MIGA projects of the cohort evaluated in RAP2023 experienced COVID-19 pandemic effects—the highest percentage among the three institutions of the World Bank. MIGA is increasingly active in supporting projects in more challenging environments with higher associated risks, including fragile settings. Indeed, MIGA’s overall portfolio in IDA and Blend countries has increased from 24 percent in FY19 to 41 percent in FY23. IEG also acknowledged MIGA’s positive role and contribution (that is, referring to outcomes associated with company behavior changes due to MIGA’s participation), with satisfactory ratings at 88 percent in FY17–22, the same level as in FY16–21.

The RAP 2023’s assessment of MIGA’s work quality indicates that evaluated projects had a lower proportion of satisfactory and above ratings in the recent cohort compared with the earlier periods. In the Future Directions section, it suggests that MIGA “enhance project preparation work quality to strengthen the performance of MIGA guarantee projects” (98). MIGA considers IEG’s observations in the Future Directions section related to its assessment of front-end work quality to be based mainly on the assessment of MIGA’s front-end work quality from 5 to 10 years ago. Since then, MIGA has made significant changes, partly due to previous IEG observations in this area of MIGA’s work quality and lessons, and MIGA has learned from previous discussions of project evaluations with IEG.

In this regard, it is important to recognize that the FY17–22 projects evaluated for RAP 2023 entered MIGA’s portfolio in about FY14–18, corresponding to a period of strong growth in guarantee issuance and product innovation, including new areas of risk-taking and testing new approaches and instruments. During this period, MIGA was in the process of adapting its work product to reflect the new challenges and lessons learned, including in the latter portion of the period by working to specify expected development impact based on the new instrument innovations and contexts. These experiences and learnings were essential for subsequent improvements, which culminated in the launch of the Impact Measurement and Project Assessment Comparison Tool (IMPACT) framework, MIGA’s ex ante development outcome assessment tool, piloted in FY19 and fully launched in FY20.

Another indication of less-than-satisfactory rated MIGA work quality is associated with IEG’s observation that scenario analysis would highlight the riskiness of the project business and that MIGA could undertake more comprehensive project risk assessments, estimate detailed operational and financial projections with clear targets and account for stricter downside scenarios. However, this observation is based on the historical cohort of evaluated projects assessed for RAP 2023 and does not represent MIGA’s current approach to its project work. MIGA’s current project work is heavily oriented toward assessing project risks, detailing downside scenarios, and mitigating risks where feasible. Most project documents now contain downside scenarios when appropriate; this development in MIGA’s project work is partly a result of prior IEG observations on this point from which MIGA has learned and implemented relevant changes. Moreover, MIGA’s IMPACT framework provides a likelihood assessment of development outcomes, which brings attention to development outcome risks and incentivizes teams to mitigate these risks to the extent feasible and, in the process, potentially achieve higher IMPACT scores. However, identifying risks is not the same as successfully mitigating risks, which needs to be balanced against the costs and practicality of mitigating risks, which has a bearing on a project’s bankability and feasibility. MIGA emphasizes calculated and appropriate risk taking to support projects in the most difficult contexts.

On the discussion of outcome orientation, MIGA is also pleased by IEG’s assessment that MIGA guarantee projects were focused on higher-level outcomes as envisaged by their specific intended outcomes, which was a main objective in introducing the IMPACT framework itself. There is a specific observation by IEG that MIGA should strengthen the measurement and tracking of intended development outcomes, particularly at the foreign investment–effects level. MIGA agrees with this observation, and MIGA is at the early stages of making changes to how MIGA tracks projects in relation to the IMPACT framework, especially (as also noted by IEG) for the achievement of intended foreign investment–level outcomes. These reforms will also be helpful in the context of launching the new World Bank Corporate Scorecard, for which MIGA is actively engaged in discussions with International Bank for Reconstruction and Development and IDA and IFC colleagues. MIGA hopes to show progress in this area in forthcoming Project Evaluation Results and future RAP reports.