The Results and Performance of the World Bank Group (RAP) 2021 report by the Independent Evaluation Group (IEG) reviews the World Bank Group’s development effectiveness for fiscal year (FY)21. The Bank Group includes the World Bank (which comprises the International Bank for Reconstruction and Development and the International Development Association [IDA]), the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA). The RAP’s overarching question is, What does the existing evidence from IEG’s project evaluation and validation work show about the Bank Group’s results and performance? The analysis to answer this question focuses on the Bank Group’s project development outcomes, project outcome ratings, project efficacy ratings, and other project- and country-level characteristics. Building on previous RAPs, this year’s approach provides a more comprehensive analysis of results and performance and assesses whether better ratings are indicative of better development results.
This RAP finds that the Bank Group’s ratings improved for the World Bank, IFC, and MIGA over the past year. It shows that these recent improvements—particularly the jump in the World Bank’s project outcome ratings in FY20—were driven by improvements across the board for all Practice Groups and Regions. The World Bank’s ratings improvements were not caused by disruptions from the coronavirus (COVID-19) pandemic. For IFC, changes in its portfolio composition help explain the recent increase in ratings. For the World Bank, IFC, and MIGA, the ratings increases are good news, although it is too early to tell if these improvements will be sustained. The RAP identifies several factors not previously explored in other RAPs or IEG evaluations that can influence ratings. These factors include a project’s level of novelty (defined as World Bank teams introducing new or expanded elements to successor projects in a sector and country), a project’s selection of indicators and targets, its outcome types, and its outcome potential.
RAP 2021 uses a novel methodology to expand on previous RAPs (see box 1.1 for key terms and concepts). First, it carries out an in-depth analysis of recent trends for both the World Bank and IFC. For the World Bank, it analyzes a recent jump in project outcome ratings from FY19 to FY20. For IFC, it analyzes the uptick in ratings during calendar year (CY)19 after several years of declining ratings and a reversal of the trend in CY18. Second, for the World Bank, the RAP uses matched data, linking successor projects in the education and transport sectors to their predecessor projects (in the same country and sector) to analyze the extent to which the World Bank either repeats project designs or introduces novelty to successor projects. The RAP does this to detect signs of risk-averse or risk-taking behavior. Third, the RAP analyzes, in detail, the World Bank’s selection of indicators and use of targets to understand how measurement practices affect ratings and performance. Fourth, for the World Bank, IFC, and MIGA, the RAP looks at the relationship between a project’s outcome types and its results. For IFC, the RAP also examines the relationship between a project’s outcome potential and its ratings. Fifth, for both IFC and MIGA, the RAP analyzes whether IFC and MIGA projects addressing corporate priorities have higher or lower overall ratings. It should be emphasized that evaluation and rating approaches are different across the Bank Group organizations. As such, overall results and performance cannot be compared across them (for more information on evaluation approaches, see appendix A).
For the World Bank analysis, we used different groupings and samples of projects for different types of analyses. Chapter 2’s analysis of the overall rating trends includes all projects that closed between FY00 and FY20 and had an Implementation Completion and Results Report (ICR) and ICR Review (ICRR) completed by August 10, 2021 (N = 5,825), with a special focus on the most recent period, FY10–20 (N = 3,080). The in-depth analysis of novelty, indicators, and outcome types uses a sample of projects that closed in FY12–14 and from FY17 to the second quarter of FY20 and are representative at the level of period and Practice Group (Sustainable Development, Infrastructure, Human Development, and Equitable Growth, Finance, and Institutions). The samples are 90 percent representative with a 10 percent margin of error. Appendix A shows the size and composition of this sample. For the most part, we relied on ICRRs as the key source of information; however, specific parts of the analysis are also based on information derived from Project Appraisal Documents and ICRs. Appendix A also presents other samples and sources of information that the team used in the analyses.
Box 1.1. Key Terms and Concepts
Outcome: A change in behaviors, conditions, or situations resulting from World Bank Group activities. Outcomes may include intended, unintended, positive, and negative changes.
Theory of change: The logic, expressed in project design documents and the Implementation Completion and Results Report, that identifies expected cause-and-effect relationships among inputs, activities, outputs, intermediate results, outcomes, impacts, and underlying critical assumptions.
Project development objective (PDO): A World Bank project’s stated objective, framed as a positive outcome.
Project claims and market and foreign investment claims: The Independent Evaluation Group considers these equivalent to the World Bank’s PDO in the Anticipated Impact Measurement and Monitoring system of the International Finance Corporation (IFC) and the Impact Measurement and Project Assessment Comparison Tool system of the Multilateral Investment Guarantee Agency (MIGA).
Self-evaluation: A formal assessment of a project, program, or policy conducted by or for those in charge of the activity. In the Bank Group, self-evaluation takes the form of a systematic written account of the results and performance of a project or operation, and those in charge assign ratings based on criteria defined in guidelines to assure comparability among reports.
Validation: The Independent Evaluation Group’s independent, critical review of the evidence, results, assessments, and ratings from self-evaluation.
Ratings: Ratings are rubrics for assessing performance relative to a project or program’s objectives. Ratings summarize the self-evaluation narrative into categories or values that enable aggregation across operations. Examples include the following:
- For World Bank projects, the “outcome” rating brings together three underlying dimensions: relevance, efficacy (achievement of objectives), and efficiency. Independent Evaluation Group validations assign ratings for a project’s efficacy in achieving each of its individual objectives and for overall efficacy in achieving the project development objective. Other key ratings are quality at entry (which, together with the quality of supervision rating, determines the Bank performance rating) and monitoring and evaluation quality.
- For IFC investment projects, the “development outcome” rating brings together four underlying dimensions: project business success, economic sustainability, environmental and social effects, and private sector development. IFC’s development outcome ratings do focus on the achievement of expected objectives and a project or company’s results against several benchmarks (such as the performance of peers, the market, or similar industries) and unintended outcomes (positive and negative). Other key ratings are for IFC’s additionality, IFC’s investment outcomes, and IFC’s work quality.
- For IFC advisory services projects, the “development effectiveness” rating brings together five underlying dimensions: strategic relevance, outputs, outcomes, impacts, and efficiency. Other key ratings are IFC’s work quality and IFC’s role and contribution.
- For MIGA projects, the “development outcome” rating brings together four underlying dimensions: project business success, economic sustainability, environmental and social effects, and foreign investment effects. Other key ratings are MIGA’s strategic relevance, MIGA’s role and contribution, and MIGA’s assessment, underwriting, and monitoring, which are aggregated under MIGA’s effectiveness ratings.
Monitoring and evaluation (M&E) quality: For World Bank projects, M&E quality is assessed at the project level and comprises M&E design, implementation, and use.
- M&E design is assessed based on the extent to which (i) the theory of change was sound and reflected in the results framework; (ii) the objectives were clearly specified; (iii) the indicators encompassed all outcomes of the PDO statement; (iv) the intermediate results indicators could adequately capture the contribution of the operation’s components (activities) and outputs toward achieving PDO-level outcomes; (v) the indicators were specific, measurable, achievable, relevant, time-bound, and had baselines and targets available; (vi) the measurement methods were adequate; and (vii) the arrangements were well embedded institutionally.
- M&E implementation is assessed based on the extent to which (i) planned baseline data collection was carried out; (ii) the indicators included in the results framework were measured and reported; (iii) weaknesses (if any) in M&E design—including specification of indicators—were corrected during implementation; (iv) the agency responsible for M&E (and any other relevant stakeholders) ensured attention to effective M&E implementation; (v) data used for M&E were found to be reliable and of good quality (important elements here include sound methodology, independence of analysts, and quality control); (vi) if relevant, beneficiaries were involved in defining target indicators and assessing their achievement; and (vii) M&E functions and processes are likely to be sustained after project closing.
- M&E use is assessed based on the extent to which (i) M&E findings were communicated to the various stakeholders (for example, to inform adaptive management); (ii) M&E information led to strategic redirection or resource allocation or to other positive or negative shifts in the implementation of the project or program; (iii) M&E data were used to provide evidence of achievement of outcomes and not just to provide evidence of application of inputs or achievement of outputs; and (iv) M&E data or findings have informed subsequent interventions or are expected to influence subsequent interventions in the near term.
- High M&E quality helps clarify the “line of sight” from projects to high-level country outcomes by explicitly defining and demonstrating project-level outcomes.
Source: Independent Evaluation Group.
For IFC’s analysis, the sample includes projects with evaluations that IEG validated from CY09 to CY19 for investment projects and FY09 to FY19 for advisory services. For the outcome type analysis on investment projects, the sample includes (i) projects evaluated by IFC and validated by IEG during FY12 and the first half of FY20 (evaluation cycles), and (ii) projects with backfilled Anticipated Impact Measurement and Monitoring (AIMM) data. The analysis compared the early period (CY12–16) and the later period (CY17–20). We relied on Expanded Project Supervision Report (XPSR), Evaluation Note (EvNote), and AIMM data as the main sources of information.
For MIGA’s analysis, the sample includes all projects with evaluations validated by IEG during FY09–19. For the analyses of outcome types, the sample includes all projects validated by IEG by December 2020 for the FY12–14 and FY17–19 cohorts. We relied on Project Evaluation Reports and Validation Notes as the main sources of information.
- The Results and Performance of the World Bank Group (RAP) reports measure performance through ratings assigned during the Implementation Completion and Results Report Review (ICRR) validation of the Implementation Completion and Results Report (ICR) project self-evaluation.
- Ratings for World Bank projects in this report are aggregated according to the closing fiscal year of the project, so fiscal year (FY)20 here refers to the group of World Bank projects that closed in FY20, completed their ICRs, and had their ICRs validated by the Independent Evaluation Group (IEG) in ICRRs by August 10, 2021.
- The more detailed definition of the FY17 cohort was projects closed in FY17, FY18, FY19, or the first two quarters of FY20 and that had ICRRs completed with ratings available in the system as of January 21, 2021. For additional information, see appendix A.
- For the self-evaluation reports of International Finance Corporation (IFC) investment projects (Expanded Project Supervision Report) and advisory projects (Project Completion Report) as well as Multilateral Investment Guarantee Agency projects (Project Evaluation Report), the trend data reported in this RAP includes those self-evaluation reports validated by IEG by August 10, 2021.
- In the backfilling exercise, the IFC retroactively applied the Anticipated Impact Measurement and Monitoring (AIMM) framework to projects whose approval predated the AIMM framework, including identification of outcome claims, conducting underlying analysis of development outcomes, indicator targets and results, and assignment of corresponding ratings (collectively we call these AIMM data in this RAP). This exercise identified outcome claims and their expected results and verified the results for each outcome claim. The IFC projects to which AIMM was applied at approval have not yet been evaluated by IEG.