The World Bank's project outcome ratings, which have been improving steadily since FY10, saw a steep increase of 9 percentage points in FY20, according to IEG's Results and Performance of the World Bank Group 2021 (RAP2021). The International Finance Corporation's (IFC) ratings for investment projects increased on a three-year rolling basis for the first time in 10 years and the ratings for advisory projects continue to improve. The Multilateral Investment Guarantee Agency (MIGA) continued a 10-year upward trend in project development outcome ratings.
See here for a summary of the report's methodology.
World Bank outcome ratings increased across the board
Ratings increased across all project categories, in all Practice Groups and Regions and for projects of most lending sizes – with notably high ratings for the largest World Bank projects. This overall ratings jump was achieved through a broad-base increase rather than improvements limited to specific portfolio segments. Higher ratings were observed for projects closing throughout FY20, before and after the onset of the pandemic. It is likely, however, that the full impact of COVID-19 on ratings will only show in the future, since projects closed in FY20 had at most just a few months of implementation under pandemic conditions.
The World Bank also increased its performance in more difficult environments and in the most vulnerable countries. Projects in Fragile Conflict-affected (FCS) and International Development Association (IDA) countries exhibited higher ratings and expanded their share of the overall portfolio. The share of projects closing between FY19 and FY20 in FCS countries increased from 15 to 23 percent, whereas the share of IDA lending increased from 47 to 51 percent. These findings are encouraging considering the World Bank continues to expand its engagement in both FCS and IDA countries.
More information regarding how regions, GPs and instruments of the World Bank performed can be found in this interactive dashboard.
IFC's development outcome ratings reversed their declining trends
IFC's outcome ratings for investment projects reversed a declining trend, which reached its lowest level in CY17 when 41 percent of projects were rated mostly successful or better. In the last evaluated period, CY17-19, projects rated mostly successful or better increased to 47 percent. While this is certainly positive news, it is too soon to conclude if the rising trend is sustainable. Development effectiveness ratings for IFC Advisory Services had fallen to their lowest level in FY15–17 but have been improving ever since.
This recent ratings improvement may partly be the result of IFC's efforts to address factors that were negatively influencing its ratings. IFC created the Economics and Private Sector Development Vice Presidential Unit to strengthen its project and macroeconomic analyses, strengthened its monitoring and decision-making mechanisms and improved the quality of self-evaluations.
As a result, the share of XPSRs nominated as best practices increased from 10 percent in 2018 to 20 percent in 2019. The advisory projects' PCRs have also shown improved monitoring and evaluation (M&E) and use of evidence.
See full IFC Ratings Analysis on the Recent Investment Ratings Improvements.
MIGA's ratings have continued to improve
In the case of MIGA, project development outcome ratings have been increasing over the past 10 years. More specifically, MIGA's development outcome ratings increased from 62 percent satisfactory or above (S+) in FY11–16 to 68 percent S+ in FY14–19.
MIGA's improved development outcome ratings are due to both increased evidence collection in recent years and improvement in performance. Meanwhile, additional analyses showed that MIGA projects achieved project-level outcomes more often than foreign investment–level outcomes and that addressing certain corporate priorities—including IDA, FCS, climate change, and South-South projects—did not show a specific impact on ratings.
See full MIGA Ratings Analysis.
Innovation or 'novelty' compatible with an increase in project outcome ratings
One of the questions the report team focused on in evaluating the longer-term upward trend in ratings was the approach to risk in designing projects building on predecessors. Did ratings increase because the World Bank avoided risks tied to innovation? IEG analyzed projects in Education and Transport—two Global Practices with larger portfolios and large shares of highly-rated projects—and found the opposite.
Most successor projects added novelty in project design by introducing new elements not included in predecessor projects. Projects that introduced novelty performed as well as or better than projects that closely replicated the predecessor project. At the same time, there was no evidence that task teams used novelty to course-correct unsatisfactory projects. Rather, task teams phased novelty into projects that already performed well and could more easily sustain expanded, or more difficult, activities while maintaining or even increasing ratings.
These results suggest that the World Bank has been able to take informed risks and introduce new elements relevant to the context, without suffering lower project outcome ratings. These findings are encouraging as they indicate that there should be no disincentives to innovate, although more analysis is needed to identify the conditions that make innovation "pay off" in terms of higher ratings.
See the full novelty study.
IFC projects that aimed for market-level outcomes or projects with high development potential were not accompanied by lower ratings.
It is more difficult for IFC to achieve and measure market-level outcomes than project-level outcomes, as market-level outcomes depend on the broader market environment and external factors. Market-level outcomes are also difficult to measure because they materialize over the long term and few indicators can measure project contributions with certainty. At the same time, projects that had market-level claims at approval did not necessarily end up with lower overall development outcome ratings, as, in addition to market outcomes, overall ratings reflect a range of other aspects, such as a project's business performance, economic sustainability, and environmental and social effects - as well as unintended outcomes.
A high development potential means a higher magnitude of development challenges that demand a more intense IFC contribution. The results show that projects with high development potential were not accompanied by lower ratings or higher variance in ratings. The results also show that IFC projects that addressed prominent corporate priorities—including climate change, IDA, FCS, and inclusive business (which includes gender)—did not have consistently lower ratings.
See full IFC Ratings Analysis on the Longer-Term Investment Rating Trends.
Improvements are seen in monitoring and evaluation quality, but more attention is needed to the choice of indicators and targets.
The report shows that higher ratings are, in part, due to an increased ability of the institutions to document and report results. Monitoring and evaluation (M&E) quality ratings for World Bank operations increased considerably in the last ten years, from 25 percent in FY10 to 59 percent in FY20. For both IFC projects and advisory services, the "rating variance" (that is, the difference between IFC and IEG ratings) has drastically decreased over time. MIGA has also increased evidence collection in recent years.
Underlying these positive results there is a more varied picture. For example, the report found that World Bank higher project outcome ratings are not necessarily matched by higher quality indicators or more ambitious targets, nor consistent clarity on the level or type of outcomes being achieved. The introduction of Anticipated Impact Measurement and Monitoring (AIMM) in IFC and the Impact Measurement and Project Assessment Comparison Tool (IMPACT) in MIGA will allow for a more direct assessment of the project and market-level outcomes.
See more details about ratings and M&E.
A word of caution on the pandemic, and the need for better defined targets and more appropriate indicators
While the RAP 2021 presents encouraging results, as the impact of the pandemic continues to permeate Bank Group operations and outcome measurement systems, outcome ratings may decline in future reviews of the World Bank Group's development effectiveness.
Ratings increases across the Bank Group signal an ongoing commitment to development effectiveness, but project ratings alone provide little evidence on the types of outcomes being achieved and the quality of associated targets and indicators. The RAP2021 calls for all project objectives to have appropriate indicators to measure them, and that measurement should be conducted with strong evidence that is as systematic, independent, and objective as possible.
The findings of this year's RAP contribute to the outcome orientation agenda and complement other IEG products, such as the evaluation of the World Bank Group Outcome Orientation at the Country Level and last year's RAP. This body of work recognizes that the country is the right level to assess contributions to development outcomes; at the same time, the findings presented in the report highlight the need to strengthen the line of sight from project results to country-level outcomes. The projects, like the country frameworks, need to "speak the language" of outcomes.
Add new comment