Management of the World Bank Group institutions welcomes the Independent Evaluation Group (IEG) report, Results and Performance of the World Bank Group 2020 (RAP 2020). Management welcomes the positive overall findings by IEG on performance at both the project and country program levels. The report’s findings provide useful inputs to both learning and strategic decision-making.
World Bank Management Comments
Management welcomes IEG’s positive overall findings regarding performance at the project and country program levels. The report notes that 79 percent of World Bank projects that closed in fiscal year (FY)19 and were evaluated by IEG were rated moderately satisfactory or better (MS+) at completion, surpassing corporate targets (75 percent), and also that project ratings by volume have remained above corporate targets since FY13. Bank Group country program outcome ratings increased from 51 percent MS+ in FY09 to 74 percent MS+ in FY17 across all reviewed country program cycles. Management believes that these positive trends are partly the result of proactive management of quality at entry and enhanced focus on supervision.
Management notes that although project outcome ratings for International Development Association (IDA) countries and countries affected by fragility, conflict, and violence (FCV) have improved, country program outcome ratings stayed flat in these countries. Ratings for projects in IDA countries increased from 68 percent to 78 percent MS+ in FY12–14. Projects in FCV-affected countries rose from 69 percent to 77 percent MS+ over the same period. Management notes, however, that these improvements were not concomitantly reflected in country outcome ratings. Management believes that in addition to the more difficult contexts, this trend is partly explained from a somewhat rigid use of results frameworks, which penalizes course correction in countries that necessitate more flexibility. Management is reassured that this challenge has also been pointed out by IEG in the report Outcome Orientation at the Country Level and that pilot solutions are being explored. Anticipating this, the February 2020 FCV strategy articulated, as one of its operationalizing measures, that the World Bank would enhance its evaluation framework for country programs in FCV settings by encouraging more realism—both in objective setting and in project design and implementation—and would also make the evaluative framework more adaptable to dynamic circumstances and to situations of low institutional capacity and high levels of risk and uncertainty.
Management is pleased to note the positive trends for Bank performance (82 percent in FY19), quality of supervision (86 percent in FY19), quality at entry (75 percent in FY18–19), and monitoring and evaluation (M&E; 51 percent in FY19) and acknowledges room for continuous improvement. Management is particularly reassured to note that sustained efforts to improve M&E are starting to yield results. Given that IEG ratings are based on closed operations, which were designed 5–8 years earlier in most cases, this improvement in M&E quality ratings demonstrates that recurrent management efforts, such as enhanced tools, guidance, training, and resources for staff to strengthen project quality and promote more robust M&E practices (including a stronger focus on intervention logic, results frameworks, results indicators, and M&E) are proving effective. Just recently, management launched an M&E Gateway to serve as one stop clearinghouse for M&E resources across the World Bank. These efforts are expected to improve M&E quality further, particularly as management rolls out its pathways to strengthen outcome orientation in operations and country partnership frameworks.
Management recognizes the need to ensure that development policy financing (DPF) projects perform as strongly as investment project financing (IPF) projects but does not see sufficient evidence of a deteriorating trend. The report notes that the share of DPF operations rated MS+ decreased modestly to an average of 69 percent between FY17 and FY19, from 72 percent between FY12 and FY14. The report rightly identifies that “some risks relate to the nature of the DPF instrument itself” and that “evaluation methods also play a role, as, de facto, they differ between IPFs and DPFs.” What the report does not make explicit is that, in FY19, only three DPFs were added to the analysis of DPF outcome ratings, with the result that performance changes period-over-period were driven in part by a sample too small to be representative.
Management notes the new outcome classification that IEG has explored in the evaluation and is reassured by IEG’s finding that operations that specify higher-level outcomes can perform well—when the operational context is right. At the same time, management urges caution regarding the inference that operations with higher-level development objectives are more ambitious. Although all operations use a theory of change approach, not all project development objectives are explicitly set at the same level for multiple reasons. For example, identifying objectives that can be attributed to World Bank interventions continues to be important for accountability and transparency. It is also part of applying the theory of change rigorously and consistently and requires elaborating objectives at lower levels, where attribution is typically stronger. This underpins the report’s finding that approximately 72 percent of IPFs state their outcome objectives at level 2, and another 26 percent at level 3. These level 2 outcomes (for example, improved quality or access to social- or infrastructure-related public services) are relatively easier to attribute to World Bank support by the time the project closes and clients often favor that. Experience has also shown that, for IPFs, level 4 outcome objectives are hard to reach. Human Development projects, for example, rarely go to level 4 outcomes because these outcomes take time to be realized and are often affected by factors outside the control of a single project or program. Having said that, management recognizes that more needs to be done to ensure that operations establish more explicit lines of sight toward higher-level outcomes and international commitments. This is an intended effect of the World Bank’s renewed efforts to strengthen outcome orientation and to connect the risk and results thinking.
Management is of the view that corporate results measurement is instrumental to advance outcome orientation. The report argues that there is a “trade-off” between the World Bank’s outcome orientation and its existing Results Measurement Systems (RMS) for tracking commitments. Management views these as complementary. The IDA RMS and the Bank Group Corporate Scorecard, along with other corporate reporting systems, are designed to monitor short- to medium-term results, which can be attributed to individual projects and aggregated for portfoliowide reporting. It is possible and often desirable to indicate how these same short- to medium-term results contribute to longer-term results that are higher up the same results chain. Both the RMS and the Corporate Scorecard incorporate long-term development outcome indicators (tier I) to contextualize the global development environment in which we are operating, in addition to reporting aggregate World Bank–supported results. Inclusion of selected indicators, accompanied by targeted actions, in these instruments has proved effective over time to advance new priorities and commitments, such as gender, climate change, and citizen engagement.
International Finance Corporation Management Comments
Management of the International Finance Corporation (IFC) welcomes IEG’s Results and Performance of the World Bank Group 2020 report. The report provides both an assessment of project performance and a review of whether the set project objectives are sufficiently focused on development outcomes, that is, whether development impact is accurately and adequately captured by aggregated project performance metrics.
IFC management appreciates the increasingly collaborative approach by IEG in support of improvement in the methodologies underpinning the assessment of IFC’s development performance. In addition to IFC’s initiative to engage IEG on the impact of the coronavirus (COVID-19; described in box 4.2 of the RAP), IFC looks forward to sustaining this engagement on methodology. IFC values both IEG’s and the Board of Executive Director’s willingness to consider this question because it is important to assess whether the metrics we have are appropriate for what we want to measure and ultimately know about our impact.
IFC management regrets that the treatment and presentation of the performance data in chapter 2 makes it impossible to discuss IFC’s current direction with respect to addressing a historic trend in performance. New readers of the report may fail to appreciate that what is being reported here is not IFC’s current performance but rather how IEG has evaluated projects for which objectives were set approximately seven years ago. The most recent investment projects evaluated in the report were approved by the Board in calendar year (CY)13, and were part of the CY18 Expanded Project Supervision Report (XPSR) cohort, which IEG evaluated during CY19. Similarly, for advisory projects, the cohort includes only a preliminary sample of projects closing in FY19, with most conclusions being drawn from projects designed an average of seven years ago.
IFC’s ongoing efforts in the past two years to improve performance have started showing noticeable results in the CY19 evaluations and ratings, which may not be apparent to shareholders and stakeholders until the 2021 RAP. For improvements related to quality at entry, an even longer period will be required for these to be reflected in the RAP results. It is worth further noting that given the evaluation and reporting time lag, projects assessed ex ante under the Anticipated Impact Measurement and Monitoring (AIMM) framework at the time of Board approval will start being evaluated as part of CY22 XPSR program with results being validated and reported by IEG in subsequent years. IFC is nevertheless pleased to see that the work mentioned to address declining development effectiveness ratings for advisory services projects in management responses to previous RAPs is reflected in recognizable year-on-year improvements in performance. IFC believes that to ensure that this is clearly understood by all readers, including new readers, IEG should provide clearer metadata and signposting with respect to exactly what is being described, including in the headings of graphics.
In this context, IFC management believes that it is worth restating observations made previously with respect to the sustained effort to turn around IFC’s performance, as this provides context and brings the report up to date. IFC has previously highlighted that the effort to deliver greater development effectiveness has included the creation of the Economics and Private Sector Development Vice Presidential Unit to strengthen project and macroeconomic analyses, the launch of the AIMM framework, and the Accountability Initiative. The latter initiative informed subsequent decisions in the operational realignment, including very significant changes to the Accountability and Decision-Making framework. The strengthening of IFC’s operational practices and processes is ongoing. IFC management has initiated multiple efforts to improve the quality of self-evaluations (Expanded Project Supervision Reports [XPSRs] / Project Completion Reports) and proactively engage on other associated activities, including the review of validation notes (EvNotes) and IEG independent evaluations undertaken on closed projects (Project Evaluation Summaries). This effort comprises targeted, expert advice to strengthen the analysis and articulation of a project’s overall outcome, including development impact, along with increased support to facilitate the effective management and processing of XPSRs and Project Completion Reports or Project Evaluation Summaries.
IFC management wishes to complement the report’s consideration of an outcome-based approach to evaluation by noting that IFC has already made a conscious decision to go beyond the direct impacts captured by the Development Outcome Tracking System and include indirect effects of IFC investments (on market creation and development impact). In addition, through AIMM, IFC introduced an ex ante component to complement the existing ex post M&E approach. Furthermore, IFC strengthened the broader corporate incentives to put development impact at the heart of IFC’s decision-making with respect to where and how to deploy IFC’s scarce resources by ensuring that AIMM scores inform the project assessment and approval process.
IFC management welcomes the restatement in the report of earlier work to consider risk factors that influence project performance. However, these findings suggest the need for an evaluation approach to account for external shocks (generally unexpected and beyond the control of either IFC or our clients) and allowing for a more systematic treatment of risk, not fully considered in the report. IEG’s machine learning exercise discussed in the report confirms the results of an earlier IFC-IEG joint study, which found that sponsor-selection risks, market risks, country risks, and transaction structuring are factors that most frequently distinguished investment projects with good ratings from less successful ones. IFC appreciates the observation made in the report that, over time, IFC has taken on (and will continue to take on as part of the IFC 3.0 strategy) greater risks over which it can only exercise limited control. In this context, which is further exacerbated by the forces unleashed as a result of the COVID-19 crisis on these extant risks, there is a clear need to pursue a more systematic approach to performance measurement and evaluation that factors in the changing nature of IFC’s business model to one with an inherently higher risk tolerance, in a dynamic private sector environment. Encouraging private sector investment as part of recovery from COVID-19 will demand that IFC encourage sponsors of projects to consider new business lines within broader reallocation of resources by the private sector across sectors within the economy, and that IFC’s clients and portfolio absorb the exogenous shock of weak consumer demand. This is in addition to the policy decision in the Forward Look to take on more country risk, implicit in the pivot to IDA and fragile and conflict-affected situations (FCS). In this context, there is a distinct possibility that the recently observed improvement in development outcome ratings as a result of IFC’s turnaround efforts, may not be sustained due to the disruption experienced by projects designed in a pre–COVID-19 environment. Even in instances in which IFC’s investments deliver a positive development outcome in the context of the current crisis, for some projects the precrisis designed objectives may not be achievable in a radically changed environment.
IFC management welcomes the inclusion of a summary outline of IFC’s reforms to strengthen upstream engagement. As the report highlights, as with previous IFC initiatives in recent years, the goal is improved coordination toward greater development outcomes. Upstream is a more proactive way of doing business by getting involved earlier in the sector and project development process, including conceiving opportunities for unlocking sectors of the economy and conducting feasibility studies to generate investment-ready opportunities. It is IFC’s most recent initiative, perhaps the most critical building block of the internal reforms IFC has implemented over the past four years. IFC believes that upstream will be an important component of the institution’s response to the restructuring and recovery phase of the pandemic and key to an effective crisis response. It highlights the potential value of a shift away from a project-by-project approach to evaluation of IFC’s performance toward a model more anchored in outcomes, as failures are implicit in the design of a more adaptive project discovery approach within IFC’s broader business model. Evaluating success or failure with respect to the impact of these actions will demand a “bigger picture” perspective of IFC’s overall performance than can currently be captured in the RAP.
IFC management notes that the report sets out that there is a need for instruments suited for collecting higher-level outcome evidence. However, it is not clear what form a new system might take or what the effect could be on resources. IFC believes that, should an outcomes-based approach be pursued, it will be necessary to explore the detailed implications, including the implications for costs, staff, and existing systems. The creation and implementation of the AIMM approach and training of staff has required a very considerable effort over the past four years, which suggests that we should approach this question as one of continuous and steady evolution of our understanding of the contribution we make to effecting change.
Multilateral Investment Guarantee Agency Management Comments
The Multilateral Investment Guarantee Agency (MIGA) welcomes RAP 2020. MIGA welcomes IEG’s Results and Performance of the World Bank Group 2020 report and finds it useful and important. MIGA commends IEG for streamlining and sharpening the focus of the RAP 2020 report and exploring new themes. MIGA thanks IEG for the productive engagement during the drafting of the report.
Historically high MIGA development results. The report presents many useful findings, and MIGA appreciates IEG’s observations. In particular, the report notes the steady increase in in the development outcome success rates of MIGA guarantee projects over the past 10 years. The development outcome success rate for the period under review, FY13–18, reached the MIGA-historic high of 69 percent by number of projects (n = 71) and 75 percent by gross issuance amount ($7,725 million). The increase in the development outcome success rate has been driven by strong performance in IDA (74 percent by number, 82 percent by amount), FCV (78 percent by number, 84 percent by amount), the Energy and Extractive industries sector (79 percent by number, 87 percent by amount), and the Eastern Europe and Central Asia Region (73 percent by number, 78 percent by amount). MIGA notes that the sustained increase in development outcome success rates to historic high levels validates the Agency’s increased emphasis on underwriting impactful projects in difficult settings and increased attention to monitoring, evaluation, and learning. In addition, MIGA’s efforts to diversify the Europe and Central Asia portfolio away from financial markets—which was adversely impacted by the 2008 global financial crisis—to other sectors has been instrumental in improving overall performance, as noted in the report.
Good performance of IDA and FCS projects. The report finds that MIGA played an active and important role in promoting private sector investment through projects in IDA and FCS countries. MIGA notes that the good IDA and FCS performance to be an important foundation for the Agency’s FY21–23 strategy, which emphasizes continued support for IDA and FCS as strategic priorities. MIGA notes that the strong IDA and FCS results bode well for the Agency’s ambition for further deepening the development impact of MIGA guarantee projects.
Remarkable progress in environmental and social (E&S) performance. MIGA welcomes the report’s recognition of the remarkable progress made regarding the E&S results of MIGA guarantee projects. During FY13–18, E&S effects was the highest-rated development outcome indicator, with a success rate of 84 percent by number and 88 percent by amount, compared with 50 percent by number and 46 percent by amount during FY07–12. MIGA notes the rapid strides made in E&S monitoring and supervision after the adoption of Performance Standards on Social and Environmental Sustainability in 2007 and the launching of E&S policy implementation monitoring in MIGA guarantee projects in 2011. MIGA notes that the strong E&S results highlighted in the report have been on account of the Agency’s enhanced E&S monitoring and supervision efforts of its guarantee projects. MIGA notes the good example cited in the RAP 2016 report of an oil and gas sector project in Uzbekistan, where the MIGA team helped solve critical E&S issues by convening external industry experts.
COVID-19 and MIGA projects. The report states that IEG and IFC are discussing potential adjustments to project ratings to account for shocks like COVID-19, including making project objectives more realistic by rating projects based on projects’ midcourse correction targets rather than those set at approval before the shock occurred and giving IFC more flexibility to choose the evaluation timing, which may help projects recover and meet targets at a later time. MIGA notes that, given the broad similarities between the ex post project evaluation frameworks for IFC investment projects and MIGA guarantee projects, COVID-19–type shocks impact MIGA projects as well. MIGA looks forward to working with IEG and exploring similar rating and evaluation timing adjustments to MIGA guarantee projects, which are facing broadly similar challenges to IFC investment projects from the COVID-19 pandemic.
Characteristics of MIGA guarantee projects. In its discussion on the historically high development outcome success performance, the report delineates some key characteristics of MIGA guarantee projects, which are reflective of the Agency’s mandate and business model: (i) MIGA’s clients are larger multinational investors; (ii) MIGA political risk insurance guarantees against political risks; (iii) the relatively large size of MIGA-supported projects makes them visible in host countries and motivates governments to help them succeed; and (iv) MIGA originates the majority of its projects from part 1 countries. In addition to these project characteristics, MIGA notes the significant initiatives that the Agency has undertaken to (i) enhance project selection; (ii) strengthen assessment, underwriting, and monitoring; (iii) bolster results measurement systems; (iv) implement an ex ante development impact assessment system; and (v) promote learning from evaluation. These measures have played a critical role in the steady improvement in the development outcome success rates of MIGA guarantee projects to the current historically high levels of 69 percent by number and 74 percent by amount. MIGA also notes an important caveat to the report’s reference to the relatively larger size of MIGA guarantee projects, due to the fact MIGA support for smaller guarantee projects—including small and medium enterprises—through the Small Investment Program (https://www.miga.org/small-investment-program) are evaluated on a programmatic basis rather than at the project level. In other words, MIGA support for small guarantee projects is not reflected in IEG’s RAP 2020 project evaluations database, and therefore the report’s reference to the “relatively large” size of MIGA guarantee projects is not fully accurate.