Management of the World Bank Group institutions welcomes the Independent Evaluation Group (IEG) report, The World Bank Group’s Results and Performance 2021 (RAP 2021). Management welcomes IEG’s positive overall findings on performance at the project level. The report’s findings provide valuable insights to both learning and strategic decision-making.

Overall

Management welcomes the overall positive findings, which place outcomes for projects at a historical high, and concurs with the overall conclusions. The report notes that 88 percent of Bank Group projects that closed in fiscal year (FY20) and were evaluated by IEG were rated moderately satisfactory or better at completion, surpassing corporate targets (75 percent), and underscores this impressive largest annual increase in outcome ratings over the past five years. These accomplishments “resulted from ratings improvements for virtually all categories of projects—all Practice Groups . . . all Regions . . . and almost all lending sizes” (72). Management is particularly pleased that “ratings also improved in the most challenging places to operate” (x) such as in fragile and conflict-affected situations (FCS) from 80 to 83 percent and International Development Association (IDA) countries from 75 to 86 percent. Management concurs with the RAP 2021’s overall conclusion that “longer-term ratings increases can occur with improved M&E [monitoring and evaluation] and selective risk taking derived from adding new activities in successor projects” (22). These variables correspond to management’s past and current efforts, and the findings of this report will help further calibrate ongoing directions.

As suggested by IEG, management remains cautiously optimistic in interpreting these findings, as the effects of the coronavirus (COVID-19) pandemic are still unfolding. Although COVID-19 does not seem to have an immediate implication for the reviewed results, management is aware that projects closing in the next few years could see some negative effects brought through the crisis. Remote implementation support together with the inability of clients to access, collect, and validate data could also affect the strength of the evidence to support project effectiveness. Management looks forward to working with IEG to assess how best to interpret and rate development effectiveness in the years to come. Management commends IEG for improving the insightfulness of the RAP in 2021 and suggests more systematic dialogue to improve the clarity and shared understanding of its newly introduced methodological approaches. This is particularly true for the so-called novelty analysis. In addition, the introduction of outcome types as a follow up to the work on outcome levels in RAP 2020 includes an interesting dimension that is currently being explored in the Bank Group’s work on indirect pathways, as part of the outcome orientation road map. While noting that outcome types (as currently defined) do not have strong explanatory power with respect to the ratings, management sees value in continuing to refine the outcome typology. The typology currently presented would benefit from scrutiny from a larger group of experts from across the Bank Group. These conversations could be held within a reinvigorated Results Measurement and Evidence Stream that brings lessons and perspectives from across the Bank Group.

Outcome Orientation

Management notes that this year’s RAP departs from precedence by not assessing Bank Group performance in country engagements and regrets that the intersection of country and project results are not sufficiently clear in the report. Management recalls that the RAP 2021 Approach Paper stated that “the RAP 2021 analysis will be carried out at the project and country level.  . . . At the country level, the focus will be on the association among aggregate project ratings across a country, patterns in outcome types across projects in a country, and country characteristics associated with greater challenge” (World Bank 2021, 8). As surfaced during the discussion on outcome orientation, including in the IEG evaluation World Bank Group Outcome Orientation at the Country Level (World Bank 2020d), the country level is the most relevant unit of analysis for outcomes, as the Bank Group supports country outcomes that are defined in clients’ own strategies and reflected in country engagement products. Individual projects and their corresponding results frameworks should be viewed in the larger context of country engagements and wider strategies to aim for and monitor results. This is particularly true for understanding risk and results considerations, institutional strengthening results, and the desirability and feasibility of novelty. These foundational considerations for outcome orientation do not often occur in the vacuum of individual projects, but in the context of portfolio-wide efforts toward high-level results.

Management welcomes the report’s conclusions regarding risk management in operations and notes its ongoing efforts to place outcome risk at the center of results programming, as part of the outcome orientation road map. RAP 2021 concludes that the Bank Group “can discern when the conditions are right for projects to support novel and complex activities. In this way, teams can take informed risks and selectively build on past experiences to elevate a project’s objectives without suffering lower project performance ratings” (75) confirming the findings of the outcome pilots, which showed that informed risk taking is an integral part of the model of how the Bank Group aims for outcomes. Management will reflect on the findings of RAP 2021 as it continues rolling out three germane activities of the outcome orientation road map, namely (i) articulating a conceptual approach to assess development outcome risk in Bank Group programs and adjusting guidance, (ii) calibrating the Bank Group’s Systematic Operational Risk Rating Tool, and (iii) building a more integrated focus on risk and results in training and outreach work on operations. These activities will help the Bank Group “inquire further about the motivations that drive risk taking [and] the evolution of project designs”(75) as suggested by the report.

Management welcomes the discussion on project novelty, yet it believes that the interplay between risks and results considerations at project design is significantly more complex and worth building on. Management appreciates the innovation in depicting risk, but notes some challenges in the definition of risk aversion and the new approach about “novelty.” A significant part of the evaluation is focused on “risk aversion” and “novelty” as they relate to outcomes. A definition of risk aversion that is predicated on the inclusion of “new activities” with a differing “degree of difficulty” in successor projects is narrow and does not fully reflect the reality of choices that teams face in project design. It also does not reflect the interplay between project and country-level considerations. Experience has shown that project activities are chosen based on how appropriate they are in the context of client needs and how they are supported by analytical underpinnings, lessons learned, and stakeholder feedback, and not necessarily based on whether they are novel. Bank Group teams also periodically assess how risks could hinder the country’s journey toward high-level outcomes through indirect and direct pathways and seek risk mitigation measures. Over time, investments in policy reform and institutional strengthening reduce the inherent capacity risks in each sector, so that the sector gradually moves from a higher to a lower capacity risk as institutions are strengthened to deliver effective services to achieve the relevant high-level outcome.

Assessing the desirability and feasibility of novelty requires a more nuanced understanding of how indirect pathways are followed and monitored, as management is doing as part of the Outcome Orientation Roadmap. Management welcomes the report’s insights concerning institutional strengthening; these insights will inform the Bank Group’s current exploration of possible methodologies to track indirect pathways to country-level outcomes to be incorporated in self-evaluation and reporting tools. Institutional strengthening activities often support longer-term, higher-level outcomes and are an essential step in helping clients achieve desired results. These activities should be viewed within the context of the entire causal chain leading to the intended outcome, as opposed to results in and of themselves. Better understanding institutional strengthening is essential to work “on the frontier” where real breakthroughs happen. Against the backdrop of renewed efforts to improve the strategic use of knowledge, management does not believe that the report has adequately reflected on how project analytical underpinnings inform the best combination of risks and opportunities and help determine the frontier at which Bank Group operations should work.

Monitoring and Evaluation Quality

Management recognizes room for further improvements in the selection of indicators and targets, and it underscores that long-standing efforts are bearing fruit. The Bank Group has made significant progress in developing robust project-level monitoring and evaluation (M&E) systems, one of the foundations for delivering and tracking progress toward achieving country-level outcomes and instilling a culture of managing for outcomes. As noted by IEG, the recent introduction of theories of change, improvement in measurement frameworks, and the implementation of the new Implementation Completion and Results Report (ICR) methodology at the project level are already helping to make evident the connections between Project Development Objectives and high-level country outcomes. The Outcome Orientation Roadmap already includes several activities aimed at improving the way indicators are designed and utilized across key thematic priorities. In certain cases, management believes that some of the findings of the report concerning targets reflect strengths of the current self-evaluation system rather than shortcomings. This is the case for the 33 percent of targets that were “exactly achieved” (35). This not only shows that teams have become more realistic at calibrating expected results, but it also includes the types of indicators that one would expect to be exactly achieved (for example, number of assessments conducted, annual reports produced, new institutions created, and so on.). Previously, many teams set targets that were aspirational rather than necessary to achieve a development impact “breakeven” point and so were downgraded on failing to meet those aspirational targets. It is important to consider that the choice of outcome indicators could be limited in some instances by, for example, the availability, consistency, timeliness, and quality of data and therefore project teams would tend to choose indicators based on available data. These indicators quite often tend to be intermediate outcome rather than outcome indicators. Contrary to the report’s conclusion, most teams and operational management have diligently focused on improving performance by specifically scrutinizing targets, therefore contributing to greater accuracy. Notwithstanding this progress, management recognizes room for further improvement, particularly in FCS and IDA FCS. In these cases, management has usually prioritized engagements to tackle the most pressing development challenges even when they were difficult to measure in such difficult contexts. The most recent practice note developed by the Bank Group on improving results in fragility, conflict, and violence (FCV) environments should support teams in improving the management of results in these countries (World Bank 2021a).

Moving Forward

Management welcomes IEG’s insight into specific project elements that relate to the achievement of project outcomes, particularly on the importance of M&E. Management is currently reviewing what has worked well concerning M&E quality to learn from different regions and Global Practices (the report does not disaggregate this rating) and to identify opportunities to generalize and scale-up good practices across the Bank Group. RAP 2021 findings are timely and substantial to inform these ongoing efforts and management will involve IEG as this work progresses. As the report hints, many factors may have contributed to improved outcomes, with concerted actions to improve M&E as the most plausible: “These factors include teams improving measurement frameworks, preparing better theories of change, and becoming comfortable with the newer ICR methodology, among others. These improvements could have resulted from ICR reforms in 2017 or internal training and informal knowledge exchanges, which led to an increased focus by operational teams and development effectiveness units on building robust theories of change and paying more systematic attention to M&E quality. It is possible these factors came to fruition in FY20 and contributed to the ratings increase, but we could not measure whether this was the case” (21). Management firmly believes that any system that incentivizes appropriate quality will be grounded, not in one-time reviews, but in a well-constituted enabling environment. Regional Development Effectiveness units play a critical role in guiding project teams, with Operations Policy and Country Services support. Operations Policy and Country Services’ learning arm, in cooperation with experienced operational staff, provides ongoing capacity-building to improve outcome orientation at the project and country levels. Processes such as the Quality Enhancement Review have supported project teams to better manage the ambitions and risks in projects. In short, improved project implementation, M&E, and outcomes result from a variety of staff roles and responsibilities, incentives, tools, systems, training, and resources.

Management believes that, to understand M&E quality appropriately, IEG and management should discuss and revisit the way this rating is measured. The M&E quality rating is derived from design, implementation and utilization, and further exploration could be helpful to build a deeper understanding in relation to the extent to which these ratings are Bank Group or client dependent, and if these ratings might be different at entry as opposed to at implementation or utilization. This may help assess where more effort should be placed in building capacities and providing support.

International Finance Corporation Management Comments

Management of the International Finance Corporation (IFC) welcomes IEG’s flagship report, Results and Performance of the World Bank Group 2021. For IFC, the report provides both an assessment of project performance and a review of the relationship between a project’s outcome types and its results, and the relationship between a project’s outcome potential and its ratings. Like the last edition, we welcome the specific focus on the World Bank, IFC, and Multilateral Investment Guarantee Agency (MIGA) with distinctive findings and suggestions for each organization because this helps each institution tailor its learnings and improvement initiatives. In addition, IFC management appreciates the exceptionally proactive engagement and collaboration IEG’s RAP 2021 task team extended to IFC throughout the process.

International Finance Corporation’s Development Outcomes and Effectiveness

Overall, IFC management is pleased with the improvements reported on both IFC’s investment development outcome rating and advisory development effectiveness rating, as measured by the respective latest three-year cohorts. These improvements followed the reversals marked last year in the downward trends of the single-year-based ratings. In noting the uptick in both ratings, the report recognizes several initiatives that IFC has undertaken over the past few years to promote its focus on development impact: the launch of the Anticipated Impact Measurement and Monitoring (AIMM) framework, the establishment of Economics and Private Sector Development Vice Presidential Unit, IFC’s greater attention to work quality, incentives and expert consultative resources for operational staff to focus on project evaluations, and the enhanced partnership and dialogue with IEG. IFC management welcomes IEG’s acknowledgment of these significant efforts by IFC to substantiate the outcomes in project evaluations and engage with IEG in a constructive and substantial manner. As a result, there has been a sharp reduction in the ratings variance, and an enhancement in learning on both sides of IFC and IEG across investment and advisory services, as the report notes.

IFC management is cognizant that the full effects of the pandemic are yet to materialize. Over the coming years, we may observe a relative stabilization or even a decline of the ratings as the full effects of the pandemic unfold. Because the data and evidence—sometimes challenging to obtain even in normal times—may not be readily available, we hope that we can work with IEG more flexibly when we assess the impact of the projects that are affected by the pandemic. The conversation with IEG in this regard was initiated in FY20 and will continue.

Specifically, on the advisory side, IFC management welcomes IEG’s recognition of the enhanced management attention to work quality and the use of improved evidence to monitor development results. Indeed, we have put considerable emphasis on ensuring that the Project Completion Reports have detailed evidence and that outcomes are well substantiated. In addition, teams have benefited from peer reviews within the results measurement team for borderline projects to ensure that the evidence is adequately presented to substantiate the outcomes.1 As a result, the share of project completion reports that used quality evidence to a “sufficient extent” and “great extent” increased from 62 and 46 percent in 2016 to 70 percent for both categories in 2019, and we find it rewarding to be recognized for this. We hope to maintain this trend.

The report also highlights the lower development outcome ratings of IFC’s investments in the Disruptive Technology and Funds sector. IFC management notes that the Disruptive Technology and Funds Expanded Project Supervision Report cohorts analyzed through the calendar year (CY)19 consisted almost entirely of equity projects. IFC’s overall development outcome rating data by instrument show that IFC’s evaluated stand-alone equity cohort has consistently performed below its loan and combined loan and equity cohorts and 17 to 19 percentage points below stand-alone loans in development outcomes in CY16–19. In fact, inadequate financial and development results of IFC’s equity investments post-2009 prompted IFC management to adjust its equity approach in 2018.2 The revised approach—which focuses greater attention on macro and sector dynamics, the ability to meaningfully influence investee behavior, and the active management of key investments—has been under implementation for the past three years, and initial results are promising. However, it is too early for Expanded Project Supervision Report sampling to meaningfully analyze the impact of IFC’s actions on turning around its equity performance.

On the Special Analyses of International Finance Corporation Investment Services

IFC management welcomes IEG’s innovative analyses investigating the relationships between a project outcome type and its results as well as between a project outcome potential and its ratings. We look forward to continuing the discussion with IEG colleagues on how we may best apply the analyses to our operational work and decision-making for improving performance. We also welcome IEG’s efforts to integrate the AIMM framework into the analyses and reiterate the need for the results from the analyses of outcome type and outcome potential to be interpreted cautiously given the early stage of the AIMM monitoring pilot and the limited data set of AIMM assessments (particularly with respect to AIMM scores “backfilled” on investment projects in IFC’s portfolio).

The report makes a good point about the challenges of achieving market-level claims. Given the factors at play in accelerating market development, it can be difficult for a single institution or project to catalyze such development. Nonetheless, IFC’s ambition to “create (and develop) markets” remains a central pillar of its strategy and a strong focus of its investment and advisory work. That is why recent initiatives, including with respect to IFC’s upstream engagement and deployment of scaled approaches, such as frameworks and platforms,3 have been a management priority. Moreover, the AIMM framework deepens IFC’s capacity to understand the extent of its ability to generate market changes and capture such changes in the monitoring and measurement of its projects. This is a welcome development, even if it means that actual results may not be as ambitious as originally expected, especially in the short- to medium-term. IFC is reviewing the periodicity and practicality of when and how best it would be able to fully capture market outcomes and has plans to prepare guidance for this shortly.

IFC management welcomes the preliminary conclusion that projects with high development potential and those supporting prominent corporate priorities are not associated with lower Expanded Project Supervision Report ratings. We have also confirmed this relationship on a preliminary basis using AIMM score data and have gained two important take-aways: (i) IFC should not shy away from undertaking projects in the most challenging environments—similar to one of the report’s concluded implications that the Bank Group could further emphasize operating “on the frontier,” and (ii) the separation of impact potential from likelihood of achievement continues to provide an important analytical construct for the assessment of development impact.

The report suggests that IFC consider assessing the prevalence of different outcome types and other characteristics in projects to help enhance the system. The practical implication of this suggestion is to incorporate risk (or likelihood of achievement) in the assessment of specific development impact claims. An important design feature of the AIMM framework is the use of a likelihood rating assigned at the dimension level (project outcomes and market creation). IEG’s suggestion to consider incorporating the assessment of likelihood (or risk) at the claim level appears to be supported by the current analysis. At the same time, the nuances of such relationships merit further investigation given the analytical and workload implications for economists and project teams as well as the range of considerations that weigh on a project’s AIMM rating during supervision. The work undertaken by IEG for this report makes an important contribution to IFC’s analysis, which we hope to build on through our ongoing work related to inventorying AIMM claims.

Microproduct Reform and Moving Forward

Finally, we are about to embark on a historic joint effort with IEG colleagues to redesign IFC’s evaluation framework for investment projects—an opportunity created by IEG’s microproduct reform. IFC management and teams look forward to constructive and focused partnership with IEG in this exercise, drawing on experience gained under the current system, while addressing the dual objective of accountability and learning. We appreciate IEG’s keen interest in better aligning the new framework with the AIMM methodology through the redesign. IFC remains committed to focusing on development impact and looks forward to working together with IEG colleagues on the microproduct reform.

In the interim, IFC management hopes to continue exploring options with IEG for addressing known weaknesses of the existing evaluation framework by allowing some elements of the reform to be already applied to the current methodology and process as much as practicable. In this regard, we appreciate continuing dialogues on such efforts as finding practical methodology for benchmarking project performance and a sampling method to generate a cohort that more closely represents key aspects of IFC’s portfolio. IFC management believes that these parallel resolutions will enhance the value of the upcoming annual evaluation programs as it looks forward to collaborating on the microproduct reform for a longer-term framework.

Multilateral Investment Guarantee Agency Management Comments

The Multilateral Investment Guarantee Agency welcomes the RAP 2021 report, and we are grateful to IEG for their engagement and dialogue on MIGA’s data and analysis of the report.

Multilateral Investment Guarantee Agency Performance

MIGA appreciates IEG’s recognition of MIGA’s development outcome ratings of satisfactory or better remaining near peak levels. MIGA’s development outcome ratings increased from 62 percent in FY11–16 to 68 percent in FY14–19 by project number and from 59 percent to 74 percent by gross issuance amount across the same periods. Among MIGA’s four sectors, the Energy and Extractive Industry sector had the highest development outcome ratings by project number at 75 percent. IEG recognized that MIGA supported several countries’ “first-of-a-kind” power projects with strong potential to achieve demonstration effects.

RAP 2021 illustrates that projects aligned with MIGA’s corporate priorities had a clear tendency for better performance. Projects in IDA countries and FCS countries outperformed those not in these countries. These IEG findings give strong assurance of MIGA’s ability to expand in these two priority areas, given that in FY21, a quarter of our guarantees supported projects in IDA countries and fragile situations. The RAP 2021 analysis of the development impact of climate change projects was inconclusive, as IEG identified a small number of evaluated projects as addressing climate change. Since that period, MIGA has scaled up its climate finance activities, including enhancing and systematizing practices about identification of projects addressing climate change. In FY21, MIGA contributed to the Bank Group’s second Climate Change Action Plan (2021–25), with 73 percent of MIGA projects supporting climate change mitigation or adaptation or both.

RAP 2021 also connected better actual outcomes with a strong focus on results measurement. IEG concluded that MIGA’s improved development outcome ratings are due to increased evidence collection in recent years and a definite performance improvement. RAP 2021 acknowledged improved evidence collection due to MIGA’s ongoing work to enhance the quality of our self-evaluation and evidence collection. Further to our emphasis on evaluation, MIGA’s learning initiatives are another vital contributor to the strong performance. MIGA is continuing to conduct joint learning events with IEG with active participation by the self-evaluation teams. These joint learning events are targeted to reach all staff within MIGA and have been recognized as one of the good practices for learning from project self-evaluations. Moreover, MIGA has also successfully launched its ex ante development impact assessment tool, the Impact Measurement and Project Assessment Comparison Tool (IMPACT), which is serving to enhance focus on project selection and development impact by assessing potential project outcomes and foreign investment effects, combined with a likelihood factor for realizing them. MIGA believes attention to the results, both ex ante and ex post project approval, as well as learning from the self-evaluation process, including IEG’s validation assessments, will contribute to “smarter” risk taking where new ideas and approaches may be needed to address increasing challenges for achieving results in MIGA’s priority areas in IDA FCS, climate change, and recovery from the pandemic.

Outcome-Type Analysis and Classifications

MIGA welcomes IEG’s outcome-type analysis to examine the relationship between a project’s outcome types and its results. Unlike IEG’s analysis for IFC’s investment projects, IEG’s analysis was, for MIGA-evaluated projects, before the IMPACT framework was introduced. MIGA welcomes the RAP 2021 outcome-type analysis to fill this gap; however, as IEG highlights, this exercise comes with many methodological challenges and is not equivalent to a full IMPACT assessment. Therefore, MIGA fully agrees with the importance of heeding the RAP 2021 call for a cautious interpretation of the results of MIGA’s outcome-type analysis.

IEG’s analysis pointed out that MIGA projects have a higher probability of achieving project-level outcomes than foreign investment–level outcomes. Given the nature of systemic “beyond the project” effects of the foreign investment mobilized by MIGA guarantees, the IMPACT framework sharpens the ex ante assessment by critically assessing plausible changes the project can bring and the likelihood of realizing such effects. MIGA is looking forward to further dialogue with IEG on pragmatic ways of evaluating systemic foreign investment-level outcomes, the timing of achieving these effects, and credible and practical indicators that measure the changes.

Moving Forward

The report suggests that MIGA consider assessing the prevalence of different outcome types and other characteristics in projects to help enhance the system. The full implementation of the IMPACT framework is intended to sharpen the focus of assessing the project’s outcome types. Also, the system uses likelihood ratings for both project outcome and foreign investment effects to incorporate the risk for realizing development impact claims. MIGA would like to monitor IFC’s experience of the AIMM framework and its likelihood adjustments, including costs and benefits of incorporating the assessment of likelihood (or risk) at the claim level, should IFC proceed to incorporate this change, and implications of likelihood adjustments for ex post evaluative development outcome ratings.

The report also suggests the Bank Group could further emphasize operating on the frontier as a goal in addition to meeting the corporate scorecards rating targets. MIGA added new impetus to its strategic focus on product innovation and product application that embraces the spirit of risk taking on the frontier and helps address the decline in foreign direct investment exacerbated further by COVID-19. Examples in FY21 include (i) new approaches to developing the solicitation and filtering of innovative ideas from staff and clients, (ii) progress on new approaches to scaling up climate finance activities, (iii) new solutions to MIGA’s support for capital markets transactions, (iv) scaling up of capital relief approaches for financial institutions, and (v) the introduction of a new trade finance product in partnership with IFC. MIGA welcomes IEG’s suggestions and looks forward to further dialogue on how operations on the frontier can be evaluated ex post without disincentivizing risk taking and innovative approaches.