This report expanded on past RAPs by focusing not only on core performance as assessed through ratings but also on outcomes and relationships between ratings and outcome levels. This chapter draws out some key findings, conclusions, and implications for the Bank Group’s COVID-19 response and then for its outcome orientation. It finds that there are trade-offs between using results measurement systems for tracking commitment targets and outcome orientation. Confronting these trade-offs is necessary if the Bank Group wants to better support outcome orientation.
Findings and Conclusions
The analysis of performance showed positive ratings trends for World Bank and MIGA projects and Bank Group country programs in IBRD countries. The analysis linked the positive outcome ratings trends to strong work quality on project design, implementation support, and M&E, and broadly conducive economic and institutional conditions in many larger countries before the pandemic. Performance trends for IFC projects and in FCV-affected countries are less positive, albeit with signs of recent slight improvements for IFC. Less successful results were often linked to large shocks and issues with projects’ and programs’ preparation for risks and their response when shocks occurred.
The analysis of outcomes showed that most IPF objectives cluster at level 2 around quality and access to services, though a few sectors state IPF objectives at level 3 with a clearer focus on end beneficiaries. Most DPFs state their objectives at level 3 with a focus on policy reform outcomes, and recently approved IFC projects often state their objectives at level 3, particularly in relation to market creation objectives. Projects’ outcome levels have only a modest relationship with their ratings, and the relationship becomes insignificant when controlling for other factors. Looking beyond projects, the analysis showed limited higher-level outcome data. The existing results measurement systems collect evidence needed for ratings and for process and compliance monitoring, which is different from evidence on outcome achievement.
Based on the evidence and findings, this RAP concludes that the Bank Group can improve how its incentives and results measurement systems support outcome orientation. At the project level, many projects with higher-level objectives manage to achieve good IEG ratings, in part by having strong results frameworks to measure outcome achievement. Even so, it would not be realistic or desirable to expect all World Bank projects to have objectives at outcome level 3 or 4, as discussed in box 4.1. At the country program level, the Bank Group has opportunities to take a broader and more strategic view beyond individual projects, yet there is often little evidence on higher-level country outcomes, as discussed in IEG’s forthcoming evaluation of country programs’ outcome orientation. At the corporate level, the Bank Group’s extensive systems cover different thematic work areas and collect process and output indicators to help senior management incentivize and report on operations’ fulfillment of corporate policy commitments, but they do not help staff to manage for higher-level outcomes.
Box 4.1. Setting Project Objectives
Objective setting needs to balance the opposing demands of realism and ambition. Realism demands that objectives be achievable, given the projects’ resources, timeline, and context. Objectives that are far removed from project interventions jeopardize the ability to show contribution. Outcome achievement should be measurable. Country and sector context, geographic scope, and beneficiaries are some of the context factors that also matter. Ambition, however, demands objectives with a line of sight to systemwide, transformative, or other important higher-level outcomes. Ambition also demands result frameworks that measure how the project changes beneficiaries’ conditions, with attention to gender and distributional aspects. Balancing realism and ambition requires judgment and dialogue between client counterparts and the World Bank. Therefore, universal rules are unlikely to be helpful. In practice, it is plausible that some projects aspire to and achieve outcomes at a higher level than those captured in their objectives.
Source: Independent Evaluation Group.
Excessive focus on monitoring targets can cause a risk-averse corporate culture and stifle staff’s intrinsic motivation to pursue positive social change. IEG’s evaluation of the Bank Group’s self-evaluation systems found that staff often have little use for the collected data and find little value in it. Instead, incentives are to focus on checking the box, meaning meeting targets and feeding the demands for corporate monitoring data (World Bank 2016a). At the same time, a corporate culture focused on compliance, disbursements, and meeting targets can induce risk aversion, reduce openness about problems, interfere with staff learning and experimentation, and stifle how staff use evidence to pursue outcomes. For these and other reasons, there are trade-offs between outcome orientation and using results measurement systems for reporting and incentivizing fulfillment of policy commitments.
Confronting trade-offs related to the purposes of the Bank Group’s results measurement systems is necessary for improving outcome orientation. The corporate results measurement systems for projects, programs, and thematic areas were purposefully designed to meet the Bank Group’s need to collect data it can report to shareholders to show attributable results and that allow shareholders’ representatives to hold it accountable. The purpose of tracking and reporting results data dictates what systems collect, leading to a focus on tracking commitments using quantifiable indicators of activities and lower-level results that can be attributed to Bank Group interventions, added up across portfolios, and used to verify whether targets have been met. Systems designed for such purposes are not geared to understanding and managing higher-level outcomes.
This RAP’s findings and conclusions have implications for the Bank Group’s ongoing response to the pandemic and other shocks. Some projects may need more robust implementation support and more frequent course correction during implementation, both to respond to shocks and unforeseen circumstances and to counter the potential effects of short preparation times on quality at entry. M&E systems across the Bank Group need to enable such responses and can do so by maintaining sight of project objectives and enabling teams and clients’ identification of issues and nimbler course corrections. The World Bank’s and clients’ administrative procedures for restructuring and canceling projects could be streamlined. Rating systems should not unduly penalize necessary changes to targets and objectives set at approval that may need adjustment in light of COVID-19 and other shocks. Box 4.2 explains what that might involve for IFC. Furthermore, the analysis of IFC’s performance highlighted the need for enhanced tools and processes to identify and mitigate market, country, and sponsor risks.
The analysis of country programs in low-capacity countries suggests that when adding new elements in response to large shocks, there is a need to simplify other program elements to avoid overtaxing country capacity. More broadly, such programs need to be designed from a premise of high risk. This includes aiming for short-term gains, sequencing longer-term reform agendas into discrete items with shorter time frames, and avoiding overburdened programs.
Box 4.2. The Coronavirus Pandemic and IFC Project Ratings
The coronavirus pandemic represents a shock to International Finance Corporation projects, putting the economic and financial sustainability of those projects at risk, at least temporarily. As one response, the International Finance Corporation and the Independent Evaluation Group are discussing how to adjust ratings processes and methodologies to account for shocks like the pandemic, which make the projects’ implementation environment and country context more challenging. Proposals include making project objectives more realistic by rating projects based on their midcourse correction targets rather than those set at approval (before the shock occurred), and giving the International Finance Corporation more flexibility to choose the evaluation timing, which may help projects recover and meet targets later.
Source: Independent Evaluation Group.
The outcome orientation findings suggest a need to rethink the approach to collecting outcome evidence beyond the project level. The existing self-evaluation instruments and results measurement systems aggregate data from individual projects, but higher-level outcomes result from the interplay of different projects over time, something that none of the Bank Group’s existing self-evaluation instruments capture. In line with past practice, this RAP does not make formal recommendations. However, IEG’s forthcoming evaluation of country programs’ outcome orientation discusses how using a wider set of methods and a focus on contribution rather than attribution could help support longer-term thinking and engagement on how the Bank Group contributes to important country-level outcomes. This includes more flexibly accounting for shocks and necessary program adjustments and better capturing the Bank Group’s contribution to institutional change in countries (box 4.3).
It would be helpful to differentiate the purpose of collecting outcome evidence. At project level, setting objectives and assessing achievements that can be attributed to Bank Group support continues to be important for the institution’s accountability and credibility. This requires realism when setting projects’ development objectives as discussed in box 4.1. But beyond the project level, for results in country programs and at thematic levels, the purpose of collecting outcome evidence should not be to track and report and hold the institutions accountable for attributable results. Assessing outcomes often requires dedicated, context-specific evidence, which does not always lend itself easily to portfoliowide aggregation. Outcome evidence can be robust when based on sound evaluation methods that use plausible theories of change and credible data to relate Bank Group activities to observed outcomes in sectors and countries.
Box 4.3. A Fresh Approach to Understanding Country Outcomes
The Independent Evaluation Group’sevaluation of country programs’ outcome orientation finds that a satisfactory self-evaluation instrument would need to go beyond the present approach, which is centered on “results frameworks premised on metrics, attribution, and time-boundedness” (World Bank 2020). A self-evaluation instrument suited for collecting higher-level outcome evidence would have to cover a longer period and focus on a sector or country to capture contributions to outcomes and assess the cumulative effects from multiple World Bank, International Finance Corporation, and Multilateral Investment Guarantee Agency lending, knowledge, and convening interventions. “A renewed country-level results system could conceive accountability differently, based on evidence of achievement and failures and description of learning and adaptation. It could acknowledge that the Bank Group can influence but not control country outcomes. It could recognize that country teams cannot decide all targets and objectives at design but must adapt during implementation” for reasons relating to shocks, uncertainty, changing circumstances, and, especially for the International Finance Corporation and the Multilateral Investment Guarantee Agency, unpredictable client demand. And it could realize that capturing contributions to country outcomes and assessing cumulative effects from multiple interventions requires dedicated evaluation inquiries, not just measurement of indicators. Data for such a renewed system could come from existing project evaluations, impact evaluations, ratings, stakeholder surveys, and other sources.
Source: World Bank 2020.
IEG plans to continue producing annual RAPs that aim to provide a broad perspective on the Bank Group’s performance. Though the exact shape of future RAPs is still undecided, IEG will continue its efforts to offer a lens through which to understand outcomes and outcome levels across sectors and Bank Group institutions. The Bank Group exists to work with its client countries on improving human conditions. A clear focus on outcomes helps it stay on course.
- Based on research covering many development agencies (including the World Bank), academic Dan Honig discusses how to promote staff’s intrinsic motivation to achieve outcomes. Arguing for more “navigation by judgment,” Honig suggests promoting a less risk-averse corporate culture that embraces bold ambitions and gathers, uses, and learns from outcome evidence (Honig 2018, 2020).