Many World Bank Group projects display strong performance. These results however tell us little about the Bank Group’s aggregate contributions and higher-level development outcomes—the results measurement systems are designed to tell a different type of performance story.
The World Bank Group’s (WBG) results measurement systems for projects, programs, and thematic areas are purposefully designed.
Generally, they focus on collecting data on indicators that can be attributed to Bank Group interventions. They help verify whether targets have been met. They provide consistency in displaying results across portfolios. Importantly, they allow shareholders to hold the Bank Group accountable.
In our flagship report, IEG aggregates and analyzes much of this data to give the big picture on the Results and Performance of the World Bank Group (RAP 2020).
The 2020 RAP report finds that the World Bank and MIGA performs strongly overall, and that IFC’s decline in ratings that started ten years ago appears to have halted. We welcome you to dive deeper into how various arms and instruments of the Bank Group perform, and what factors help explain these trends.
At the same time, while being able to attribute results to Bank Group projects remains important for reasons of accountability, there is also a need to better capture how the institution contributes to lasting positive change in client countries.
We refer to this as higher-level development outcomes—social and economic change on the ground which Bank Group interventions cannot achieve singlehandedly, but which the institution can meaningfully add to, over time.
How well does the World Bank Group contribute to these sorts of development results? RAP 2020 finds that answering this question would require searching beyond the institution’s existing results measurement systems.
Limited evidence on higher-level outcomes is gathered
Following an analysis of the Bank Group’s results reporting—from projects to country programs to thematic priorities such as gender and climate change—the RAP 2020 arrived at the conclusion that the World Bank Group collects limited systematic evidence on its contribution to higher-level outcomes.
Higher-level outcomes stem from the interplay of different projects and types of Bank Group engagements—lending, knowledge, and convening—over time. The World Bank Group’s Board has requested more evidence on these sorts of development outcomes and how the interventions help achieve the Sustainable Development Goals (SDGs). Better evidence on higher level outcomes would also help with learning, reflections on strategy, and course corrections where needed.
Many of the World Bank Group’s existing corporate results measurement systems were designed to collect data needed for ratings and for process and compliance monitoring. These systems do what they were designed to do.
For example, individual projects’ self-evaluations mostly do a good job at capturing project-level results. IEG rates these self-evaluations, as part of a system that provides consistency in displaying results across sectors, across types of projects, and across time. Yet combining performance ratings from individual projects cannot give a clear picture of the important development outcomes to which the Bank Group contributes—the Bank Group’s contribution to higher-level outcomes amounts to more than the sum of its parts. And ratings are not the same as outcomes.
IFC, under its 3.0 strategy, has adopted tools to help it more clearly work toward outcomes. Through its AIMM system, IFC assesses all investments for their anticipated direct and indirect effects, including catalytic effects on markets. It is too early to tell how IFC’s new tools will influence outcome achievement and incentives. Although IFC can ensure alignment between projects and IFC’s higher-level goals because of its focused business model, the World Bank operates with objectives that are more diverse because of its diverse sector and country contexts.
Results in thematic areas focus more on compliance than outcomes
Consider gender and climate change. The Bank Group is committed to achieving important outcomes for its global work in key areas such as gender and climate change. Its measurement systems, however, place more emphasis on the input level—such as compliance with climate co-benefit commitments and gender tags—than on capturing changes arising from Bank Group operations.
The climate change results measurement system, for example, uses 35 targets and indicators to monitor how well the WBG integrates climate change into operations and strategies. 90 percent of these indicators relate to actions under the Bank Group’s control, including inputs (such as financing for climate action), internal processes (such as greenhouse gas accounting), and outputs (such as the number of activities that support cities with climate-related policies).
This approach is useful to track fulfillment of corporate commitments, drive accountability, and ensure operations adhere to process requirements. In a nutshell, the system creates incentives to mainstream climate action. But the system has limited focus on the quality of programs and on understanding the higher-level change the programs are meant to support.
What an outcome orientation would mean
Imagine a situation where staff spend less time checking boxes and collecting data that the Bank Group needs for its own internal reporting purposes and have more time engaging with clients on programs’ contribution to development outcomes. Imagine a situation where data, evidence, and systems oriented the Bank Group toward achieving higher-level outcomes. We call this outcome orientation.
An outcome orientation means both generating evidence on what works, what does not, and why; and using this feedback to engage clients and adapt programs to boost contribution to development outcomes.
Many parts of the World Bank Group are already studying their outcomes, learning from data and evidence, and adjusting interventions. Some of the corporate results systems could better support staff in this. IEG’s newly published evaluation on The World Bank Group Outcome Orientation at the Country Level finds that the country-level results measurement system does not effectively support teams in making course corrections to country programs.
The World Bank Group’s results measurement systems bring rigor and discipline to performance assessment, and there remains a strong case for assessing Bank Group achievements at the project level so that they can be accounted for in a focused way.
At the same time, more deliberately contributing to development outcomes requires shifting focus beyond the project level and beyond current results measurement and incentives structures. IEG is to this end committed to supporting the World Bank Group’s outcome orientation.
This is an outstanding…
This is an outstanding article and captures well the misplaced emphasis given to “results” and its concomitant regime of compliance box ticking as opposed to focus and doing the work to actually deal with/measure/monitor outcomes. A fundamental issue is that in the previous decade the majority of the “results” cadre in the Bank has been made up of relatively junior and development inexperienced individuals who know well the language and jargon of monitoring & evaluation but without knowledge of development on the ground. This has led to a tyranny of logframes, theory of change etc. limited to cut charts, annexes, graphs that neither the borrowers or management pay scant attention and in the end, has been a box ticking irritant to technical staff.
Shouldn't management be…
Shouldn't management be encouraged to lay out the higher level thematic outcomes it is aiming at and to articulate transparently how it proposes to achieve such outcomes through its country programs?
Excellent blog. The question…
Excellent blog. The question is: how to effectively move towards a more outcome oriented approach? As the blog correctly points out, achieving sustainable outcomes would mean looking at the broader program /sector scale rather than the project scale. The M&E system would need to be mainstreamed into the country systems so that it can measure the impact of various projects / policies over time. This calls into question the project approach as we know it. We need to put a stronger focus on how the results we achieve through our projects can be expanded to the entire country or province (whatever the appropriate scale is). A transformational impact can only be achieved if local / country systems (like the education system, or a transport system) are improved in order to deliver a better outcome for the people they serve. This in turn requires long-term collaboration and partnership for trust building.
Of course, we (the technical teams) try doing that with our clients through our engagement over time (over multiple projects). But the rapid staff turnover and the lack of incentives for long term outcome orientation severely reduces our ability to do so. My claim is that the development project (be it an IPF or a P4R or a DPO) is the wrong tool to deliver development assistance and that we need to engage in longer term, flexible partnerships in order to achieve meaningful development outcomes.
Happy to discuss that further.
Dear Francois Thank you for…
Thank you for the comment which I am just seeing now. IEG's evaluation of country programs' outcome orientation suggests many of the same things you mention, including a focus on the long-term, looking beyond what individual projects can accomplish or be held accountable for, and working with clients to develop and use their systems far more.
See here: https://ieg.worldbankgroup.org/evaluations/world-bank-group-outcome-ori…, and
Add new comment