How do we know when budget support has been effective?  When does budget support as provided by multilateral development banks like the World Bank achieve durable and meaningful development progress? As an evaluator, I am the first to admit that the answer to this question is more challenging than it seems at first. Above all, it is critical to distinguish the budget support dimension from the development program that is being supported by a given operation. While we often hear terms like “budget support operations” we have to be clear that the impact of the “budget support” dimension of those operations is not what is being evaluated. This is because by the nature of these operations the financing goes into the central treasury of the government in question, free of conditions. Even if one wanted to track this spending, by design it would not be possible to distinguish it from the overall budget expenditures. This is why, instead of budget support operations the World Bank refers to development policy operations (DPOs). It is the development policy that we seek to evaluate.

The evaluation of development policy operations (DPOs) therefore focuses on the impact of the reforms supported by the DPO which are reflected in the set of “prior actions” that must be completed for budget financing to be approved. 

The World Bank defines prior actions as policy and institutional actions deemed critical to achieving the objectives of a program supported by the development policy operation. These present the legal terms defined in the financing agreement. The borrower is required to complete all prior actions before the DPO is approved by the Executive Board and budget support disbursed.   

“Program” in this formulation is understood to be the set of reforms that the government commits to implement or is implementing that supports achievement of a particular policy objective or objectives.  At the World Bank, this objective (or objectives) is referred to as Program Development Objectives (or “PDOs”). However, a ‘reform program’ is often not clearly circumscribed, either in substance or time. This raises challenging questions about the relationship between the “policy and institutional actions” supported by the DPO and the “program” supported by the DPO. 

Unlike investment projects which generally involve concrete and finite tasks that are well defined in time and space, DPOs support a set of discrete “actions” to move a new or ongoing reform program (or part thereof) forward along a causal or results chain. At times, and depending on the quality of the underlying documentation, that chain is more implicit than explicit, leaving the evaluator in the position of having to reconstruct it from program documentation. 

A Government’s reform program is almost always more extensive and wider ranging than the set of prior actions contained in a DPO. One difficulty is that the objectives articulated for a DPO are often pitched at a relatively high level (e.g., supporting fiscal consolidation, improving the quality of public service delivery, promoting financial stability). Objectives that are more closely aligned with the scope and ambition of prior actions do make it easier to assess performance (e.g., “raise VAT revenues” rather than “increase domestic revenue mobilization” or “foster fiscal sustainability”).  While a high level of ambition can help spur change, it can unfortunately create a disconnect between the scope of the prior actions and that of the formal objective. That said, scrutinizing prior actions can help determine the de facto scope of the operation, which is important from an accountability standpoint.

Moreover, DPOs can have a duration of as little as a single year (in the case of a stand-alone operation) and (infrequently) up to four years (for some programmatic series of DPOs), with most DPOs having a duration of one to three years.  This is often too short a period to move from a prior action to the desired outcome and measurable results.

How then does one evaluate a DPO when achievement of its approved objectives often extends beyond the time horizon of a DPO and when the focus of its prior actions represents only a part of the effort required to implement a reform program?  This is a challenge that is increasingly front of mind for independent evaluation offices of major multilateral development banks, particularly as DPOs are being used to support an increasingly wide range of reform objectives, including gender, climate change, the mobilization of private finance, and sector-specific reforms.  Having a clear and consistent answer to this question is of increasing importance given the increase in the use of development policy operations, a trend which was in evidence even before COVID. 

For example, it is noteworthy that two decades ago, it was rare for the World Bank to provide budget support to countries with weak institutional capacity (which included many lower-income countries and countries in fragile and conflict-affected situations).  Since 2019, just over one third of IBRD resources (35.3%) and more than half of IDA resources (54.2%) have financed budget support operations in about 83 countries.  For fragile states, while the number of DPOs has averaged around 10 per year in the last decade, average commitment size has tripled from US$43 million over Fiscal Year (FY) 10-15 to US$127 million over FY16-22. Today, it is common practice, particularly in the wake of the COVID-19 pandemic and Russian invasion of Ukraine when budget support was one of the few options available to rapidly channel support to economies impacted by high food and fuel prices, and sharp slowdowns in the global economy.

Given the magnitude of financing being allocated to budget support, and methodological challenges in evaluating DPOs, it is not surprising that several MDBs have recently changed or are in the process of reviewing their approaches to evaluating the impact of their budget support operations.  

The World Bank’s Independent Evaluation Group has been at the forefront of these efforts having adopted a new framework for evaluating DPOs back in 2020.  This new framework builds on changes adopted by World Bank management for self-evaluation of DPOs but extends it and is more granular in an effort to derive more concrete, focused and operationally relevant lessons from the evaluation process.  

The next blog in this series will describe IEG’s new approach for evaluating budget support operations and how it can be used to improve the design and impact of World Bank budget support.