As explained in my earlier blog, evaluation of a budget support operation differs from evaluating an investment project for several reasons. These include the fact that the time needed to achieve objectives often extends beyond the time horizon of the operation and that the individual conditions that need to be met for disbursement (i.e., prior actions) represent only a part of the effort required to implement the reforms supported by the operation.  As a result, evaluation of a budget support operation needs to pay particular attention to the credibility of the results chain linking prior actions to the objectives of the operation.  And while rarely a sufficient condition for achieving policy objectives, a prior action should represent a meaningful and credible movement along the results chain, perhaps by meeting a necessary condition for achievement of a targeted outcome.  This requires an understanding not just of evaluation methodology, but also subject matter expertise.  

To address this and other challenges, in 2020, the Independent Evaluation Group adopted a new framework for evaluating development policy operations (DPOs), the name for World Bank budget support operations. This framework, which is fully consistent with the OECD-DAC evaluation criteria, focuses on the relevance of individual prior actions and results indicators and the adequacy of the evidence that objectives were achieved. One of the goals of the new framework is to increase the learning value of evaluation by providing more concrete feedback on what “good” looks like to staff designing and implementing budget support operations.

Capturing the credibility and relevance of prior actions

Assessment of the relevance of prior actions requires an awareness of the underlying country context and should be informed by an understanding of the plausibility of subsequent, necessary steps along the results chain.  This implies that the credibility of an identical prior action (and therefore its relevance) could differ between countries.  For example, while submission of a piece of legislation necessary for policy reform may be a credible prior action when a government has a majority in its legislature and has demonstrated the ability to pass the legislation it supports, the same prior action would not be a credible move along the results chain in a country where there is no reasonable expectation of passage by the legislature, for example because of known vested interests with the power to block passage.   A prior action that – a priori – stands little chance of triggering further movement along a results chain is therefore unlikely to be considered sufficiently relevant.

Given that DPOs often support complex reforms that require time to implement, a “relevant” prior action is not, on its own, expected to be sufficient for achieving the desired outcome.  Therefore, documentation for a well-designed DPO is expected to demonstrate awareness of the subsequent actions that are required for impact and their feasibility in context.  These could entail passage of legislation submitted to the legislature under the operation, adoption of implementing regulation, training and capacity building of officials, credible enforcement mechanisms, including a functional judiciary. 

Awareness of the full results chain and its feasibility is critical for informed risk taking given that a prior action is only a single intervention along the results chain.  Informed risk requires awareness of the weak links in the results chain (e.g., a hostile or captive legislature). As such, program documents for DPOs would be expected to identify the main risks to achieving development impact and identify mitigating measures where feasible.  In IEG’s new framework for evaluating DPOs, assessing the adequacy of the discussion of ex ante risk is an important component in rating the adequacy of Bank Performance. 

Measuring the effectiveness of budget support

The new approach also scrutinizes more closely and systematically the relevance of the indicators used to measure progress toward objectives.  A “relevant” indicator will measure progress to the desired development outcome as a result of the prior action or actions in the DPO.  Results indicators that measure progress to the development objective, but which cannot be attributed to the prior action would not constitute evidence of DPO impact. For example, if a prior action requires changes in the personal income tax rate, a results indicator that measures VAT revenue would not be viewed as providing evidence of impact even if the overarching objective of the operation were to increase tax revenue.

Similarly, a results indicator that captures the impact of a prior action, but which does not measure progress toward the development objective would also not provide evidence of DPO effectiveness. For example, if the objective were to improve public investment management, a results indicator tracking the number of meetings held by the oversight committee for public investment would not provide strong evidence of impact.   Or if the objective were to increase agricultural productivity and the results indicator measured tons of wheat produced, the results indicator would not be adequate as it would not capture the critical relationship between inputs and outputs that is the essence of changes in productivity.

More meaningful prior actions and better measures for impact

While not without its own challenges, the new methodology does provide a more credible basis to identify weaknesses in the logic underpinning the selection of prior actions and the credibility of the framework used to measure impact. Over time, we hope that this will lead to use of more meaningful and impactful prior actions and better measures for assessing their impact. The new framework also provides a more structured assessment of Bank performance which draws on criteria that better reflect the efforts expected of staff in designing and implementing DPOs. Its use will be critical to accountability and more effective learning as budget support is used to pursue an increasing array of development objectives. In the end, we hope this leads to more effective use of budget support in pursuit of development goals.