Strengthening Results Frameworks in Development Policy Operations
Three key ingredients to support a strong results framework in Development Policy Financing operations
In an earlier blog, Is Budget Support an Endangered Species? Why giving cash directly to developing countries may (still) be a good idea, we looked at some of the reasons why budget support (development policy financing or DPF) remains an important instrument for many developing countries and why it accounts for nearly a third of the World Bank's total lending today.
The key challenge for many projects is to ensure that the objectives reflect the binding constraints that borrowers face in an effort to maximize the returns from reform.
In recent years, IEG has been working on various aspects of DPF to derive lessons from the World Bank's experience with this important instrument. There has been a special focus on learning more about results frameworks as the core of DPF results orientation.
The World Bank uses an objectives-based model to underpin the results framework, which captures agreed-upon specific policy and institutional changes the DPF is designed to influence. In looking at the World Bank's DPF programs, IEG's reviews have identified three key ingredients to support a strong results framework - setting the right objectives, identifying credible prior actions; and strengthening reporting at completion.
In reviewing the World Bank's DPF portfolio, IEG found that, for the most part, the World Bank projects strike a good balance between breadth and specificity when defining lending objectives. However, the key challenge for many projects is to ensure that the objectives reflect the binding constraints that borrowers face in an effort to maximize the returns from reform. In this respect, the use of systematic country diagnostics (SCD) recently introduced by the Bank to identify borrowers' key constraints may help to strengthen policy focus of DPF. IEG also identified examples where development policy operations (DPOs) were prepared to address sudden external financing shocks but without reflecting intended macro and fiscal objectives in the policy matrix. Lack of "operationalization" of macro objectives (setting specific macro objectives supported by necessary actions) when it is necessary undermines DPF relevance in critical macro and fiscal issues. This is likely to be a more significant weakness in the absence of parallel IMF programs.
A second key ingredient in building strong results frameworks has to do with identifying prior actions. Prior actions refer to the actions that a country has to meet to demonstrate its commitment to seeing through the policy reforms. They are an important indicator of a government's commitment to reform. IEG found wide variation in Bank experience, including examples of excellence as well as instances where prior actions were not credible or deep enough to trigger change (and should have raised red flags). IEG is working towards strengthening evaluation criteria to better assess the quality of prior actions with special focus on criticality, relevance, value-added by the World Bank, and measurability.
In many cases, IEG attributed the weaknesses in prior actions to weak implementation of programmatic series rather than to the original design. Flexibility over PAs, a major feature of the instrument that allows to adjust DPOs to possible changes in country circumstances, can be quite useful in addressing some of the shortcomings of the Bank's previous "rigid conditionality". However, the World Bank's experience of past decade shows that balancing flexibility and performance remains a challenge. IEG found examples of significant deterioration in the quality of prior actions as a result of non-implementation of triggers - to the extent that the final operations would likely not have been approved by the Bank as stand-alone operations.
A third and equally important ingredient is the role of completion reports. DPO teams often are better equipped to identify output-oriented rather than outcome-oriented results indicators due to intrinsic difficulties in measuring policy and institutional change. Reporting at completion, thus, is a supplementary tool to reflect on important developments beyond specific results indicators, and is important for understanding the extent of achievement of expected results. For example, if a given development policy operation identified the clearance of arrears as the results indicator to measure strengthening of public financial management, it would be useful for the completion report to go beyond this output indicator and outline other important changes in public financial management that would point to the direction of overall change. Recent changes shifting the preparation of investment completion reports for DPOs from 6 to 12 months after project closure are a positive step that will enhance the World Bank's ability to better capture development results in line with stated objectives.
With the move to systematic country diagnostics and their use in country strategies to achieve a clear "line of sight" from growth and poverty reduction priorities, the World Bank has an important opportunity to strengthen the design of DPF operations, with clear objectives, credible prior actions, and increased emphasis on learning from evaluation at completion.