Clearly define objectives and assess benefits, costs, and feasibility of policy alternatives to ensure the most appropriate use of SSNs. Comparing alternative options for reaching the specific objective(s) is particularly important in contexts of high poverty and tight budgets and encourages specific SSN objectives.
Original Response: Management only partially agreed with this. In its proposed actions, Management follows IEGs suggested focus on the formulation lending operations, but with the understanding that at that stage of decision-making the range of options under consideration by government is rather narrow, essentially focusing on alternative features of a given type of program or more commonly, further development of an existing program. In the tools to improve consideration at this stage, the Bank will focus its efforts on the core objectives of poverty reduction through cash or in-kind transfers and building effective institutions.
The Global Practice has made advances in using economic analysis to identify the role of the SSNs at the country level. The newly introduced Systematic Country Diagnostics (SCD) allows strategic questions about alternatives to be ask and analyze the contribution of SSNs in reducing poverty. The Global Practice reports that it have been engaged in all on-going SCDs work. In addition, the Global Practice is carrying systematic assessment of alternatives for SSN based on their costs and expected benefits through its regional reports.
At the project level, the GP is relying on cross-agencies cooperation to measure the Value for Money (VFM) of SSN interventions. This has led to the presentation of several toolkits and guidance notes and the GP is starting to include the methodology in its project design.
For example, the economic analysis of the Madagascar Social Safety Net Project presents some evidence of the effectiveness of similar interventions, although this is based on a rather selected number of cases. It also presented a quantitative analysis of the project's benefits which outlined the project's value added. However, the economic analysis did not include a cost-benefit analysis or, more importantly, a discussion of alternatives. In contrast, the Burkina Faso Social Safety Net Project included an extensive discussion of the potential impact along with an analysis of the project's costs and benefits. The South Sudan Safety Net and Skills Development Project also included an extensive discussion of the different options as well as the economic justification for the project.
The Global Practice has launched ASPIRE, which consolidates data on spending on SSN programs for over 100 countries with complete inventories. Likewise, the Global Practice has developed the ADEPT, which contains a module with reports on targeting effectiveness of SSN transfers. The manual for users has been finalized and will be published. The SPL has organized a number of standalone training courses on ADEPT (Mongolia, Tunisia, Morocco, South Africa) as well as incorporating it in its Core Course on SSN. Despite these advances in developing tools, there remain data gaps in costing existing SSN in many countries and there is often not comprehensive reporting on the fiscal spending on SSN. In part, this reflects the range of SSN interventions, including cash transfers, the provision of in-kind support (often from the Ministry of Agriculture), and the provision of temporary employment and training (provided by a wide range of Ministries).
The GP has made significant advances in improving its capacity to measure the benefits of its support at the country level. It also appears to be improving the quality and depth of its economic analysis of projects. The collection of additional data will play an important in measuring information.
Based on this evidence, IEG rates the progress on this recommendation as substantial.
The way to address the IEG recommendation heads-on is engagement with the countries in Systematic Country Diagnostics when strategic questions about alternatives are being asked. SPL GP is engaged in all on-going SCDs work and flagship examples with deep reflections on strategies and priorities is recently completed SCD for Brazil.
Another avenue for systematic assessment of alternatives for SSN based on their costs and expected benefits are regional reports
The GP is relying on cross-agencies cooperation in the area of documenting practice in assessing Value for Money in SSN. There were several toolkits and guidance note prepared by ODI with the financing form DFID, which were extensively reviewed by WBG SPL and disseminated to the staff. A number of joint initiatives with DFID ensures that the principles of VFM assessments are used while designing projects and programs.
There is a serious data gap on costing existing SSN globally. Current statistical standards do not contain comprehensive reporting on the fiscal spending on SSN. IN parallel with data work with the statistical community through SPIAC B to improve the standards, SPL GP has launched ASPIRE , which now contains data on spending on SSN programs for over 100 countries with complete inventories.
The SP ADEPT is a software program maintained by the SPL GP. It contains a module with reports on targeting effectiveness of SSN transfers. It is being applied across countries to produce summary outcome indicators made available in ASPIRE. The manual for users has been finalized and will be published. The SPL has organized a number of standalone training courses on ADEPT (Mongolia, Tunisia, Morocco, South Africa) as well as made it integral part of the Core Course on SSN.
IEG notes management's greater emphasis on improving targeting effectiveness, of disseminating information on SSN effectiveness in general, and of improving the capacity of project teams to conduct cost-effectiveness analysis, including as part of impact evaluations. Cost effectiveness analysis was recommended by IEG because it requires a clear definition of specific objectives and assessment of alternatives. IEG's analysis of the results frameworks indicated very general objectives related to poverty reduction (or even reaching the poor). IEG agrees with management that cost-effectiveness analysis can be undertaken to clarify project design; however, it also notes the importance of using this tool more upstream in the country team deliberations of its poverty reduction strategy and defining clearly how SSNs fit into that. Unfortunately, management provides no evidence that the cost-effectiveness of policy alternatives are actually being assessed, which is the thrust of the recommendation, IEG rates adoption of this recommendation as medium.
The issue of effectiveness of SP was a key theme of one Background paper for the strategy: Productive role of Social Protection. The paper is published and is being disseminated (Marc h 2012).
The SP ADEPT module reports on targeting effectiveness of SSN transfers. It is being applied across countries to produce summary outcome indicators made available in SP Atlas. The manual for users has been finalized and will be published and disseminated by the end of FY12. The new version 5 is about to be launched (March 2012)
The SP Atlas (ASPIRE) was launched on a pilot basis in November 2011 and officially at the spring meetings. It contains information to benchmark results in SSN across 50+ countries and regions. SP&L anchor staff are working on analytical notes on the results, to be posted on the dedicated web site FY12-13.
A workshop on assessing cost-effectiveness took place in February 2012 for Bank economists to improve project team capacity to carry out the necessary analysis.
The sector is carrying out training (through the HD Chief Economist's office) to support teams to improve the quality of economic analysis and has included a training course for preparation teams that has been offered three times in the current fiscal year. It has also supported the HD's anchor knowledge exchange to improve the quality of the economic analysis. The social protection sector has also started to require that all new projects include cost-benefit analysis if possible or to discuss the choice of alternatives and how the final design was chosen.
These steps are important initial steps to strengthen and improve the use of economic analysis for strengthening the design and implementation of social safety net projects. However, QERs are not done on all SSN projects and it remains to be seen how deeply thisincreased attention to analysis and designwill be reflected in improving quality at entry over time.
Cost effectiveness analysis is being increasingly incorporated in the PADs for the projects that are developed. Among 8 QERs conducted this FY, 4 have focused on strengthening this component. In case there is not detailed CB analysis, there is a request to compare alternatives.
Developed ADEPT training courses (delivered 3 times in F13, including 1 focused session at the HD week for Bank operational staff)
Developed material and included a course module on cost-effectiveness analysis as part of the SIEF-supported impact evaluation regional training workshops (4 delivered in FY13 for HDN staff and clients)
Using TF resources the SPL anchor, in cooperation with the regions, is conducting detailed assessment of cost effectiveness of SSN components across a number of countries. The analytical results will be published as SSN Primer Notes on the cost-effectiveness of delivering cash transfers: beneficiary registries, payment mechanisms and MIS (FY13-14)
Continued support to the regions
All SSN Quality Enhancement Reviews in FY2014 focused on strengthening the economic analysis and results framework. The previous MAR indicated that the Social Protection sector developed training and tools for economic analysis in projects.
The Bank has provided support to both project and government teams in designing social safety nets, including choosing among options. This includes training on designing pensions and social safety net. The Bank is also developing tools to improve the targeting of social safety nets, through the existing ADePT tool. This was supported by complementary training.
The effort described in FY13 was sustained. The main vehicles to pursue this agenda are two fold:
(i) Quality enhancement through peer review and Quality Enhancement Reviews
(ii) Training programs and resource for WBG staff.
Cost effectiveness analysis is being increasingly incorporated in the PADs for the projects that are developed. All 9 QERs conducted this FY 14 (Bangladesh, Burkina Faso, Cameroon, Congo, Ethiopia, Mali, Mozambique, Senegal, Lebanon), have focused on strengthening the economic analysis component, results framework and indicators.
ADEPT training courses were delivered 4 times in F14, including one on-demand session at the HD week for Bank operational staff:
The following trainings have been delivered by the ASPIRE team on the use of ADePT SP and pension:
World Bank Core Course: Pension - Trainings delivered to 50 policymakers and practitioners from 24 countries on Poverty Analysis Using Household Surveys , ADePT SP Overview and Coverage, and ADePT Pension , March , 2014.
Pensions in the Middle East and North Africa: improving Retirement Income Protection Programs - Training delivered on survey analysis and applications, as well as on ADEPT Poverty, SP, and Labor to 36 participants from 16 countries, February, 2014
World Bank Core Course: Social Safety Nets - Trainings to 120 policymakers and practitioners from close to 50 countries on Poverty Analysis Using Household Surveys, ADePT SP Overview and Coverage, and ADePT SP and Targeting. The core course included policymakers practitioners from all the Bank regions. December 2013.
Training on ADePT Labor: Overview and Applications to LAC HD and SP staff October 2013