Improve the results frameworks of Bank supported SSN projects to (1) more clearly identify and address the needs of specific groups of poor or vulnerable, and (2) identify how project objectives fit within longer-term objectives for development of country SSNs. This involves improvements in the quality of objectives, design, and monitoring within projects, as well as development of a longer-term results framework for building effective SSN programs and systems
Original Response: Management partially agreed with this recommendation. Management agrees with and fully appreciates the need for improving results frameworks and of identifying project impacts on beneficiaries, as was borne out by our 2010 internal HDNSP results-readiness review. However, as long-term results and outcomes go beyond the scope of individual projects, management will focus on strengthening the application and consistency of medium term indicators in the results chain to monitor performance of projects. The HD network will also promote strengthened and consistent quality across the Bank through tools, analysis of outputs, impact evaluation and support to task team in results.
While SSN projects have generally performed well, they often have weak result frameworks. Many projects have used impact evaluations to design the project's interventions as well as to assess the project's impacts. However, this is a largely an ex-post analysis and does not provide a logic model to understand how the intervention functions. This is particularly true for the increasingly complex projects that the Bank supports.
The Global Practice has invested in improving the quality of its results framework. In particular, the Global Practice reports that it has included advanced training in the development of result framework both for staff (through its annual Global Forum) as well as in its online training. This had led to clinics in seven countries. In the 2015 fiscal year, the Global Practice conducted 17 (ex-ante) Quality Enhancement Reviews that included extensive discussions on the results framework.
The Global Practice has also invested in strengthening a global database on social protection outcomes. This includes data that better allow the measurement of the distributional impact of World Bank's interventions as well as consolidating data on impact evaluations related to SSN. This is ongoing work that will support the development of result frameworks.
While these activities lay groundwork for improving the quality and relevance of result frameworks, there is no evidence that this has been incorporated at the project level. The evaluation requires additional evidence of changes in the design of result frameworks.
Core responses to IEG recommendations continued to be along the following two main pillars:
1. Strengthening results frameworks in the new projects. This includes the use of core sector indicators in project results frameworks for all new investment projects. This also included on demand training and mentoring in the use of the new indicators. The practice has supported operational staff by organizing multiple quality enhancement reviews that, among other activities, help task teams develop improved results frameworks The supporting materials include technical guidance notes on results readiness in safety net operations and a technical note on result- based lending. The GP maintains TORs for assessing results readiness in projects and put in place a consultant roster that includes technical specialists in this area ( http://ssnroster and http://safetynetstor .)
2. Investment in creating a data portal with country level indicators of performance of SSN that are easily accessible , up to date and accurate (ASPIRE portal), as well as development of tools and capacity building for comprehensive country-led diagnostic and progress assessment through Inter agency social protection assessment tools (ISPA).
In FY 15 the practice conducted 17 QERs on new projects in preparation, all of them including in-depth discussion of results frameworks. The most in-depth discussions took place for the results indicators in LIC settings. For example, in Madagascar (P149323), a country with extremely high poverty rate and malnutrition, the team had proposed in the original design the indicators of improved food security that would be impossible to monitor properly given the statistical capacity and the factors going beyond the project. The team has managed to streamline the indicators, linking them directly to project outputs.
The GP conducts annual SSN core course (attended by close to 100 practitioners form client countries, donor agencies and WBG staff), where SSN results framework is one of the main training modules.
The in-depth advanced training on Impact evaluations and results was provided during the June 2015 SPL GP Global Learning forum for staff. The material with presentations are available on line, and the staff with the proven skills in developing results monitoring frameworks are part of the global experts poo in the new architecture of knowledge work in the Global solutions groups established by the practice in FY15. The staff also use on-line training platform (jointly developed with FAO on SSN). In addition, there were several country-level clinics in the use of results indicators and in-depth analysis of impacts at the country and regional level (Philippines, Morocco, Romania, Georgia, Indonesia, Tanzania, Mozambique). All these in-depth workshops led to the preparation of notes and other knowledge outputs.
The results agenda has been mainstreamed into practice through the development of the ASPIRE portal (www.worldbank.org/aspire), a global database on SPL programs in terms of their coverage, efficacy and adequacy. The management has given a priority in allocating scarce BB funds to the data work and it continues to serve as the flagship initiative ion the Global knowledge work. This allows to look at results beyond individual projects, at the country level over time. By measuring the distributional impacts of social protection and labor systems, the ASPIRE database also directly monitors the contribution of social protection policies and programs to achieving the Banks twin goals. The ASPIRE database, accessible online (www.worlbank.org/aspire ) includes key country- and program-level indicators for SSN, social insurance and labor programs. These are calculated using nationally representative household surveys, and are the result of a careful process of quality assuran
IEG notes the considerable emphasis that the Human Development network placed on improving results frameworks of social protection projects, including the use of core sector indicators, the preparation of technical notes on results readiness, quality enhancement reviews to help task teams prepare results frameworks. This is commendable, however, evidence that the results frameworks have in fact improved due to these efforts would be useful. IEG emphasized that results frameworks need to be expanded beyond the duration of the specific projects when objectives are longer term, even if intended results requires the involvement of different actors. Accountability for this set of SSN recommendations is for the WB management, and not only for the SP sector board. IEG agrees with management rating of medium (i.e. adopted in some operational and policy work but not to a significant degree in key areas) and recommendations working beyond the SP sector to measure progress towards intermediate development outcomes. IEG rates adoption of this recommendation as medium.
The Social Protection and Labor Sector Board has introduced the use of core sector indicators, as appropriate, for all new investment projects. Training in the use of the new indicators is on-going.
The HDNSP anchor has prepared technical guidance notes on results readiness in safety net operations and a technical note on result- based lending. It has also developed TORs for assessing results readiness in projects and put in place a consultant roster that includes technical specialists in this area.
HDNSP has supported operational staff by organizing multiple quality enhancement reviews that, among other activities, help task teams develop improved results frameworks.
A major conference on impact evaluation of CCT took place in November 2011 (sponsored by the office of the Chief economist of HDN) and was widely attended by development practitioners, staff from regions. Over 20 evaluations, majority linked to the Bank's projects, have been presented.
The sector is an ongoing effort to improve the quality of its results frameworks. If fully implemented, this will lead to a change in practices in the sector. These actions include:
The sector has developed a set of core indicators for monitoring by all regions and it has introduced these in new projects as appropriate.
The HD Chief Economist's office has started providing training on the design and development of results framework, targeting staff preparing new projects. This training is complemented by a recently prepared paper (October 2012) that provides detailed guidelines to project teams to prepare and utilize results frameworks specifically for social safety nets.
The sector has initiated quality enhancement reviews (QER) for projects under preparation. These QERs have largely focused on helping the preparation team improve the results framework and ensure that there is a logical causality chain. To date, these QERs have been voluntary. The sector reported that teams are incorporating the comments received from the QERs.
The sector is actively disseminating toolkits and information on how to incorporate results in the design of investment project. This standard package is disseminated to all teams as they begin project preparation.
These actions represent important steps towards strengthening the result frameworks of social safety net projects, and IEG looks forward to reviewing their progress next year.
The SPL strategy puts results as a central focus, with a focus on evidence as the first principle of engagement. There is also a strong results framework, linked to core sector indicators, that has targets and annual updates.
Continued support to the regions on results frameworks is taking place, notably through quality enhancement reviews
A compilation and update from the IEG work on social safety net impact evaluations is being completed.
The ASPIRE database on country results from household surveys and administrative data currently includes data from 150 countries and is being updated for FY14.
The Bank has made a major effort to increase the availability of data for social safety nets
The Bank is continuing to develop the Atlas of Social Protection Indicators of Resilience and Equity (ASPIRE), which consolidates a variety of data sources on social protection programs. While coverage is far from universal, the ASPIRE database has extensive information on the targeting of benefits from a variety of SSN programs. This largely focuses on income distribution, such as the benefits received by the poorest 20 percent of the distribution. ASPIRE is growing, with more countries participating (data was received from 44 new countries in 2014) as well as covering new technical areas. This database provides access to information on how SSNs and other projects target poorer groups. However, it does not appear to have information on targeting by other criteria, such as gender or ethnicity.
At the country level, the Bank has worked to develop social or poverty registries. These are important elements in a number of investment projects, including in Bangladesh and Jamaica. The Bank also supported countries with well-developed poverty registries, such as Mexico, to improve their systems.
Despite the improvement in the collection of data, there is no evidence about how this has improved the quality of objectives and result frameworks. Nor is there any evidence of the development of long-term result frameworks.
The results agenda has been mainstream into practice through the development of the ASPIRE portal (www.worldbank.org/aspire ). As part of the institutional priority to monitor country progress in their social protection and labor system performance, the Bank has made a major effort to pull together, for the first time, a global database (ASPIRE) on SPL programs in terms of their coverage, efficacy and adequacy. ASPIRE indicators contribute to monitor country progress on social protection and labor outcomes and the implementation of the WB 2012-2022 social protection and labor strategy, which aims to help countries build social protection and labor systems. By measuring the distributional impacts of social protection and labor systems, the ASPIRE database also directly monitors the contribution of social protection policies and programs to achieving the Bank's twin goals.
During FY 14 performance indicators (generated through ADePT SP) extended to 60 new countries to update 8 existing countries (using the most recent available household surveys have been harmonized). In addition serious progress has been made to create full country level inventories of all social protection programs. Total number of countries available in ASPIRE now is 122, which makes it THE LARGEST AND MOST COMPREHENSIVE data source on social protection in the world. ASPIRE also includes a number of methodological and capacity building activities in partnership with other agencies to improve quality and harmonization of data on social protection. The ASPIRE external portal (www.worldbank.org/aspire) has been launched in May 2013 and it is linked to the WB Databank. Since July 1, 2013 to present, there have been 16,943 page views, 6,068 visitors and 7,974 visits.
Building on the Independent Evaluation Group's 2011 meta-analysis, the SSN team has updated a comprehensive impact evaluation database. After careful quality verification, a total of 52 additional impact evaluations were added to the existing database. Currently, the database contains 220 impact evaluations published between 1999 and 2014, and 129 of them study the impact of the Bank-supported interventions. This review focused on findings from the evaluations that were carried out between 2007 and 2014 and examined key results and trends in those evaluations. Impact evaluations were considered to be credible if they met four criteria: (i) they had a development focus; (ii) they included counterfactuals; (iii) they were robust; and (iv) they included quality checks. Some of the results are summarized in the report 'state of Safety Nets 2014 described above.