Develop and pilot territorial and national-level
measures of adaptation-related outcomes and
impacts for inclusion in an improved results
framework.
Current results frameworks on resilience are
not outcome-oriented and risk emphasizing
spending over results.
Original Response: Agreed: World Bank: Multiple challenges need to be addressed
to make progress in implementing this recommendation.
These challenges include developing
sound methodologies, gathering consistent and
comparable baseline data, and developing indicators
that cover institutional, household, and biophysical
characteristics in pilot projects. Working
with the new World Development Report (WDR) on
"Risk and Uncertainty," management will develop
methodologies and processes to capture intermediate
and long-term outcomes. Management will
work to mobilize resources in FY13-14 to develop
this system and design pilots.
Action 2A: Synthesize challenges and opportunities for developing individual and composite indicators and score cards from existing relevant work within WB, multilaterals and individual countries.
Indicator: Synthesis of outcome indicators and results framework of relevance for measuring resilience outcomes.
Baseline: No existing synthesis.
Target: Knowledge product on outcome indicators, challenges and opportunities for measuring climate resilient development.
Timeline: End FY14
Action 2B: Develop quantitative and qualitative outcome indicators for enhanced resilience in socioeconomic (including households), biophysical systems and institutions and pilot test them in at least four countries.
Baseline: No existing and acceptable outcome indicators, scorecards for climate risk included in results framework.
Target: 4-5 core and intermediate resilience outcome indicators and at least one scorecard system at national and sub-national level (including rural and urban areas) included in results framework and piloted in at least four countries, including Kenya and Zambia.
Timeline: FY14-17
The Bank has addressed many of the elements of the original recommendation, but as of 2017 these largely remain as pilot efforts and there is little evidence that these have been mainstreamed into World Bank programs.
The Bank has carried out and implemented a significant work program that develops measures of resilience or approaches for developing these at a national level and at a project level. The socioeconomic resilience measure represents a useful and workable national level measure of resilience to natural disasters. By focusing on well-being rather than purely on economic value, it could potentially help to influence project selection and design decisions in ways that will support building resilience for the poor. However, work on using the tool at the country level is in early stages, and no evidence has been presented that the tool is being applied in World Bank projects.
As noted in earlier updates, the PPCR indicators provide a useful first step in assessing resilience in 11 countries, focused on the existence of certain institutional functions and other outputs, rather than on their performance. But they do not have the ability to assess results on resilience from most interventions, this will likely require resilience indicators at the project level.
The Bank's work on resilience indicators at the project level has made significant strides, with the development of an overarching results framework for resilience and a detailed sectoral adaptation of this for agriculture. However, other sectors have not yet carried out similar work, and as of 2017 only a handful of projects have used this to inform their design of indicators. Adoption has been slower than originally envisioned the action plan had envisioned that the knowledge product would be completed by the end of FY14, but in practice it was completed only in FY17. In FY17, IEG supported a joint learning engagement with Bank management to support development of indicators and dissemination of the overarching results framework.
Thus, the Bank has made significant strides in developing resilience measures, but further work is needed to operationalize these in the Bank's core business. Nonetheless, the action plan has been largely implemented and significant progress has been made.
During FY17, the WBG continued to work on adaptation-related outcomes and impacts to be included in results framework. The WBG's efforts took the form of preparation of a flagship report, technical assistance, and continued monitoring of adaptation-related outcomes Regarding 2A, the WBG published a flagship report ("Unbreakable: building the resilience of the poor in the face of natural disasters") in November 2016 that include a new quantified definition of socioeconomic resilience, taking into account characteristics like poverty, poverty incidence, coverage by social protection and safety nets, financial and technical capacity of the government to provide post-disaster support, financial inclusion for households and access to insurance. This national-level indicator of socioeconomic resilience has been calculated in 117 countries and can be used to track progress of countries over time, and to assess the resilience benefit of various intervention (especially non-structural interventions). Country-level applications of the methodology have started, such as in the Philippines and Sri Lanka. In the Philippines, the government has announced that the tool will be used to prioritize investments on disaster risk management. As for 2B, the "Results Monitoring & Impact Evaluation for Resilience-Building in Operations" project is linked to the broader programmatic TA: "Enhancing climate and disaster resilience of World Bank Sustainable Development Operations." Since the last Management Update, the project has developed an overarching results framework for climate and disaster resilience, and indicator 'clusters' and features, which can serve as an organizing framework within the Sustainable Development Practice Group (SD PG) and be further adapted to specific contexts. The project is partnering with IEG (through its learning fund) to facilitate sector-specific conversations on theories of change/results frameworks, indicators, and impact evaluation. The IEG also supported a workshop in June 2017 to bring together internal and external experts to discuss the progress achieved by the project to date, and share learning and experience from across other institutions. A decision meeting for this project will be held in September 2017, at which time a mainstreaming strategy will be discussed. In addition, the World Bank is leading the monitoring and reporting of resilience outcomes at project and national level in 11 countries that are supported by MDTF Pilot Program for Climate Resilience (PPCR). During FY17, five core indicators have been tracked across all the PPCR countries. Robust and replicable methodology and criteria for scoring are being used for monitoring. The aim of such exercise is also alignment or linkages to the national Monitoring and Reporting system. The scoring process enables stakeholder (government, non-government) to assess on a qualitative basis the outcomes on resilience. A scoring exercise has enabled countries (such as Niger, Samoa, Zambia) with inputs from communities, business and government agencies to assess progress towards resilience, specifically in relation to institutional strengthening to coordinate efforts across sectors, inclusion of climate change into national planning processes, enhanced implementation and mainstreaming of climate and disaster risk in development, as well as the types and number of direct and indirect beneficiaries that have been able to better cope with climate-related events. One outcome has been (for example in Samoa and Zambia) the inclusion of climate and disaster risk and resilience in all sectors with links to budgetary processes.
The continued work on the resilience indicator pilot is positive and consistent with the IEG recommendation. As noted in earlier updates, the PPCR indicators provide a useful first step in assessing resilience in 11 countries, focused on the existence of certain institutional functions and other outputs, rather than on their performance. But they do not have the ability to assess results on resilience from most interventions, this will likely require resilience indicators at the project level. The âResults Monitoring & Impact Evaluation for Resilience-Building in Operationsâ program is intended to provide this support, however it has not yet produced its reports or sector-specific results frameworks.
The 2015 management update summarized the ongoing work on the pilot resilience indicators focused on flood risk. The national-level pilot resilience indicator has been further tested and adjusted. The model to calculate the ability of a country to minimize the impact of asset losses on well-being has been further refined, and results are now available for floods for 90 countries (see update under action 0252). Further work is ongoing to apply this indicator to other hazard contexts. In addition, analytical work is advancing in selected countries (Bolivia, Philippines, and Vietnam) to apply this model and indicator at the sub-national level.
The "Results Monitoring & Impact Evaluation for Resilience-Building in Operations" project is linked to the broader programmatic TA: "Enhancing climate and disaster resilience of World Bank Sustainable Development Operations." Since the last Management Update, under this project, a multi-GP effort a process to develop a sector-specific results frameworks and indicators, and associated guidance notes and toolkits. The project will also produce a study by the end of calendar year 2016 on evaluation approaches for WB resilience building projects. Finally, direct support to operations across the GPs is being provided to improve the results frameworks and indicators.
The World Bank is leading the monitoring and reporting of resilience outcomes at the project and national level in the 11 PPCR countries. Five core indicators are tracked across all the countries. The work includes developing baselines, robust and replicable methodology and criteria for scoring for each indicator in a consultative manner. The aim of such exercise is also alignment or linkages to the national Monitoring and Reporting system. The scoring process also enables stakeholder (government, non-government) to assess on a qualitative basis the outcomes on resilience. A scoring exercise and the has enabled countries (such as Niger, Samoa, Zambia) with inputs from communities, business and government agencies to assess progress towards resilience, specifically in relation to institutional strengthening to coordinate efforts across sectors, inclusion of climate change into national planning process, enhanced implementation and mainstreaming of climate and disaster risk in development as well as the types and number of direct and indirect beneficiaries that have been able to better cope with climate-related events. One outcome has been (for example in Samoa and Zambia) the inclusion of climate and disaster risk and resilience in all sectors with links to the budgetary processes. The PPCR countries have thus successfully integrated core and intermediate resilience outcome indicators in their development strategy.
The Bank is also initiating work in two PPCR countries with designing and implementing impact evaluation for resilience building projects in these countries.
The development of a national level resilience metric represents important progress in adopting a more outcome-oriented approach. The approach is consistent with the IEG evaluation, which emphasized the need to make resilience indicators actually measure resilience in terms of vulnerability to and response to shocks. Continued work on this measure through initial pilots for a range of countries and stressors, followed up by mainstreaming into Bank SCDs and CPFs will be important.
The creation of the Results Monitoring & Impact Evaluation for Resilience-Building in Operations work program demonstrates that Bank Management has recognized the importance of improving results frameworks and is making this a priority. Continued support for the program will be important for completing implementation of the recommendation.
As noted in the 2014 IEG update, the PPCR indicators provide some useful qualitative assessments of capacity they are limited to PPCR countries or to ouputs directly provided by PPCR projects, but represent a useful first step. It will be important to learn from this experience and apply it to broader Bank operations.
IEG notes that while Bank Management indicated at the time of the evaluation that it partially agreed with some recommendations, this recommendation was fully accepted by Bank Management (see original evaluation page xxxvi http://ieg.worldbankgroup.org/Data/reports/cc3_full_eval_0.pdf ). In drafting the recommendation, IEG took note of Management concerns that full adoption and implementation of improved results frameworks in operations might not be feasible within the four year timespan of the Management Action Record, so the final language of the recommendation required only development and piloting.
Estimates of the costs of building resilience and on methodologies for decision-making under uncertainty are covered by other recommendations this recommendation is focused on results framework development.
IEG looks forward to specific examples of the types of indicators recommended in the evaluation:
a) Institutional measures of adaptive capacityâ including the status of hydromet systems, disaster relief management systems, and agricultural extension systems and the geographical coverage of vulnerability assessments.
b) Household measures of vulnerability and exposure: e.g. based on household surveys that combine information on exposure to climate and other shocks with measures of consumption or food insecurity.
c) Biophysical measures of vulnerability and resilience: such as measures of water use sustainability and of recurrent urban flooding.
2A, B: 2A, B: The national-level resilience indicator was further developed in FY15 to comprise a new quantifiable definition of socio-economic resilience the ratio of asset losses to welfare losses, or the ability of an economy to minimize the impact of asset losses on well-being and resist, absorb, accommodate and recover in a timely and efficient manner to asset losses. The resilience metric was applied to rank policy levers in 90 countries, according to their country-specific efficacy to reduce risk of flooding to welfare. A key result is a set of scorecards, which describe each country with a set of sub-indicators. In FY16 this work is being tested in selected pilot countries, starting with the Philippines and will also seek to expand the national resilience indicator to other stressors (e.g., droughts).
A new work program, âResults Monitoring & Impact Evaluation for Resilience-Building in Operationsâ was initiated in May 2015 to develop a systematic approach and support the application of M&E for lending projects with climate change adaptation and/or disaster risk management objectives. The work has produced a draft scoping study and an expert workshop that brought together about 60 M&E/resilience experts (including 25 external) to inform recommendations for strengthening M&E systems. The outcome would thus be results frameworks, indicators, and evaluation techniques. These will be tested in a small number of projects and the framework further developed.
By July 2015, all 18 PPCR countries have submitted their national Monitoring and Reporting progress reports. These provide progress on resilience against 5 core indicators through the use of score-cards (on national processes) and against progress on specific project. The broad-based participatory projects included representation across sector agencies, private sector, CSOs/NGOs, and academia.
Early stages of implementation are showing considerable costs of resilience, and the willingness of the governments to continue with the implementation because of the well-articulated socioeconomic benefits. The Bank only partially agreed to this recommendation and articulated what would be needed to measure impacts and sustainability. Through a range of work, including implementation of operations, analytical work, a range of cost estimate for âresilienceâ or âadaptationâ measures.
Implementation of ongoing operations in Nepal, Samoa and Zambia show that including climate and disaster considerations into the design of infrastructure (dams, canals, and coastal roads) investments can take about 12-18 months and cost US$1.5 to 2 million. The costs of building resilient infrastructure (with climate change projections to 2050) varies considerably depending on the activity and the location. A coastal road very close to current sea level has been estimated to cost three times the base price. The socio-economic benefits for this road for trade, access to communities and airport and lower maintenance as well as sustainability means that this is likely to be funded.
Methodologies to support Decision Making Under Uncertainty are applied to address some of the issues related to the costs, benefits and sustainability of resilience measures, and consider a range of plausible future climates. Three main types of activities are underway: (i) Raising awareness on the importance of an explicit management of uncertainty via interactive presentations and serious games (ii) Mainstreaming uncertainty management in Bank projects' economic analysis â ongoing work with TTLs from the Global Practices to develop guidelines (iii) Implementing DMU methodologies to support investment decisions in water supply systems in Peru and Mexico, transport sector in Peru, Ecuador, and Colombia, hydropower planning in Nepal, and robust urban wetland management in Colombo, Sri Lanka.
IEGâs evaluation recommended that indicators be developed to address institutional measures of adaptive capacity, household measures of vulnerability and exposure, and biophysical measures of vulnerability and resilience (page xxv).
IEG agrees that no single indicator can measure and track changes in resilience, but believes that intermediate indicators could cover many of the items suggested in the IEG evaluation.
The Bank has made progress in developing some outcome-oriented indicators that will be assessed for all countries. The primary corporate scorecard indicator assesses the number of countries prioritizing disaster risk management, which covers an important but incomplete portion of resilience. Other relevant indicators in the scorecard include the proportion of population living under water stress and the number of beneficiaries covered by safety net programs. Not all resilience indicators need be climate-specific.
The relevant IDA 17 indicators are largely the same as the scorecard indicators, plus input indicators of commitments to actions labeled as adapative or not about which IEGâs evaluation expressed caveats and concerns (page 14-15).
The PPCR indicators provide some useful qualitative assessments of capacity they are limited to PPCR countries or to ouputs directly provided by PPCR projects, but represent a useful first step.
But there is more scope for developing more detailed indicators at the project level on a pilot basis that assess the effectiveness of system performance rather than just the stated goals of systems. Measures of whether groundwater use is sustainable would help to demonstrate long term vulnerabilities. Measuring whether disaster risk reduction is a formal national priority is a good start â but it could be even more useful to assess in qualitative terms whether disaster risk management agencies are functioning well. Hydrometeorological system performance could be assessed by asking whether weather stations were reporting and whether data was publicly available. Exposure to natural disaster risk (especially floods and cyclones) could be assessed as well as exposure to water stress.
The IDA-related work, PPCR-related indicators and a work program for developing resilience indicator(s) form the main part
of the work under this recommendation.
Existing indicators, indices and approaches used by other institutions were synthesized and formed background for an expert
workshop in July 2014. The conclusion was that there was no single indicator that can measure and track changes in
resilience.
IDA17 Results Measurement Framework and the WBG Corporate Scorecard includes a set of indicators that can be used to
measure outcomes of resilience to climate and disaster risks in core development sectors, e.g., water and agriculture.
These were considered to be intermediate indicators.
Development of disaster and climate resilience indicator(s) to measure progress at national level was started at this
workshop and included experts from a range of development, private sector and research organizations.
The PPCR, funded through the CIFs and implemented through the MDBs, has five core indicators which countries use to track
progress towards climate-resilient development at the national level (including activities directly funded by the PPCR).
These indicators are: 1) mainstreaming climate risk and resilience in sectoral and national policies, 2) evidence of
strengthened government capacity and coordination needed for such efforts, 3) quality and extent to which climate
responsive instruments/ investment models are developed and tested, 4) extent to which vulnerable households, communities,
businesses and public sector services use improved PPCR supported tools, instruments, strategies and activities to respond
to climate variability or climate change, and 5) number of people supported by the PPCR to cope with the effects of climate
change.
The PPCR countries submitted their baselines and targets progress towards targets for year 1 is underway since June 2014.
There is also a set of intermediate indicators that are being tested by the PPCR countries.
Lessons learned from developing and applying the indicators have been developed as part of the PPCR support including that
through the WBG. Knowledge products are available from the PPCR websites.