Scope and Questions

The evaluation covered project approval FY 2012–22, and the focus was on country-level social protection systems and responses. The 10-year period allowed the evaluation to capture evidence of effectiveness from the closed portfolio of lending projects with social protection activities. The period aligns with the World Bank’s 2012–22 social protection and labor strategy and a particular interest in studying the World Bank’s approach to adaptive social protection (ASP) since its inception in the Sahel.

The evaluation team took a consultative and modular approach to engagement. It engaged with World Bank management and project teams, operational support and Country Management Units, and technical experts to discuss sampling considerations, analyses, share preliminary findings, and receive feedback.

The evaluation aimed to answer two questions:

Evaluation question 1: To what extent has the World Bank support for ASP been relevant?

To answer this question, the evaluation included three subquestions:

• Evaluation question 1a: To what extent has the World Bank supported ASP elements in countries where vulnerability to covariate shocks is higher?
• Evaluation question 1b: To what extent has the World Bank incorporated ASP elements into its social protection support, and to what extent are these aligned with good practice and evidence of what works?
• Evaluation question 1c: To what extent is the World Bank ASP framework a realistic model in different settings?

Evaluation question 2: How effectively has the World Bank supported ASP outcomes (timeliness and adequacy of social protection response) in client countries?

To answer this question, the evaluation included two subquestions:

• Evaluation question 2a: How effective has the World Bank’s support been for ASP practices and activities?
• Evaluation question 2b: What has worked to achieve successful ASP outcomes in client countries? What factors explain success, and what was the role of the World Bank?

Conceptual Framework

The evaluation adopted a theory-based approach. It developed a detailed conceptual framework to guide its understanding of the World Bank’s contributions to ASP in client countries. The framework was developed by consulting the building blocks from the World Bank’s ASP framework—programs, data and information, finance, institutional arrangements, and partnerships (Bowen et al. 2020); structured literature review; a review of relevant Independent Evaluation Group (IEG) evaluations; and consultations with ASP experts. Figure A.1 is a simplified version of the team’s more detailed conceptual framework.

Figure A.1. Conceptual Framework

Image

Share Tweet Facebook WhatsApp Linkedin Email

Figure A.1. Conceptual Framework

Image

Source: Independent Evaluation Group.

Note: ASP = adaptive social protection; SP = social protection.

Defining Key Outcomes

As figure A.1 shows, the evaluation focused on timeliness, coverage, and adequacy of shock responses. The team’s definitions for these outcomes (presented in chapter 3) built on “Social Protection Responses to COVID-19 in the Global South: Tracking Matrix” and a methodological note on “Social Protection Responses to COVID-19 in the Global South: Online Dashboard” (IPC-IG 2021a, 2021b).

Timeliness. For sudden-onset shocks, timeliness is defined in terms of the number of days between the shock (hurricane, earthquake, and so on) in the country and the implementation of social protection responses (excluding subsidies). For example, for COVID-19, in many cases, it is possible to count the number of days between the first COVID-19 case (World Health Organization declaration), or government declaration of an emergency, and the provision of new or scaled-up social assistance measures. Where available for World Bank–supported programming, the case study authors reviewed either the International Policy Centre for Inclusive Growth’s database (IPC-IG 2021b) or government or media publications that mentioned the date of the first payment, date at which applications and registration started, or the date of budget disbursement. In addition, where possible, the case study authors reviewed response timeliness for vertical expansions to existing beneficiaries and horizontal expansions to new beneficiaries. For slow-onset disasters, the date of the shock is less exact, making assessment more complex, and the case study authors relied on various sources to determine when the crisis was considered sufficiently serious in different areas to require additional assistance. This is the case of lean season assistance in the Sahel, for example, where governments establish clear timelines for assistance (June, July, August).

Coverage. Measuring coverage relates to the proportion of people (or households) covered by programs as a proportion of the population affected by a shock. In the case of this evaluation, coverage expansions refer to the inclusion of previously uncovered individuals (or households) either through the expansion of an existing social protection program or a new intervention. Calculations are often difficult to make because of a lack of available, up-to-date, and accurate data on how program expansions relate to particular shocks. The World Bank–supported programs tend to report cumulative beneficiary numbers without necessarily having an annual breakdown or connecting these annual beneficiary numbers to the numbers of people affected by particular shocks.

Adequacy. For social assistance, two key dimensions of adequacy are the value of transfers and the duration of transfers. The case study authors also assessed whether the monetary value of programs had increased relative to inflation. Duration is typically measured in months (although this was rarely measured or made available). In some cases, the team could consider the comprehensiveness of the assistance package to address the multidimensional needs of affected populations—that is, whether additional components are added, such as for health needs, livelihoods support, psychosocial support, addressing violence against women and girls, disability inclusion, and so on.

Methods Used

The evaluation adopted a mixed methods approach to answer key evaluation questions.

Structured Literature Review

The evaluation team conducted a structured literature review to identify good practices and evidence with respect to ASP, to assess the comprehensiveness of the World Bank ASP framework, and to understand what works and in what context more broadly. The review identified literature on ASP published in English, including both formal and gray sources. It focused on peer-reviewed scholarly sources (books, journal articles, and theses); gray literature from well-respected research institutes and think tanks; and reference documents from other international organizations with ASP-related projects, with the aim of exploring additional adaptive features emphasized by these organizations. The review used expert judgment to select and prioritize literature. Available stress tests were also reviewed to help understand levels of social protection maturity, and the evaluation team conducted various consultations and interviews with global social protection experts inside and outside the World Bank. Together, these helped put findings from a more focused portfolio review and the case studies in context and answer the evaluation questions about the relevance of the World Bank’s ASP framework and intervention areas.

Portfolio Review and Analysis

The evaluation team conducted portfolio analysis across 70 high-risk countries for both lending and nonlending support. For lending support, 67 out of the 70 countries had relevant social protection content, according to IEG’s manual review of project documents, and for advisory services and analytics (ASA), 51 countries had relevant content on social protection or disaster risk financing.

The evaluation’s portfolio included all lending operations where the Social Protection and Labor Global Practice is either a leading or contributing Global Practice. Lending operations consist of 202 projects in 67 high-risk countries and $52.6 billion in commitments approved between FY12 and FY22, which were manually reviewed and characterized against foundational, adaptive, and dual-use social protection interventions. This included identification of foundational social protection and ASP intervention areas in project development objectives and project components for investment project financing and Program-for-Results operations and development policy financing prior actions, an assessment of investment project financing and Program-for-Results results framework indicators and reported results achievement, and identification of project- and indicator-level factors of success and challenge. In addition, for a randomly selected 50 projects, gender-related activities were identified from Project Appraisal Documents using concepts defined in IEG’s evaluation on gender equality over the past 10 years (World Bank 2024).

For ASA, the evaluation team (i) conducted a targeted keyword search to identify a purposive sample of 141 ASA addressing ASP,1 which accounts for $141 million in total cumulative expenditures, and from which specific activities were reviewed in the country case studies; (ii) conducted a separate manual review of ASA to identify World Bank support to disaster risk financing;2 and (iii) identified and reviewed ASA tied to lending operations. The total number of ASA covered by the evaluation is 189 and accounts for $184 million in cumulative expenditures.

Country Case Studies

The evaluation team conducted six country case studies with in-country field visits and five desk-based case studies, selecting the countries by pursuing variation across cases using the following criteria: strength of World Bank engagement (including total engagement, emergency engagement, length of engagement, and coverage of social protection areas) and social protection maturity, exposure to shocks, political and institutional context of the country, income level, fragility status, and region. The team also considered logistical considerations. The selected countries were divided into two groups:

• Field-based case studies: Burkina Faso, Colombia, the Dominican Republic, Mauritania, Mozambique, and Nepal.
• Desk-based case studies: Ethiopia, Jamaica, Lebanon, Pakistan, and the Philippines.

Table A.1. Priority Shocks in Case Studies

Image

Share Tweet Facebook WhatsApp Linkedin Email

Table A.1. Priority Shocks in Case Studies

Image

Source: Independent Evaluation Group.

Note: Shaded cells indicate the shocks that were covered by the evaluation. This is not an exhaustive list of all covariate shocks experienced by the countries during the evaluation period.

The team developed a protocol for the field-based case studies that covered the conceptual framework and evaluation questions. The team tested and refined the protocol in Colombia and then used it in the other cases. A shorter and less comprehensive protocol was also developed for the desk-based cases. Each of the case studies conducted document review of sources internal and external to the World Bank.

The field-based case study protocol outlined a systematic approach for conducting country case studies, assessing the effectiveness, timeliness, and adequacy of ASP interventions from 2012 to 2022. The protocol was informed by established case study methodologies and research, incorporating insights from the Social Protection Approaches to COVID-19 Expert Advice Service and academic frameworks on case study evaluation. It provided a structured case study format, requiring cases to follow an inverted pyramid structure, in which key findings were presented first, followed by supporting evidence. The document emphasized transparency in data limitations, ensuring that evaluators acknowledge constraints in available information. The protocol established a rigorous evaluation methodology, defining key indicators and identifying data sources, including World Bank project reports, external literature, and international databases (for example, the Atlas of Social Protection Indicators of Resilience and Equity, INFORM Risk Index, and the Social Protection Floor Index).

A stakeholder mapping tool was incorporated to ensure that diverse perspectives are considered. In line with the protocol, for the field-based case studies, the team conducted key informant interviews with World Bank staff (task teams, country management, and other experts); government counterparts, such as senior civil servants and social protection experts in relevant ministries; development partners; implementing partners from civil society and humanitarian actors; universities; and think tanks. The desk-based case studies had far fewer interviews and consequently less rich evidence.

Semistructured interviews were conducted involving a sequential purchase of information approach (Raimondo 2023), starting with broad open-ended questions on the factors that facilitated or hindered the materialization of the outcomes of interest, followed by structured questioning on the subset of variables of interest.

The case studies mapped project intentions against the building blocks, analytic clusters, and intervention areas; identified internal and external enabling and disabling factors; used rubrics to understand the levels of progress and strength of evidence, in some cases; and were reviewed multiple times by the evaluation team.

The team performed comparative cross-case analysis by mapping evidence of performance in shock responses to (i) shock types and (ii) enabling factors and hindrances.

In addition to the country case studies, the team conducted a more targeted review of World Bank support to the Sahel Adaptive Social Protection Program, given its importance in pioneering the concept of ASP.

Global Interviews with Partners

The team carried out semistructured “global interviews” outside the case studies, with respondents from partner agencies and academia. These interviews focused on eliciting the respondents’ views on the evaluation questions.

World Bank Internal Expert Consultations

The team conducted several consultations with World Bank staff in social protection and disaster risk management. These consultations focused on validating the evaluation methodology, eliciting respondents’ views on an emerging hypotheses, and ensuring that the results resonated with operational realities.

Country-Level Analysis of Responses to COVID-19

The team used the “Social Protection Responses to COVID-19 in the Global South: Tracking Matrix” produced by the International Policy Centre for Inclusive Growth to estimate country-level measures of the characteristics of countries’ ASP responses to the pandemic. The rationale for using these data was to assess social protection responses to the only shock during the evaluation period that was common to all countries and the extent to which those responses were related to the World Bank Group’s engagement. This matrix aimed to include all social protection responses to COVID-19 in the Global South (Beazley et al. 2021; IPC-IG 2021b). The data set included information for 589 responses. There were responses from 121 countries, of which 53 matched the countries targeted in this evaluation.

Triangulation and Quality Assurance

The team systematically triangulated the evidence and mapped it to the evaluation questions (figure A.2):

• Its triangulation within the case studies emphasized drawing on multiple sources of secondary data (all cases) and primary data (field-based cases) and considering the “probative value” of evidence—that is, the extent to which evidence makes a particular explanation better or worse.
• Its triangulation across the case studies covered revision of the theory of change, factors of effective shock responses, and comparison across shock types.
• It also triangulated between global interview informants and country case study informants.
• Many of the final findings resulted from triangulation among the portfolio, the case studies, and the literature review. For example, the team looked for ways to ensure findings from both the portfolio analysis and the case studies on similar topics. It also considered portfolio and case study findings taking into account the literature review’s findings on good practices.

The evaluation team ensured evidence quality and integrity in multiple ways:

• The portfolio review ensured that more than one evaluator coded the data and that there was at least one reviewer of those data.
• Weekly team meetings provided regular opportunities for collective sensemaking and analysis.
• The evaluation team organized two analysis workshops to identify emerging key messages.
• The draft report underwent IEG’s standard quality enhancement process with multiple discussions of emerging findings and a comprehensive peer review of the draft report.

Figure A.2. Methods and Triangulation

Image

Share Tweet Facebook WhatsApp Linkedin Email

Figure A.2. Methods and Triangulation

Image

Source: Independent Evaluation Group.Note: ASP = adaptive social protection.

References

Beazley, Rodolfo, Marta Marzi, and Rachael Steller. 2021. Drivers of Timely and Large-Scale Cash Responses to COVID-19: What Does the Data Say? Social Protection Approaches to COVID-19: Expert Advice Service, DAI Global.

Bowen, Thomas, Carlo del Ninno, Colin Andrews, et al. 2020. Adaptive Social Protection: Building Resilience to Shocks. World Bank.

IPC-IG (International Policy Centre for Inclusive Growth). 2021a. “Social Protection Responses to COVID-19 in the Global South: Online Dashboard.” IPC-IG.

IPC-IG (International Policy Centre for Inclusive Growth). 2021b. “Social Protection Responses to COVID-19 in the Global South: Tracking Matrix.” IPC-IG.

Raimondo, Estelle. 2023. The Rigor of Case-Based Causal Analysis: Busting Myths Through a Demonstration. IEG Methods and Evaluation Capacity Development Working Paper Series. Independent Evaluation Group. World Bank.

World Bank. 2024. An Evaluation of World Bank and International Finance Corporation Engagement for Gender Equality over the Past 10 Years. Independent Evaluation Group. World Bank. https://ieg.worldbankgroup.org/evaluations/evaluation-world-bank-and-international-finance-corporation-engagement-gender-equality.