Back to cover

World Bank Group Gender Strategy Mid-Term Review

Chapter 5 | Ensuring and Measuring Progress

The gender strategy outlines a vision that targets key outcomes and pays significant attention to monitoring commitments and enhancing the quality of project design, but less attention to implementation. The World Bank and IFC both measure progress on closing gender gaps through reporting on commitments, project design, and implementation. The assessment of project design through tagging and flagging is subject to mandatory procedures that receive management attention because of target setting that overlaps with corporate commitments. Corporate commitments receive attention because they are of interest to the Board and shareholders. The contrasting limited attention to the monitoring and project evaluation processes during implementation introduces the risks of missing opportunities for adaptation and providing insufficient evidence on how the strategy contributes to outcomes.

Monitoring for Organizational Commitments and Corporate Reporting

The two main sets of commitments over the past 10 years that produce monitoring requirements are the IDA replenishments and the capital increase. Since the introduction of a gender strategy focused on results, IDA commitments on gender have shifted.1 The commitments under IDA16 and IDA17 focused on monitoring World Bank operational issues and collecting information on direct beneficiaries, specifically those related to human development indicators. In IDA18 and IDA19, the commitments shifted to a results focus based on the objectives of the gender strategy. The evolution of the IDA commitments is outlined in box 2.1 (chapter 2). Respondents in both the World Bank and IFC consistently reported that the required measurement of progress on IDA commitments continues to focus the attention of staff and management on gender.

The World Bank collects corporate data and reports on all IDA commitments and the capital increase; IFC reports on most of the capital increase gender commitments. For IDA, all commitments are reported through Mid-Term Reviews and through indicators in its corporate scorecard. IFC, meanwhile, has incorporated three out of the four capital increase commitments on gender into its corporate scorecard.2 The indicator yet to be reported by IFC is the amount of annual financing dedicated to women and women-led small and medium enterprises. These data are already available for FIG’s Banking on Women clients, and GBG has started discussions in FY21 to develop a relevant methodology for applying this indicator across other industries. Interviewees in both institutions regularly cited these commitments as drivers to consider gender in project design, particularly as management reviews progress against commitments.

IDA commitments contribute to enhanced corporate reporting of data relevant to the gender strategy and project design. Project monitoring data feed sex-disaggregated and gender-relevant indicators in both separate and joint scorecards for the World Bank and IFC. In 2014, before the implementation of the gender strategy, only four indicators in the Bank Group Corporate Scorecard provided information on progress related to gender gaps. By 2019, 10 indicators reported on issues relevant to the closure of gender gaps, mainly through sex-disaggregated indicators. As discussed in chapter 2, the strategy has driven a real increase in projects that are designed to close gender gaps.

IFC also monitors a range of indicators that feed into its corporate reports. The IFC Corporate Scorecard and year-end key performance indicators results report for FY19, for example, details the percentage of gender-flagged AS projects; new billion-dollar, long-term financial commitments with financial institutions targeting women; and the percentage share of women directors nominated to IFC Board seats. IFC also requires that all operations measure two standard indicators in the implementation of IS projects: the number of women employed through these projects and the number of women in executive management positions. Changes in these two indicators are not necessarily attributable to specific project activities, so they are more useful as a description of client contexts than as reflections of organizational performance. These two mandatory indicators are augmented by sector indicators that can be selected for both AS and IS projects within the reporting system, AIMM.3 Together, the IFC key performance indicator report and the standard and project-specific indicators assist in reporting as well as in the design of project monitoring systems that track the closure of gender gaps. Results and measurement staff reported that they expect to use data from these indicators to understand trends in how clients approach gender gaps.

Monitoring Implementation

Measuring contributions to the closure of gender gaps during the implementation of operations receives limited attention in the current system, with a few exceptions. The focus on the quality of project design and corporate reporting leaves fewer resources for monitoring implementation. The data analyzed here highlight a risk that paying limited attention to assessing progress through supervision and monitoring and evaluation will lead to issues reporting on the results of the strategy and missed opportunities to adapt implementation.

Interviewees reported various constraints on assessing implementation progress. Budget limitations were reported to prevent adequate monitoring of gender gaps during supervision. Existing data on gender gaps are often limited, and surveys are not resourced or are difficult to conduct in a project’s context. In IFC IS projects, interviewees reported focusing on monitoring output indicators (such as the capital increase and the key performance indicator long-term financial commitments targeting women). The exception was when project loan agreements require, as a condition of disbursement, the monitoring of gender issues related to risks identified and triggered in the IFC (2012) Environmental and Social Performance Standards. The latter include, for example, gender issues related to land resettlement; community health, safety, and security (including GBV); and labor conditions. Finally, World Bank staff report challenges in defining and applying indicators on gender gaps across all Regions. These challenges were more acute when project budgets did not afford projects the benefit of assistance from gender focal points or consultants (see chapter 4 on the constraints of focal points).

The results frameworks in the CPFs of the seven countries reviewed do not fully monitor implementation based on the gender gaps identified in their strategic pillars. The CPFs reviewed each have at least one indicator to monitor the closure of identified gender gaps through sex-disaggregated data. Across the CPFs, however, measures associated with gender gaps in jobs are absent. In five of the CPFs, although jobs are identified in the pillars, there is no corresponding indicator. For example, the Egypt CPF results framework does not include indicators to assess whether the program is closing gaps in the employment of women (World Bank 2015a). Disconnects were also noted in voice and agency in Peru’s and Tajikistan’s CPFs (World Bank 2017a, 2019b): though both identify GBV issues, neither has a corresponding indicator.

A review of the indicators from all 97 tagged and flagged projects from FY17 to the second quarter of FY20 in the sample countries found widespread planned measurement for removing constraints on jobs and control over assets, with limited measurement of voice and agency and of specific human endowments.4 The planned measurement of jobs and assets occurs because these topics are often paired with other gender gaps in projects. For example, a project focused on social protection, an endowment gap, will also aim to affect jobs or assets. In total, 52 and 38 out of 97 projects have planned measures on jobs and assets, respectively. Further, the largest number of indicators measures jobs (40 percent), followed by assets (26 percent). In contrast, only 15 percent of indicators measure changes in voice and agency. Human endowments cover about a quarter of projects and 20 percent of indicators. There is great diversity in the planned measurement of endowments; the indicators included access to digital learning, women’s representation in research at universities, and access to hepatitis C screening. The indicators for jobs and assets have a good degree of consistency, often focusing on the number of jobs created or the number of loans provided to individuals. For voice and agency, although Project Appraisal Documents often highlight GBV issues or a need for behavior change, as in CPFs, no indicator was present in the project results framework. These hard-to-measure areas require additional human and financial resources, which, as highlighted above, are unavailable to teams.

Review of the supervision reports of the same tagged and flagged projects found limited evidence of projects monitoring indicators that assess contributions to the closure of gaps. Table 5.1 shows that out of 91 projects approved between FY17 and FY19 (68 World Bank, 9 IFC IS, and 14 IFC AS), only 19 (13 World Bank and 6 IFC) reported progress on closing gender gaps in their Implementation Status and Results Report or in internal supervision reports.5 About 75 percent of projects have yet to report progress against indicators that pertain to gender gaps (54 World Bank and 14 IFC). Only 4 projects (4 percent) had no indicators to report on progress pertaining to the closure of gender gaps. Three of the 4 are IFC projects, which reflects issues that may occur with IFC self-reporting on the gender flag.

Table 5.1. Reports Tracking Progress Indicators Related to Gender Gaps

Tracking Status of Indicators Related to Gender Gaps

Year

Total

2017

2018

2019

World Bank

19

21

28

68

Progress already reported

4

9

0

13

No progress reported yet

14

12

28

54

Gender gaps not tracked

1

0

0

1

International Finance Corporation

6

5

12

23

Progress already reported

2

0

4

6

No progress reported yet

4

5

5

14

Gender gaps not tracked

0

0

3

3

Source: Independent Evaluation Group review of projects tagged and flagged for gender in Bangladesh, Côte d’Ivoire, the Arab Republic of Egypt, Kenya, Peru, Tajikistan, and Vietnam.

Limitations in measuring the results of the gender strategy are also evident in the lack of planned project-financed evaluation. Of the 91 projects reviewed, 69 (76 percent) did not specify plans to undertake their own additional assessments. This result is to be expected for IFC investment operations, as explained in the next paragraph, but there could be opportunities for further evaluation of World Bank projects.

IFC commissions few impact evaluations of gender gaps in its investment operations. An inherent challenge in evaluating IFC’s investment operations is that private sector clients may not agree to set aside financing for development impact studies as part of their IFC loans (box 5.1 comments further on the need to shift client perceptions). This review was not able to identify any impact evaluations of investments that contributed to addressing gender gaps outside of FIG. For FIG, three evaluations and one synthesis report were identified (IFC 2016a, 2016b, 2019b, n.d.). The four studies assessed the financial performance of client banks and recipient firms and, in one case, job creation, rather than changes in the lives of recipients. The synthesis study also usefully identified some success factors in implementing the Banking on Women program. The studies do not, however, present a case for the effectiveness of the investments based on experimental or quasi-experimental evaluation designs, nor do they identify innovations that enhance the effectiveness of the intervention or provide insights on the development effects of the lending.6

Box 5.1. Contributing to Business Intelligence While Expanding Knowledge in International Finance Corporation Gender-Flagged Investments

The evaluation of development outcomes is often perceived as a compliance activity to provide a report to development partners who fund, but do not manage, an intervention. This perception contributes to the limited monitoring and evaluation of investment operations. For example, the lack of evaluation of small and medium enterprise financing was the single most common project flaw found in the Independent Evaluation Group’s 2014 small and medium enterprise evaluation and its 2019 small and medium enterprise synthesis report. This finding highlights the need to shift client perceptions of evaluation from being a compliance activity to a business intelligence one, while also contributing knowledge of development effectiveness. This may be supported by the International Finance Corporation’s work with some clients in the Banking on Women portfolio of $2.5 billion to examine existing data and target data collection related to income, employment, and poverty reduction to identify development outcomes.

Source: World Bank 2014, 2019d.

The findings on project measurement from the seven countries included risks that may constrain the Bank Group’s measurement and reporting on gender gaps. First, there is limited reporting on project indicators in the sample. Second, if indicators were to be measured during the latter half of the strategy, the sample implies there will be limited measurement of two of the four gaps: voice and agency, and specific human endowments at a country level. Third, both projects and CPFs have a paucity of planned measurement on voice and agency, especially related to GBV. Fourth, according to documents, there are few planned evaluations of the projects tagged and flagged for gender outside of routine reporting processes, which means there would be limited additional evidence to draw on. Together, these risks highlight a lack of attention to the supervision and monitoring and evaluation systems during implementation. This could mean that at the end of the strategy period, the Bank Group will have limited evidence on its contributions to closing gender gaps.

The World Bank and IFC both offer examples of operational adaptations that support an enhanced focus on gender gaps during implementation. In these examples, task team leaders and project leads were able to build measurement systems that supported the adaptation of projects. The combination of gender and measurement expertise, interest from the clients, and the commitment of task team leaders and project leads to understanding progress on gender gaps contributed to improved measurement. Box 5.2 describes the benefits and opportunities offered by monitoring implementation.

Box 5.2. Monitoring and Supervision Supporting the Closure of Gender Gaps

Monitoring the implementation of approaches to close gender gaps can facilitate improved quality and the development of innovations when combined with supervision processes. Project implementation reports offer useful examples that highlight opportunities.

The National Agricultural and Rural Inclusive Growth Project, Kenya (P153349), was approved in fiscal year 2017. Through project monitoring and supervision, adaptations were identified to improve the quality of the project’s work in closing gender gaps. The work of developing gender-sensitive value chains highlighted the specific targeting of women entrepreneurs, especially those with business development services, capable of enhancing project outcomes. Further, a Gender Action Learning System is planned as part of this project. The system supports changes in household decision-making and collects related data. It exemplifies an approach that uses monitoring at a community level to support adaptation and measures progress that can be aggregated to higher levels.

An International Finance Corporation advisory services project was approved in fiscal year 2016 with the aim of incubating a commercially viable and effective technical and training program for women-owned and women-led small and medium enterprises. Ongoing monitoring and four assessmentstwo impact evaluations and two learning reportssupported progress toward this aim. Monitoring and evaluation were reported to have supported the supervision of the project, leading to the refinement of the delivery models and contributing to the successful development of innovations. The aim of the project was achieved, along with increased demand from clients. In addition, the products developed are being scaled up, having been incorporated into successful fundraising for the We-Fi initiative, a $50 million fund dedicated to women’s small and medium enterprises.

Source: IFC, forthcoming b; World Bank, forthcoming.

  1. Individual staff designated to support work on gender rarely have time to work across the range of functions due to the amount of support they devote to tagging and flagging processes. In a survey of 32 staff designated to support work on gender, half (16) allocated less than 15 percent of their time to working on gender. Interviewees highlighted that staff designated to support work on gender in the World Bank prioritize tagging processes above policy and client dialogue and operational solutions. The crowding-in of resources focused on tagging leaves fewer resources for other functions, especially support for implementation supervision.
  2. According to the Group Internal Audit advisory review, the vast majority of the World Bank Gender Group’s $8.8 million fiscal year (FY)19 funding came from the World Bank budget ($6.3 million or 72 percent). The remaining $2.5 million came from trust funds. The total budget from trust funds increased from approximately 20 to 30 percent between FY18 and FY19.
  3. The expansion of coordination of the International Finance Corporation (IFC) Gender Business Group (GBG) has been supported by donor funding as well as an increasing total and proportional commitment from the IFC core budget and client fees. The total budget of the GBG roughly doubled between FY17 and FY19, from about $3.5 million to just over $7 million. This budget has proved adequate to meet current targets. The IFC core budget now pays a greater share of costs, at 41 percent in FY19 compared to 31 percent in FY17. The additional funding from IFC has decreased the GBG’s dependence on trust funding, although 55 percent of the GBG budget comes from trust funds, amounting to almost $4 million in total. The remaining 4 percent of funding is generated through client fees.
  4. Powered by Women is led by the environmental and social advisory and the GBG.
  5. Gender was introduced as a special theme in the 16th Replenishment of the International Development Association (World Bank 2010b).
  6. The indicators measured are the percentage of advisory services and investment services projects with gender flags; the percentage share of women directors nominated to International Finance Corporation (IFC) Board seats; and new billion-dollar, long-term finance commitments to financial intermediaries specifically targeting women.
  7. In 2017, IFC piloted a new, ex ante project impact assessment tool—the Anticipated Impact Measurement and Monitoring (AIMM) system. In 2018, IFC began scoring all of its investment projects for development impact using the AIMM system. In 2019, AIMM was extended as a pilot for advisory services projects. AIMM is an end-to-end support system for impact assessment, designed to drive project selection ex ante, identify lessons learned, and promote learning and accountability ex post. Under the AIMM system, each IFC investment project is given an ex ante AIMM score, which is presented in the project board paper and is then monitored while in the portfolio. Client-facing advisory projects above $1 million are also given an ex ante AIMM rating, which is included in the project implementation plan. Gender and economic inclusion are embedded in the AIMM system, and specific guidance has been developed on how projects can integrate these aspects into the AIMM assessment. Specific gender and economic inclusion indicators have been included in sector frameworks as relevant. IFC’s dedicated GBG assists project teams in articulating a project’s potential impacts when these claims are made. Claims are tracked via specific identified indicators documented in the relevant project documents and monitored in portfolios. Interviewees reported that an increased score in AIMM creates an incentive for investment projects to seek to close gender gaps.
  8. These 97 projects represent all projects tagged and flagged for gender from the seven countries in this study—Bangladesh, Côte d’Ivoire, the Arab Republic of Egypt, Kenya, Peru, Tajikistan, and Vietnam—from fiscal years 2017 to 2020, quarter 2.
  9. Six projects from fiscal year 2020, which were still in their inception phase, were excluded from this analysis.
  10. Examples of microfinance and small and medium enterprise lending can be evaluated through evidence in the work of 3ie, Innovations for Poverty Action, and Abdul Latif Jameel Poverty Action Lab. Specifically, existing methods have already been applied in linking private sector development and gender issues, such as quasi-experiments and machine learning approaches in gathering insights from loan applications (Law and Chung 2020); using World Bank data to provide evidence for public policy processes (Cox et al. 2020); and assessing the effectiveness of interventions focused on women’s empowerment (Lombardini and McCollum 2018).