Estelle Raimondo: Ensuring equal opportunities for women and girls has been a longstanding priority for the World Bank Group. Over the past decade, the share of projects focused on gender equality has increased across all sectors of the Bank and the International Finance Corporation (IFC).  

But how does the World Bank Group know if these efforts are working? Welcome to "What Have We Learned?", the evaluation podcast. I'm Estelle Raimondo, the head of Methods at the Independent Evaluation Group (IEG), and I will be your host today. 

In this episode, we will discuss how the World Bank Group measures the impact of these efforts and why this is crucial for their success. We are thrilled to welcome two outstanding guests with extensive experience in measuring and evaluating development outcomes.  

Lisandro Martin heads the Department for Outcomes at the World Bank Group and leads corporate-level results reporting through the results measurement system for the Bank's fund for the world's poorest countries, IDA, and the New World Bank Group corporate scorecards. Warm welcome, Lisandro. 

Lisandro Martin: Thank you very much, Estelle. 

Estelle Raimondo: For the second time on this podcast, we also have with us Elena Bardasi, a senior economist in IEG and the author of IEG's evaluation of the World Bank and IFC's Engagement for Gender Equality over the past 10 years. Elena leads IEG's work stream on gender equality, focusing on assessing the differentiated impacts that the World Bank Group's interventions have on women and girls. Warm welcome, Elena. 

Elena Bardasi: Thank you, Estelle. It's great to be here again. 

Estelle Raimondo: So, let's get started with some questions. Elena, let me start with you. Why is it important to track the Bank Group's impact on gender equality, and why is it more complicated than just generating sex-disaggregated data? Can't we just assume that the Bank Group's interventions benefit equally men and women, boys and girls? 

Elena Bardasi: It's not straightforward to measure gender inequalities, and we can't just assume that men and women benefit equally from our interventions. The reason is that men and women are very diverse. They have different roles, opportunities, needs, and face different constraints. Essentially, there is a power asymmetry between men and women that typically places women at a disadvantage. So, we need to keep testing our assumptions about who benefits from interventions and check it. 

Measuring outcomes requires also measuring how empowerment changes and how inequalities decrease. This may imply specific measurements like gender-based violence and empowerment, which go beyond sex-disaggregated statistics.  

Achieving gender equality is a long-term goal, and therefore it's crucial to track the outputs, processes, and outcomes that emerge from this process, adjusting our actions along the way. Measurement plays a strategic planning, adaptive management, and learning role, not just accountability. 

Estelle Raimondo: Thank you, Elena. So, it is critical, but it's complex. I'm sure Lisandro will agree with that. Can you introduce your goals at the World Bank’s Outcomes Department and how they intersect with measuring gender equality and perhaps other impacts more broadly? 

Lisandro Martin: Thank you very much, Estelle. Very happy to be here and to have this conversation with Elena. The goal of the Outcome Department is to create a strong outcome-oriented culture in the World Bank Group. We aim to do this by improving the way we track and measure our results, centered around the new World Bank Group scorecard. This scorecard has 15 outcome areas, one of which is entirely dedicated to gender, and 22 indicators. 

We also help teams and senior management not only to measure but also to deliver better results. We generate data and analytics to ensure that as teams implement their programs towards the targets, they know where they stand, the progress they are making, and can make mid-course corrections. Finally, on building capacity, we focus on generating tools and frameworks, ensuring we have the skills to collect and interpret data, both internally in the Bank and in our project implementation units. It's an ambitious program, but we're making solid first steps. 

Estelle Raimondo: Thank you. What comes out clearly from both of your interventions is that there are quite a few trade-offs that the Bank Group needs to navigate on this measurement agenda. If we go back to what the gender strategy was trying to achieve for the fiscal year 2016 to 2023, it aimed to move away from the pitfalls of gender mainstreaming which was trying to achieve some measure of gender equality, oftentimes quite superficial, for all projects. However, the evaluation highlights that it didn't fully succeed in this. 

Elena, could you tell us more about what you found? Considering that almost all World Bank projects were tagged or flagged as gender relevant, can you talk about the pitfalls of measuring widely versus measuring deeply? 

Elena Bardasi: The gender tag and the gender flag (gender tag in the World Bank and gender flag in IFC) were introduced to achieve a more targeted approach to identifying gender gaps to address. However, the gender tag was not the right tool to incentivize gender-transformative interventions and acted more as an awareness tool than a measurement tool. It quickly became a marker of success, with many regions pushing for increased targets beyond what was originally in the scorecards. The implementation of the scorecards is critical because there could be intended and unintended effects that need to be addressed. 

Part of the gender strategy was left unfinished, particularly the country engagement part. This part recognizes that individual projects alone can't address gender inequalities; these need to be diagnosed and addressed at the country level, requiring targeted effort and selective identification of priorities. Measurement hasn't moved much in that direction, but hopefully, with the new strategy recognizing the centrality of this approach for attaining development impact for gender equality, we can make more progress. 

Lisandro Martin: The ongoing reforms to the country engagement model offer an opportunity to assess cross-cutting results requiring multiple interventions better. The new model aligns with the scorecard, allowing us to unpack scorecard indicators at the country level. The gender tag has proved useful but has limitations. We are using new technologies, like artificial intelligence, to analyze gender-tagged operations and identify those with characteristics to be transformational. This helps us build indicators that track results effectively. 

In the scorecard, a new indicator on gender equality moves us away from counting projects dedicated to gender, focusing instead on measuring outcomes. For example, a project aiming to bring more women into a male-dominated technical field, like energy or water, might implement an internship program. Our scorecard measures the number of women hired from the program, not just those participating, focusing on the result of the operation. 

Elena Bardasi: Many achievements have been documented through focus groups, interviews, and field visits but weren't reflected in project documentation, which diminishes our ability to learn. We have a strong culture of quantitative indicators, but some aspects are better captured through qualitative indicators and narrative.  

Projects often measure results based on what they can individually produce, what they feel they are accountable for. There's tension in moving away from attribution to embracing contribution, as many results are achieved through partnerships and complex pathways not attributable to a single action. We need to improve in this area. 

Lisandro Martin: I agree with Elena. Gender outcomes result from multiple operations, knowledge products, and collaboration with development partners. My department is working on harmonizing indicators across MDBs, hoping for scorecards that allow us to compare numbers using the same methodologies and indicators, including gender results. 

Estelle Raimondo: Thank you. I'd like to pause for a moment and shift our focus from the more aggregate, corporate side of how we try to demonstrate the value of the Bank Group and the MDBs on this agenda, to what is actually happening on the ground. Lisandro, I appreciated your example of measuring whether women end up being employed, rather than just participating. Elena, I'd love to hear some examples of gender transformative interventions you've observed, the factors needed for these transformations, and your thoughts on whether the Bank Group can and should aim for these types of transformation at scale. 

Elena Bardasi: We've seen some interesting examples in the field. One such example is the Sahel Women Empowerment and Demographic Dividend (SWEDD), a regional project implemented in several countries in the Sahel. This project operates at multiple levels, aiming to strengthen the empowerment of women and girls. It seeks to shift gender norms by involving men, boys, traditional leaders, and religious leaders to create more awareness and acceptance. It also works at the systems level, focusing on data production, policy change, and improving sexual and reproductive health services. 

These interventions not only aim to address specific gaps in inequalities, such as facilitating women's access to services or resources, but also tackle the root causes of these inequalities. They attempt to transform gender norms, roles, and relations while redistributing power and resources.  

Such projects are complex and not every project in the Bank can achieve this level of quality. However, every country can and should conduct a thorough analysis and diagnostic of the conditions creating inequalities as a basis for prioritizing gender equality objectives and developing a programmatic gender agenda. 

Estelle Raimondo: Thank you very much. Achieving transformational impact, delivering it well, and reaching scale are all central to what you're trying to accomplish, Lisandro, with your work. Do you have any thoughts on the ambition of delivering transformational impact at scale? Is it something the Bank can realistically achieve? What role do you see your department and the scorecard playing, along with other tools for managing impact? 

Lisandro Martin: That's a great question, Estelle. I think Elena's description was about going beyond business as usual to achieve transformational results. Part of the work my team does in monitoring and supporting operations on targets is something that is very close to my heart. Is something I call meeting ensuring we meet the targets without missing the point. Missing the point means we want to see development impact across all client countries, not just reaching targets by focusing on countries with the largest economies or beneficiaries. We ensure all countries, from small island states to fragile states, are included in the plans we are developing with teams. 

These plans help identify countries where we can expand operations, moving into new areas to increase impact. They also identify actions to scale up successful ongoing operations, providing additional financing or restructuring to benefit more women. We also have programs that we can accelerate to bring benefits to women faster.  

Additionally, our team serves as secretariats of the High-Level Council on Jobs that the president has created, and as we roll out this agenda, we will need to ensure that employment also reaches women, and that we create conditions that factor in gender dynamics to create work. 

Estelle Raimondo: Let's collectively commit to not missing the point when we measure. That's a great line, Lisandro. Elena, what would you expect to see at the end of the next gender strategy in 2030 that would differ from your evaluation of the past strategy? 

Elena Bardasi: I'm glad to hear from Lisandro about the country engagement model being central to the Bank's thinking on capturing and incentivizing results. Based on our evaluation findings, this approach is crucial for establishing ownership of sustainable and scalable results. If the new strategy achieves this by 2030, it will be a great success. 

Estelle Raimondo: Great. Lisandro… 

Lisandro Martin: You mentioned the future, and there's a phrase from Borges, the Argentinian writer, that I like: "The future is what we will do." Here are three things we will do to shape the future in terms of gender measurement and outcomes. First, I hope we rethink our self-evaluation system to measure long-term results the Bank contributes to, beyond project timeframes focused on individual operations. Second, we'll work more closely across the World Bank Group to have a common vision for tracking and measuring gender results. Finally, my hope is that we invest in obtaining granular data about women and the impact of our operations on their well-being, rather than relying solely on 50% desegregation, which often leads us to hit targets but miss the point. 

Estelle Raimondo: Thank you both very much.