Search

Topic:Evaluation Capacity Development
Displaying 1 - 10 of 182

Why evaluators should embrace the use of geospatial data during Covid-19 (Coronavirus) and beyond

Web Resource
Why evaluators should embrace the use of geospatial data during Covid-19 (Coronavirus) and beyond
Geospatial data encompass all information that is ‘geotagged’ to an exact geographical location on earth. This information can be remotely sensed from space—i.e. satellite imagery—but can also be collected from databases, surveys, project documents, and Monitoring & Evaluation (M&E) systems. The use of geospatial data on project variables has become an attractive solution to fill the void Show MoreGeospatial data encompass all information that is ‘geotagged’ to an exact geographical location on earth. This information can be remotely sensed from space—i.e. satellite imagery—but can also be collected from databases, surveys, project documents, and Monitoring & Evaluation (M&E) systems. The use of geospatial data on project variables has become an attractive solution to fill the void of field missions during the Covid-19 pandemic. Evaluators, however, were using geospatial data in evaluation even before travel was restricted. There is now an incredible opportunity for evaluators to use geospatial data more effectively and efficiently. The last decade has seen rapid advances in all aspects of geospatial data, especially remote sensing data. First, satellite imagery has become more readily available, at lower (or zero) cost, and with higher quality. Terabytes of free and high-resolution raw data are created every single day. But more importantly, this raw satellite imagery is now rapidly processed into meaningful geospatial data by using machine learning. For example, raw images from the MODIS satellite are daily processed into geospatial data on land cover and forest fires that evaluators can use directly. Second, along with the revolution in big data, many data collection efforts—ranging from open-sourced platforms to household surveys—record the location of their observations. Similarly, more projects report on the geographical targeting of project investments. As a result, all of this geotagged information can be combined into one geospatial dataset. Finally, the analysis of geospatial data has become more efficient and user-friendly through open-source statistical programs.  Innovative geospatial data and software provide evaluators at the Independent Evaluation Group (and other evaluation functions) with unique tools to better address evaluation questions around the relevance and effectiveness of World Bank Group interventions. To assess the relevance of development interventions, evaluators can compare the spatial variation in a project variable with the spatial targeting of development interventions. For example, in a recent Country Program Evaluation for Mexico, IEG assessed whether investments to reduce poverty were directed towards areas with the highest poverty levels. Using regression analysis, and controlling for relevant exogenous variation, the analysis showed that World Bank support at the state level is positively correlated with the presence of the poorest 40% and is fairly independent of national public spending. To assess the effectiveness of development interventions, evaluators can use geospatial time-series to proxy changes in the project outcome indicators and construct a spatial counterfactual. A previous blog elaborated on how IEG exploits the spatial and temporal aspects of geospatial data in a robust impact assessment of World Bank projects in Mozambique, India, Ethiopia, and Madagascar. Geospatial data also help overcome some of the methodological challenges to rigorously assess the sustainability of project impacts. Remote sensing data (e.g. satellite imagery) can provide unbiased and objective information on project outcomes on a granular level in every part of the globe. The availability of such data over time enables us to understand the evolution of particular variables over the entire life span of an intervention (and many years after the intervention). For example, IEG’s ongoing evaluation of Bank Group support to Municipal Solid Waste Management is using geospatial data to assess the sustainability of such support regarding the intended and unintended environmental and land use impact around supported landfill sites long after the respective projects have ended. Until recently, such analyses typically used to be beyond the scope (and feasibility) of a conventional (project) evaluation. The use of geospatial data is, however, no silver bullet. Whether evaluations can apply geospatial data depends on the nature of the evaluand (e.g. the sector and type of intervention to be evaluated) as well as the analytical skills of the evaluators. Moreover, the objective and rigorous assessment of effectiveness using geospatial analysis is not sufficient on its own to assess why interventions are effective (or not). For example, it remains difficult to proxy political economy and human behavior aspects from available geospatial data. Ideally, any geospatial analysis requires some type of verification and triangulation ‘on the ground’. One of the most challenging constraints regarding the use of geospatial data in many multilateral and bilateral international development agencies (as well as some other organizations) is the disconnect between operations (which focuses on design and implementation) and evaluation. Independent evaluation functions are not directly involved in the intervention cycle (especially project design and implementation). Evaluators, therefore, have relatively little influence on the M&E frameworks of the interventions financed by their organization. A well-known consequence is that public and private investments often lack granular information on project implementation which complicates the use of geospatial data in evaluation afterwards. Going forward, how can evaluation functions like IEG enhance their use of geospatial data? The first step is to focus on some of the low-hanging fruits. In the examples mentioned above, IEG has applied geospatial analysis to a particular set of interventions with a clear temporal and spatial nature. The analyses have been facilitated by the availability of numerous data portals with open access and ready-to-use geospatial data on a wide range of economic, environmental, and agroecological indicators. In some cases, evaluations have benefited from collaborative efforts with research colleagues. These examples have not only generated interesting and useful findings, they also provide useful lessons on the potential feasibility and desirability for conducting geospatial analysis in the framework of an evaluation. To better understand which geospatial data are useful and for which purposes, piloting new methods in the framework of different evaluation modalities should be encouraged. Investments in staff capacity development, hiring external experts, computing capacity and specialized software should be weighed against the results of these pilots. The next step constitutes an organizational dialogue on the integration of geospatial data (collection and analysis) in the design and implementation of interventions. Evaluators can help make a stronger case for informed investments in geospatial data collection and analysis, leveraging the support from like-minded champions in research and operations departments. For example, the World Bank has launched two mobile applications (the Geo-Enabling Initiative for Monitoring and Supervision, GEMS, and the Smart Supervision App, SSA) that precisely register project locations and collect information that feeds into remote and real-time M&E systems. The GEMS initiative and the Geospatial Operations Support Team (GOST) also provide trainings and advice to build the capacity of clients and World Bank staff for remote project monitoring and supervision. This fits in the World Bank’s broader strategy to support client countries in developing the infrastructure, legal framework, and human capacity needed for the management and utilization of geospatial data. After the necessary ‘proof of concept’ experiences, a concerted organizational effort is needed to unleash the potential of geospatial data for better intervention design, implementation, and M&E.

Evaluation of International Development Interventions

Web Resource
Evaluation of International Development Interventions
This guide provides an overview of evaluation approaches and methods that have been used in the field of international development evaluation. This guide provides an overview of evaluation approaches and methods that have been used in the field of international development evaluation.

Building on what is already there: Africa launch of the Global Evaluation Initiative

Web Resource
Participants of the Virtual Africa launch of the Global Evaluation Initiative
“The monitoring and evaluation process is essential for governance and the development of evidence-based policies,” announced Mozambique Vice Minster of Finance Dr. Carla Alexandra Oreste do Rosário Fernandes Louveir at the start of a virtual conference focused on the launch of the Global Evaluation Initiative and its potential role in Africa. “Monitoring and evaluation are powerful decision- Show More“The monitoring and evaluation process is essential for governance and the development of evidence-based policies,” announced Mozambique Vice Minster of Finance Dr. Carla Alexandra Oreste do Rosário Fernandes Louveir at the start of a virtual conference focused on the launch of the Global Evaluation Initiative and its potential role in Africa. “Monitoring and evaluation are powerful decision-making tools for managers,” added the Vice Minister. The Global Evaluation Initiative (GEI) brings together a broad and inclusive coalition of governments, citizens and experts to close a global gap in monitoring and evaluation capacities. According to a recent report, more than half of all countries now have high-quality national development strategies and almost all of them are aligned with the Sustainable Development Goals (SDGs), but only one third of them have data and systems to track implementation of their policies. Responding to an urgent need A sense of urgency was palpable during the launch event, “Evidence and the Road to 2030,” which brought together government and independent evaluation experts from across Africa, as well as representatives of international organizations and donor countries. The Sustainable Development Goals were already off track as the 10-year countdown to the 2030 Agenda began this year, and the COVID-19 pandemic is derailing them further, threatening to push millions into poverty. “We need reliable evidence to guide our path to economic recovery, “ said event participant Godfrey Mashamba, South Africa’s Deputy Director General of the Department of Planning and Monitoring and Evaluation. Monitoring and evaluation (M&E) can deliver invaluable evidence on what is working and what is not. They can be a powerful compass, helping governments learn from experience and use the lessons to adjust course, scale or target public policies more effectively. There is strong demand from countries in Africa and around the world to strengthen their M&E capacities as critical for rebuilding better from the pandemic and getting back on track to the SDGs. Linking national, regional and global knowledge GEI aims to respond to the demand by building on the capacities, experiences and knowledge of local actors and matching these with coordinated support and global knowledge, according to representatives of two organizations at the heart of the  partnership who spoke during the event: Alison Evans, Director-General of Evaluation at the World Bank Group, and Oscar A. Garcia, Director of the Independent Evaluation Office of the United Nations Development Programme (UNDP). For example, Cabo Verde, a small country of 10 islands off Western Africa, has many sources of data but does not yet have the institutional capacity to process and take advantage of this information, according to panelist Gilson Pina, National Director of Planning at the Cabo Verde Ministry of Finance. GEI will help train and build the capacities of staff to use the data of a new M&E platform that his planning department is building, Pina said. “The initiative’s goals coincide with our own goals,” Pina said during his intervention at the conference. Beyond building basic capacity, GEI should serve to “support a culture of evaluation” and encourage shared progress for African countries, said panelist Abdoulaye Gounou, Head of Benin’s Office for the Evaluation of Public Policies and Analysis of Government Action. “The partnership on evaluation is a very powerful tool for institutionalizing and promoting evaluation in our countries,” Gounou said. “We are learning through our peers and we are progressing together.” The partnership on evaluation is a very powerful tool for institutionalizing and promoting evaluation in our countries. We are learning through our peers and we are progressing together. Abdoulaye Gounou, Head of Benin’s Office for the Evaluation of Public Policies and Analysis of Government Action. Independent evaluators in Africa can also play a role in expanding M&E capacities in the region and developing African approaches to evaluation, said panelist Fazeela Hoosen, co-chair of the Young and Emerging Evaluators Network of the African Evaluation Association. As the next generation of evaluators, “Young and emerging evaluators are the ground or fertile soil to embed the seeds of what we have been talking about,” she said. Watch a re-play of the Africa Launch of the Global Evaluation Initiative {"preview_thumbnail":"/sites/default/files/Data/styles/video_embed_wysiwyg_preview/public/video_thumbnails/lA9WZ9aWxAg.jpg?itok=bALL_eQR","video_url":"https://youtu.be/lA9WZ9aWxAg?t=3734","settings":{"responsive":0,"width":"854","height":"480","autoplay":0},"settings_summary":["Embedded Video (854x480)."]}   Regional and global cooperation GEI aims to collaborate with the diverse set of stakeholders involved in the development of M&E capacity, from government officials to independent evaluators, and to serve as a global platform for the curating and sharing of local and global M&E knowledge. The partnership has already been in touch with the African Evaluation Association (AfrEA), an umbrella organization for individual evaluators in countries that lack national evaluation associations, said panelist Rosetti Nabbumba of Uganda, president of AfrEA’s Board of Directors. GEI has begun talking with AfrEA about working together to compile a database of African evaluators. “I’m looking forward to a very collaborative arrangement,” with GEI, Nabbumba said. The Swedish International Development Agency (Sida) also has high expectations for GEI, with the hope that it will take the evaluation capacity agenda forward with a focus on country-led efforts according to panelist Pernilla Rafiqui, Sida’s Senior Program Manager for Capacity Development. Rafiqui described GEI as “both timely and well conceptualized,” and added, “we expect GEI to contribute toward better policies for better lives for poor and vulnerable people, and we very much look forward to making this happen.” In closing remarks, Dr. Anthony Akoto Osei, Ghana’s Minister for Monitoring and Evaluation, emphasized the urgent need in the uncertain context of COVID-19 for robust M&E systems that enhance the chances of success of government policies. “The need for and use of evidence in support of decision-making have never been as important as it is now,” Dr. Osei said. “The required improvements in M&E need to be underpinned by what is already in place. The GEI is an example of that kind of strategy, in that it seeks to build on what is already there, and to operate collaboratively and through partnerships.” Learn more about GEI Watch the re-play of the launch event in English, French, or Portuguese.

The deepening and broadening discussion on assessing ‘Who Benefits?’ in evaluation practice

Web Resource
High peak demand for energy in FYR Macedonia can result in power outages around the country. Here a young child does homework by lamplight in a home outside of Skopje. Photo: Tomislav Georgiev / World Bank
After attending the UNDP National Evaluation Capacities and the American Evaluation Association Conferences in 2019, I wrote a blog Are Evaluators Ready to Answer Question: Who Benefits? The answer was that increasingly evaluation practice was grappling with ‘Who Benefits’ through increasingly engaging with issues such as inequality, inequity, the differing impacts of climate change, and the Show MoreAfter attending the UNDP National Evaluation Capacities and the American Evaluation Association Conferences in 2019, I wrote a blog Are Evaluators Ready to Answer Question: Who Benefits? The answer was that increasingly evaluation practice was grappling with ‘Who Benefits’ through increasingly engaging with issues such as inequality, inequity, the differing impacts of climate change, and the role of youth. This year I attended the virtual American Evaluation Association (AEA) conference interested in how the debates on assessing benefits had developed. The theme of the AEA conference was ‘How Will You Shine Your Light’, which relates to the idea that evaluators shine their light for the purpose of “improving conditions for others to shine theirs”. Throughout, the conference focused on equity, the fair distribution of benefits. The presidential track of the conference reflected on equity through discussions on restorative practice, racism in qualitative practice, disability in evaluation, community-based foresight practices, climate change and presenting a framework for assessing community harmony. Outside of the presidential strand, presentations also focused on gender equity, equity-based policy advocacy and equity and inclusion. The emergence of equity as a major theme in evaluation practice reflects trends where evaluation gatherings embrace a diversity of perspectives, including indigenous, youth and social justice perspectives. Many of the presentations at the conference reflected critically on approaches to engage marginalized groups and called for more context specific and culturally appropriate methods. Some of the interesting resources that I would reference again as I develop evaluations were related to youth, the rubric developed to measure love, equity in data science and foresight techniques. Doing No Harm in Evaluation Harm is not a word that evaluators have engaged with deeply very often. The presenters of this Think Tank initially came together to reflect on harm in evaluation practice when organizations that were meant to protect rights but caused harm were exposed. Over a period of two years we have further understood that evaluation can cause harm through its: handling of allegations of sexual exploitation; inaccurate findings; privileging of voices; myopically focusing on donor values; careless storing of data; cultural insensitivity; and not evaluating harm. This think tank introduces these issues and seeks to identify where our principles and standards need updating to better reflect the potential for harm. The Think Tank will engage participants in a participatory reflection process aimed to develop adaptations for where our principles are found wanting. My understanding of the ‘Who Benefits’ question was especially deepened in facilitating a Think Tank entitled Doing No Harm in Evaluation that drew on a two-year action learning process. Discussing harm is the ‘shadow’ of the equity and benefits discussions, as one of our discussants highlighted. Evaluators and project managers who have the best of intentions and seek to diversify who, what and how benefits accrue can do harm – think of abuse in humanitarian settings. In debating harm, the participants went beyond concerns with ethics panels and informed consent. They questioned how harm could be caused by funding and commissioning mechanisms, a tendency to drop into contexts with limited preparation, and the answering of questions that were defined far from intended beneficiaries. Emerging research that was presented within the Think Tank made the argument that in evaluation there is inherent reductionism and carcerality, which requires changing listening practices, reciprocity with knowledge, and undertaking truth-telling on harm. The online chat during the think tank produced just less than 300 comments in 45 minutes. In Interpreting these afterward the following five themes emerged: Trade-Offs – In identifying who benefits, harm is embedded. Consequently, trade-offs between benefit and harm should be acknowledged in the evaluation processes, for example, in the (i) methods deployed; (ii) scope of the evaluation; (iii) engagement with commissioners; (iv)  management process. Reflexivity – Evaluators hold a position of privilege in defining benefit and should give attention and attend to personal growth in their own judgements, beliefs and practices as benefits can be defined in diverse ways. Trauma informed and restorative practices – Identifying who benefits may also bring an evaluation into contact with past traumas. Trauma-informed and restorative practices arose as approaches to help account and tell the truth on the harms of past practice while providing a framework to help manage trade-offs in current evaluation practice. Inclusivity – To understand who benefits requires improvements in h who evaluation practice includes, through listening skills and incorporating feedback loops. Creating space for listening will require responding to new and unexpected information on diverse perspectives. Transformative lens – Working with a transformative evaluation perspective helps to identify different benefits and harms by reinforcing attention to voice and liberation in discussions. Transformation provides potentially a useful ethical principle in reframing practice to better respond to harm. These themes and the discussions presented here reflect a small part of exchanges on equity that evaluators engaged in during the conference. Terms such as trade-offs, reciprocity, restorative, reflexivity, transformative are increasingly eminent in evaluation practice and represent a deepening discussion amongst evaluators to engage with ‘Who Benefits?’   Pictured above: High peak demand for energy in FYR Macedonia can result in power outages around the country. Here a young child does homework by lamplight in a home outside of Skopje. Photo: Tomislav Georgiev / World Bank

Evidence as a guide in uncertain times - IEG in 2020

Web Resource
Evidence as a guide in uncertain times
As the World Bank Group responded to the emerging pandemic, the Independent Evaluation Group (IEG) quickly realigned its work program to focus on the new priorities. Our capacity to adapt and provide insights relevant to the unfolding crisis was largely determined by two key factors: A reform process launched earlier in the year to increase our agility and ability to innovate, and a large body of Show MoreAs the World Bank Group responded to the emerging pandemic, the Independent Evaluation Group (IEG) quickly realigned its work program to focus on the new priorities. Our capacity to adapt and provide insights relevant to the unfolding crisis was largely determined by two key factors: A reform process launched earlier in the year to increase our agility and ability to innovate, and a large body of evidence built over years of evaluation.  We immediately set to mining this body of evidence to identify lessons from past global crises to help inform the Bank Group’s pandemic response. What began with a synthesis of lessons drawn from evaluations of Bank Group responses to Ebola, Avian Flu and other public health crises, led to a demand for evidence to inform a variety of other decisions, such as what worked best during past crises to support the private sector or reach the most vulnerable with social protection measures. These lessons are now collected in a library as a resource for the range of development actors and policymakers coping with the multiple consequences of the pandemic. A library that we add to continually as new challenges appear. While evaluation necessarily requires looking backwards to assess what has worked, the evidence gathered plays a vital role in mapping out what needs to be done to stay on track or adjust course. This has been especially challenging in the fast changing environment created by the pandemic, where restrictions on movement has placed severe limitations on the way we conventionally collect evidence and engage with stakeholders. Yet in uncertain times, more data is needed, not less, and maintaining the cycle of feedback, learning and course correction is even more crucial. As the World Bank Group and other development organizations moved quickly to launch bold programs to address the impacts of the pandemic, evaluators have had to act with equal agility to orient what they do and how they do it to the ‘new normal’.  That same agility is needed as we shift our focus to assessing the early results of the response, coupled with innovation to identify new sources of data and forms of engagement. With the pandemic hitting the most vulnerable the hardest, and threatening to drive millions into extreme poverty, it is essential that we provide our counterparts driving the pandemic response with robust evidence on what is working and for whom. {"preview_thumbnail":"/sites/default/files/Data/styles/video_embed_wysiwyg_preview/public/video_thumbnails/wPlEjy-ufDE.jpg?itok=IdjjxA09","video_url":"https://youtu.be/wPlEjy-ufDE","settings":{"responsive":0,"width":"854","height":"480","autoplay":0},"settings_summary":["Embedded Video (854x480)."]} At the start of the last financial year, before the pandemic changed everything, IEG had launched an extensive internal process to assess the alignment of our work with World Bank Group priorities. This involved a review of our products and listening carefully to counterparts, to determine whether we were delivering the right evaluative evidence at the right time to maximize learning and contribute to greater development effectiveness. Early feedback revealed a demand for more just-in-time evidence to inform decision-making alongside insights on longer-term development challenges.  We responded with reforms to increase our flexibility, developing a work program mapped to World Bank Group priorities but with enough room to respond to changing circumstances and emerging demands. We also focused on the need for innovation, in the use of new methods and technologies to provide real-time data while retaining a clear line of sight to the Bank Group’s twin goals and the Sustainable Development Goals. Even before the advent of COVID-19, IEG had begun innovating with a diverse set of new technologies to gather data from drones to geo-mapping and machine learning to increase our capacity for mining data across a broader range of data platforms and resources.  Once the pandemic struck, our internal process took on added significance and IEG has drawn on the reforms to increase our agility, diversify our approaches and face up to a new set of unprecedented development challenges.  To address these and many other challenges posed by the pandemic, IEG drew on lessons learned from evaluating in fragile and conflict-affected settings and taking on board the merits of emergent learning in times of increased uncertainty. We prepared guidelines for adapting evaluation methods in current circumstances, from remote missions to tackling ethical and data protection challenges.   Much like its internal reforms, IEG support to partner countries to build up their monitoring and evaluation capacities has also taken on added significance. With a growing number of countries basing their national development strategies on the Sustainable Development Goals, there has been increasing demand for help with developing their ability to gather evidence and use it to chart a path forward and monitor their progress. IEG has been coordinating with a broad range of countries and organizations to establish a partnership capable of meeting the scale of this demand. The pandemic has reinforced the urgency of this initiative, as the lack of robust evaluation systems leaves many countries blind, without the capacity for evidence-informed policies to cope with the health and economic shocks of the coronavirus.   Going forward, IEG will continue to build on the reforms launched earlier this year. We will continue to invest in strong relationships with counterparts even as we cannot meet in person and find ways to dive deeper into data even as collecting new information poses a challenge. We also remain committed to supporting our partner countries build their capacity for monitoring and evaluation. While we still have much to learn about performing the evaluative function in these challenging times, IEG is committed to approaching the problem with agility, flexibility, and innovation. To learn more about IEG’s FY20 programs and evaluations, read this year’s annual report.

Investing in Evaluation Capacity Development in India: Why it Matters Now More than Ever

Web Resource
Investing in Evaluation Capacity Development in India: Why it Matters Now More than Ever
Download the infographic Download the infographic

When evaluators cannot make it to the field, they can always observe from space

Web Resource
change in forest cover of the land surface in Madagascar from 1990 to 2017.
Field missions are at the very core of project evaluation. An evaluator will start with a desk-based review of available project information and prepare a methodology to assess the effectiveness of a project. However, it is only by interacting with policymakers, implementing agencies, and project beneficiaries that the evaluator gets a better understanding of the reality affecting the design and Show MoreField missions are at the very core of project evaluation. An evaluator will start with a desk-based review of available project information and prepare a methodology to assess the effectiveness of a project. However, it is only by interacting with policymakers, implementing agencies, and project beneficiaries that the evaluator gets a better understanding of the reality affecting the design and implementation of projects. This ‘reality-check’ stimulates learning and allows the evaluator to fine-tune their questions and methodology. The current COVID-19 travel restrictions pose significant challenges to field-based assessments of project effectiveness. So, what can evaluators do when they can’t get in the field? One possibility is to observe project impacts from space. Geospatial data is information collected by satellites pinpointed to an exact geographical location on earth. It is often freely available, covers several time periods, and offers a wide range of interesting indicators. Popular geospatial data are indicators of market accessibility, agroecology, and the environment. A geospatial dataset can thus be constructed by linking multiple geospatial data points with the geographical location of project activities and their surroundings. The possibility to construct a geospatial dataset for evaluating a project provides a unique opportunity for a robust quantitative assessment of project effectiveness. Beyond effectiveness, geospatial data can also provide a wealth of descriptive information that allows evaluators to better understand the local context. Even if visiting a project site is no longer possible due to COVID-19 related travel restrictions, evaluators can get a detailed picture of what is happening where in the project area by observing from space. IEG is analyzing geospatial datasets in several of its ongoing evaluations including an urban transport project in Mozambique, a sustainable land- and water-management project in Ethiopia, and a biodiversity project in Madagascar. Geospatial analysis usually involves two steps. First, geospatial data is used to precisely and accurately measure an indicator of project effectiveness. When a chronological series of geospatial data is available, changes in the indicator can be calculated using different measurements over time. The geospatial data on land use and road infrastructure are of particular interest to IEG’s evaluations. The ‘vegetation greenness’ of the land in Ethiopia is measured by looking at changes over time in the coverage of land with green vegetation. Similarly, deforestation rates in Madagascar are measured as the change in forest coverage of the land surface over time. In Mozambique and India, the density of social and economic activities is measured by the travel distance to urban amenities using roads. Second, as geospatial information is available for locations beyond the project boundaries, a proper ‘counterfactual’ can be constructed. The counterfactual illustrates a ‘with and without’ scenario - what would have happened at the project location if project activities were not implemented there. Combining the temporal and spatial variation in geospatial data provides a very robust ‘difference-in-difference’ assessment of project effectiveness. The temporal variation identifies the ‘before-and-after’ difference, and the spatial variation identifies the ‘with-and-without’ difference. The ‘difference-in-difference’ assessment of project effectiveness is applied as follows. In Mozambique and India, IEG compares changes in economic activity between urban areas that were either adjacent to a road improved by the project or adjacent to a nearby but non-improved road. Similarly, long-term changes in vegetation cover in Ethiopia are compared between land parcels in treated watersheds with similar parcels in untreated watersheds within a reasonable distance from the project site. Finally, IEG compares changes in deforestation rates between patches of forests on either side of the border of conservation areas in Madagascar. Then, these changes are compared between conservation areas supported by the World Bank and areas without project support. In each of these scenarios, the analysis informs the broader question of ‘what difference did the project make?’.      This 3D map shows the changes in the height of the built-up area in Mumbai. However, not all projects allow for a geospatial analysis of effectiveness. The availability of geospatial data to measure project indicators depends on the sector, the type of project, and the nature of activities. Projects without a specific geographic location, such as projects supporting a development policy at the national level, do not lend themselves to a geospatial analysis. But even if a quantitative geospatial analysis is possible, asking whether a project was effective might not be the most important question for the project evaluation. The more interesting evaluation questions are often those looking at the factors limiting the project’s impact. These factors are often highly contextual and linked with human behavior, which is much more difficult to measure from space. So, the quantitative geospatial analysis is an important first step to assess project effectiveness, but evaluations need to go further and understand why the project has been effective or not. But geospatial data can have an important contribution here as well. The geospatial information on contextual factors, such as the cover of the land or travel time to reach a given location, can help to identify different levels of project effectiveness and understand the role of underlying drivers in explaining the observed differences. In a follow-up blog, we will elaborate on how geospatial analysis can guide the design of a qualitative data collection method. Pictured at the top of the page: This image displays the change in forest cover of the land surface in Madagascar from 1990 to 2017. The black line represents the boundary of two Protected Areas for biodiversity conservation in Madagascar, Manongarivo and Tsaratanana. The blue dotted line is the 5 km buffer around the border of the Protected Area. Green dots are land that remained forest over time, white dots are land covered with forests, and red dots are land that were deforested during the period 1990 to 2017. IEG analyzed the share of different dots on each side of the border of Protected Areas to assess deforestation rates in and around Protected Areas.

Meet the Evaluator: Lauren Kelly

Web Resource
Meet the Evaluator: Lauren Kelly
The Independent Evaluation Group’s Lauren Kelly speaks on the role of the evaluator during the ongoing pandemic – and infodemic. The Independent Evaluation Group’s Lauren Kelly speaks on the role of the evaluator during the ongoing pandemic – and infodemic.

Building the First-Ever Partnership Focused on Addressing Global Gaps in Evaluation Capacity

Web Resource
Building the First-Ever Partnership Focused on Addressing Global Gaps in Evaluation Capacity
With the ten-year countdown to the Sustainable Development Goals (SDGs) now underway, and countries across the globe struggling to cope with the COVID-19 pandemic, the capacity to gather data to inform decisions, and to monitor and evaluate the impact of policies, is now an urgent priority. A broad coalition of governments and national and international organizations have agreed to establish the Show MoreWith the ten-year countdown to the Sustainable Development Goals (SDGs) now underway, and countries across the globe struggling to cope with the COVID-19 pandemic, the capacity to gather data to inform decisions, and to monitor and evaluate the impact of policies, is now an urgent priority. A broad coalition of governments and national and international organizations have agreed to establish the first-ever global partnership focused on addressing the worldwide gaps in monitoring and evaluation capacity. In June 2020, a range of donor countries and organizations met for the first time in a Co-Creation Workshop to discuss concrete steps towards establishing an inclusive partnership to meet the global demand from developing countries for stronger monitoring and evaluation systems and capacity. The aim of the partnership is to increase coordination for greater impact among the various national and international initiatives aimed at building evaluation capacity, and to pool resources and draw on local and global expertise and knowledge to scale up these efforts. The workshop was a three-day virtual brainstorming discussion focused on building consensus around a joint vision of the global partnership. Ahead of the workshop, a series of consultations were held with representatives from countries committed to strengthening their M&E systems and capacities, to understand the challenges they face, and how best to support their programs. {"preview_thumbnail":"/sites/default/files/Data/styles/video_embed_wysiwyg_preview/public/video_thumbnails/ftcYucMsIlE.jpg?itok=y3Fn6dDT","video_url":"https://youtu.be/ftcYucMsIlE","settings":{"responsive":0,"width":"854","height":"480","autoplay":0},"settings_summary":["Embedded Video (854x480)."]} Effective monitoring and evaluation systems are an essential ingredient for advancing the sustainable development goals as they foster accountability and evidence-based policy making. This innovative partnership will take us a step closer in addressing the worldwide demand from countries for stronger M&E systems and capacities for more inclusive and sustainable development results. Oscar A. Garcia, Director, Independent Evaluation Office, UNDP The current demand for evaluation capacity development far outstrips the resources and reach of any single institution, and the impact of the many programs launched to meet this need is diluted by a lack of coordination. Earlier this year, IEG and the UNDP’s Independent Evaluation Office (IEO) signed an agreement on closer collaboration on meeting this need. Recently, IEG also signed an agreement  with Canada’s École nationale d'administration publique (ENAP) in order to coordinate actions and pool expertise and resources towards meeting the need for stronger M&E systems and capacity in key, under-served regions of the world. The lack of robust monitoring and evaluation systems leaves many countries at a disadvantage and has become an ever more urgent development challenge in the face of the fast-moving coronavirus pandemic. Only by working together will we be able to address the global gaps in evaluation capacity, and ensure no communities or countries are left behind. Alison Evans, World Bank Vice President and IEG Director-General Along with developing a joint vision, the participants in the Co-Creation Workshop also discussed the key lines of business and activities in providing countries support on strengthening their monitoring and evaluation systems and capacities. They also discussed other important aspects of the partnership, such as its operational principles, budgetary and administrative arrangements, and its governance structure. The workshop concluded with an agreement amongst the participants on key steps and actions they will be taking in collaboration with the co-hosts of the workshop - the World Bank’s Independent Evaluation Group (IEG) and the Independent Evaluation Office (IEO) of United Nations Development Programme (UNDP)  - and other partners towards launching the partnership later this year.  Watch and hear from Wilson Braganca, the Director General of the Ministry of Planning and Finances from Sao Tome to learn more about the global demand for evaluation capacity development from countries. Read more about the current challenges in global M&E capacity and the need for joint action. Sign up to receive updates about the growing global partnership to close the gap in monitoring and evaluation capacity worldwide.

Independent Evaluation Group and École nationale d'administration publique formalize collaboration towards building a new global partnership

Web Resource
Independent Evaluation Group and École nationale d'administration publique formalize collaboration towards building a new global partnership
The World Bank's Independent Evaluation Group (IEG) and the École nationale d'administration publique (ENAP) have signed a Memorandum of Understanding (MOU) to pool resources and expertise to meet the global demand for stronger monitoring and evaluation capacity.  Working together, they will be in a much stronger position to meet this demand in key regions, including Francophone Africa, Show MoreThe World Bank's Independent Evaluation Group (IEG) and the École nationale d'administration publique (ENAP) have signed a Memorandum of Understanding (MOU) to pool resources and expertise to meet the global demand for stronger monitoring and evaluation capacity.  Working together, they will be in a much stronger position to meet this demand in key regions, including Francophone Africa, North America and the Caribbean, and the Middle East and North African regions.   The agreement comes at a time when there is a great need for evaluation capacity building around the world, which no single organization can meet on its own. Formalizing the collaboration between IEG and ENAP will enable the two institutions to work together to share knowledge and lessons learned, coordinated their efforts and expertise, and capitalize on their respective networks to advance evaluation capacity development in key areas of the world. The collaboration will help expand the delivery of ENAP's Programme International de formation en évaluation du développement (PIFED) to geographic and linguistic spaces that remain currently under-served. Watch IEG Director-General Alison Evans and ENAP Director-General Guy LaForest introduce this new collaboration for global evaluation capacity development. {"preview_thumbnail":"/sites/default/files/Data/styles/video_embed_wysiwyg_preview/public/video_thumbnails/oi7W5IK2YkQ.jpg?itok=GtdFWLJh","video_url":"https://youtu.be/oi7W5IK2YkQ","settings":{"responsive":0,"width":"854","height":"480","autoplay":0},"settings_summary":["Embedded Video (854x480)."]}  "This agreement will allow ENAP and IEG to go much farther in building and developing evaluation capacity in developing countries," said Guy Laforest, the Executive Director of ENAP. “I am pleased that our school's expertise in evaluation is now available on a wider scale, especially in this year of the 10th anniversary of the PIFED.” “IEG looks forward to strengthening its partnership with ENAP in order to generate synergies and thus extend the impact of the monitoring and evaluation support programs of our two institutions,” said Alison Evans, the Director General of IEG. “This partnership will enable us to support more governments and institutions in strengthening their systems and capacity to support data-based decision-making and results-based information, and thus accelerate their progress towards the sustainable development goals (SDGs).” As a step towards signing the MoU, ENAP hosted an event during the 2020 gLOCAL Evaluation Week, which took place from June 1 to 5, and was organized by IEG and the CLEAR Initiative. ENAP held a joint panel with the Francophone Evaluation Network of Canada and International Program for Development Evaluation Training (IPDET) on the theme "The Online Transition of Pandemic Assessment Capacity Building Practices: Challenges, Opportunities and Limits" in which hundreds of people participated. Note: This is a translated version of the original news story in French. Sign up to receive updates about the growing global partnership to close the gap in monitoring and evaluation capacity worldwide.