Search

Content type
Topic:Evaluation Capacity Development
Displaying 1 - 10 of 188

Geospatial Analysis in Evaluation

Web Resource
Social infrastructure and communication technology concept. IoT (Internet of Things). Autonomous transportation. Misato junction. Credits: metamorworks/iStockphoto
Jos Vaessen, Maria Elena Pinglo, and Victor Vergara also contributed to this blog. The availability of geospatial data can be vital to better understand development issues and to ensure development efforts are directed to the places where they are most needed. Geospatial data refer to any data containing information about a specific location on the Earth's surface. This encompasses a wide Show MoreJos Vaessen, Maria Elena Pinglo, and Victor Vergara also contributed to this blog. The availability of geospatial data can be vital to better understand development issues and to ensure development efforts are directed to the places where they are most needed. Geospatial data refer to any data containing information about a specific location on the Earth's surface. This encompasses a wide variety of data types such as project activities’ coordinates, political boundaries, crop patterns, road networks, and geolocated survey indicators.  Imagery data -such as satellite imagery- have also been traditionally used for geospatial analysis. However, until recently, their use remained mostly constrained to the domain of military applications given the vast computational resources needed to store and process these data. This has drastically changed due to advances in machine learning algorithms and an increase in computational capabilities, which have made this data type more generally accessible. Satellite data are particularly relevant for geospatial analysis in that they are often publicly available at a global scale, can be used to understand a broad range of phenomena, and are available over long periods of times making them suitable for time-series analysis. Although less widespread than satellite imagery, digital photos - such as streetscape images of urban scenes - are also becoming an important data source for geospatial analysis, especially when combined with the application of Artificial Intelligence (AI) techniques, which can assign meaning to features from these images.  The Independent Evaluation Group (IEG) has been exploring the use of new techniques of geospatial analysis -including the use of satellite and digital images- to understand change in spatial phenomena of interest over time, and to help answer questions on relevance and effectiveness of development interventions.  Understanding change  Geospatial data can be used to describe how some spatial phenomena changed over a period of time, by creating a chronological series and quantifying the amount of change. This technique has many potential areas of application, such as understanding changes in weather phenomena or in deforestation patterns.  IEG conducted a study to evaluate an urban development project implemented in Bathore (Albania). The study aimed at ascertaining the extent of urban growth of upgraded neighborhoods. The analysis relied on the use of publicly available satellite images during the period 1999-2010. The team trained an algorithm to classify the images’ pixels across four classes of land cover: built-up environment, forest, water, and agricultural land. This supervised classification algorithm helped group the image data into the four categories and allowed the team to see the evolution of land cover classes in the area and to detect a clear increase in the built-up environment. (See Fig. 1.) Fig. 1. Land Use/Land Cover classification of the area of interest for the period 1999-2010   Understanding relevance IEG has been piloting an approach across its Country Program Evaluations (CPEs) to help ascertain whether the Bank (and other development partners) are targeting the areas - such as regions or provinces - where there is a greater need. The analysis relies on building a customized dataset using multiple variables with geocoded data, (such as project locations), macro variables (such as population and GDP per capita), and sector-specific variables (such as education and energy). These data are typically derived from a variety of sources, including geocoded survey data and official statistics, publicly available gridded datasets, and remote sensing data. A key advantage of this approach is that it leverages both traditional and novel data sources, which are then combined to produce granular, subnational estimates. This is critical to move beyond national averages, which can hide, in many cases, regional disparities, for example, in terms of the level of access to basic services. Understanding effectiveness As geospatial data are typically available for locations beyond the specific project boundaries, it is also possible to use these data to build a spatial counterfactual, which measures what would have happened in the absence of the intervention. This methodological design is based on the identification of both a “treatment” and a “control” area (i.e., areas that benefited from the program and those that did not). It requires sufficient relevant data on outcomes and key factors affecting the outcomes, including how the factors interact among one another, before and after the implementation of the project. IEG conducted a geospatial impact assessment on the change in urban density over time around a road improvement project in Maputo (Mozambique) as part of the Managing Urban Spatial Growth evaluation. The study used a combination of machine learning techniques and econometrics. Applying a “difference-in-differences” approach, the team assigned as “treatment area” the plots that were within a buffer distance from both sides of the road improvement project, and as “comparison” or “control” area contiguous land from the north of the treatment area that was not included in any of the project’s road improvement activities. The team used a grid cell as the unit of analysis, and data sources included project locations, satellite images, digital elevation models, road networks, and points-of-interest. The study demonstrated that the horizontal density, i.e., building outwards through new urban areas, increased over time in the project area compared to the control area.  At the same time, there were no statistically significant differences between the project and control areas in terms of changes in vertical density, i.e., building upwards and filling open spaces between existing buildings. (See Fig. 2). Fig. 2: Horizontal and vertical growth of Maputo (Mozambique) Conclusion Geospatial analysis can be instrumental towards helping identify and understand the geographical impact of interventions and directing development efforts where they are most needed. Such analyses are, however, not devoid of limitations. Geospatial data are stored in specific formats, requiring specialized knowledge and expertise to manipulate the data. Additionally, although computational capabilities have greatly increased recently, some applications—especially those based on the manipulation of large amounts of image data—remain computationally intensive and might require access to additional computing resources. Finally, it is important to note that many geospatial data are essentially proxies of more complex phenomena (e.g., poverty, environmental degradation) of interest. Therefore, geospatial analysis is not a substitute for but rather a complement to on-the-ground (qualitative) data collection and analysis, with the latter still vital for strengthening the validity of findings.   Pictured above: Social infrastructure and communication technology concept. IoT (Internet of Things). Autonomous transportation. Misato junction. Credits: metamorworks/iStockphoto

Learning from Data Innovation

Web Resource
Learning from Data Innovation
Data are vital for understanding the progress and impact of development strategies. New technologies coupled with increased computing power are creating opportunities for gathering and analyzing ever larger amounts of data from a greater range of sources. In addition, remote data collection has played an important role in getting around the restrictions put in place to prevent the spread of the Show More Data are vital for understanding the progress and impact of development strategies. New technologies coupled with increased computing power are creating opportunities for gathering and analyzing ever larger amounts of data from a greater range of sources. In addition, remote data collection has played an important role in getting around the restrictions put in place to prevent the spread of the coronavirus. But innovative use of technology began before the pandemic. How have new technologies influenced data gathering and use, and what are their implications for learning from evidence and for evaluating development? Host Brenda Barbour speaks with Jos Vaessen, IEG's methods advisor. Listen on  Spotify, Apple Podcasts, Stitcher or Google Podcasts. Related Resources: Evaluation Methods Resources World Bank Support to Reducing Child Undernutrition: An Independent Evaluation Managing Urban Spatial Growth: World Bank Support to Land Administration, Planning, and Development Why evaluators should embrace the use of geospatial data during Covid-19 (Coronavirus) and beyond Adapting evaluation designs in times of COVID-19 (coronavirus): four questions to guide decisions Conducting evaluations in times of COVID-19 (Coronavirus)

Delphi Technique: Predicting Emerging Opportunities and Challenges in Renewable Energy

Web Resource
Abstract image created by Luísa Pinheiro Ulhoa Tenorio
This paper outlines the use of the Delphi forecasting method to share, aggregate, and analyze inputs from a panel of global experts on renewable energy. This paper outlines the use of the Delphi forecasting method to share, aggregate, and analyze inputs from a panel of global experts on renewable energy.

Advanced Content Analysis: Can Artificial Intelligence Accelerate Theory-Driven Complex Program Evaluation?

Web Resource
Abstract image created by Luísa Pinheiro Ulhoa Tenorio
This paper presents the methodology and results used to assess the applicability and utility of artificial intelligence for advanced theory-based content analysis. This paper presents the methodology and results used to assess the applicability and utility of artificial intelligence for advanced theory-based content analysis.

Using Qualitative Comparative Analysis to Explore Causal Links for Scaling Up Investments in Renewable Energy

Web Resource
abstract image with varying shades of blue.
This paper illustrates how qualitative comparative analysis (QCA) was used to identify causal pathways for scaling up renewable energy to meet sustainable development and climate goals. This paper illustrates how qualitative comparative analysis (QCA) was used to identify causal pathways for scaling up renewable energy to meet sustainable development and climate goals.

Impact Evaluations and Development: NONIE Guidance on Impact Evaluation

Web Resource
IEG Impact Evaluations and Development, NONIE guidance on impact evaluations
This document discusses questions of what impact evaluation is about, when it is appropriate, and how to do it. The Network of Networks for Impact Evaluation (NONIE) was established in 2006 to foster more and better impact evaluations by its membership.NONIE uses the definition of the Organisation for Economic Co-operation and Development’s Development Assistance Committee (DAC), defining impacts Show MoreThis document discusses questions of what impact evaluation is about, when it is appropriate, and how to do it. The Network of Networks for Impact Evaluation (NONIE) was established in 2006 to foster more and better impact evaluations by its membership.NONIE uses the definition of the Organisation for Economic Co-operation and Development’s Development Assistance Committee (DAC), defining impacts as “[p]ositive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended” (OECD-DAC, 2002: 24). The impact evaluations that NONIE pursues are expected to reinforce and complement the broader evaluation work by NONIE members.

The Results Agenda Needs a Steer—What Could Be its New Course?

Web Resource
A ship cruising across the current on a literal cascade of project management reporting requirements.
Is it time to rethink Results Based Management in international development? We imagine a system that favors learning over compliance.Is it time to rethink Results Based Management in international development? We imagine a system that favors learning over compliance.

Why evaluators should embrace the use of geospatial data during Covid-19 (Coronavirus) and beyond

Web Resource
Why evaluators should embrace the use of geospatial data during Covid-19 (Coronavirus) and beyond
Geospatial data encompass all information that is ‘geotagged’ to an exact geographical location on earth. This information can be remotely sensed from space—i.e. satellite imagery—but can also be collected from databases, surveys, project documents, and Monitoring & Evaluation (M&E) systems. The use of geospatial data on project variables has become an attractive solution to fill the void Show MoreGeospatial data encompass all information that is ‘geotagged’ to an exact geographical location on earth. This information can be remotely sensed from space—i.e. satellite imagery—but can also be collected from databases, surveys, project documents, and Monitoring & Evaluation (M&E) systems. The use of geospatial data on project variables has become an attractive solution to fill the void of field missions during the Covid-19 pandemic. Evaluators, however, were using geospatial data in evaluation even before travel was restricted. There is now an incredible opportunity for evaluators to use geospatial data more effectively and efficiently. The last decade has seen rapid advances in all aspects of geospatial data, especially remote sensing data. First, satellite imagery has become more readily available, at lower (or zero) cost, and with higher quality. Terabytes of free and high-resolution raw data are created every single day. But more importantly, this raw satellite imagery is now rapidly processed into meaningful geospatial data by using machine learning. For example, raw images from the MODIS satellite are daily processed into geospatial data on land cover and forest fires that evaluators can use directly. Second, along with the revolution in big data, many data collection efforts—ranging from open-sourced platforms to household surveys—record the location of their observations. Similarly, more projects report on the geographical targeting of project investments. As a result, all of this geotagged information can be combined into one geospatial dataset. Finally, the analysis of geospatial data has become more efficient and user-friendly through open-source statistical programs.  Innovative geospatial data and software provide evaluators at the Independent Evaluation Group (and other evaluation functions) with unique tools to better address evaluation questions around the relevance and effectiveness of World Bank Group interventions. To assess the relevance of development interventions, evaluators can compare the spatial variation in a project variable with the spatial targeting of development interventions. For example, in a recent Country Program Evaluation for Mexico, IEG assessed whether investments to reduce poverty were directed towards areas with the highest poverty levels. Using regression analysis, and controlling for relevant exogenous variation, the analysis showed that World Bank support at the state level is positively correlated with the presence of the poorest 40% and is fairly independent of national public spending. To assess the effectiveness of development interventions, evaluators can use geospatial time-series to proxy changes in the project outcome indicators and construct a spatial counterfactual. A previous blog elaborated on how IEG exploits the spatial and temporal aspects of geospatial data in a robust impact assessment of World Bank projects in Mozambique, India, Ethiopia, and Madagascar. Geospatial data also help overcome some of the methodological challenges to rigorously assess the sustainability of project impacts. Remote sensing data (e.g. satellite imagery) can provide unbiased and objective information on project outcomes on a granular level in every part of the globe. The availability of such data over time enables us to understand the evolution of particular variables over the entire life span of an intervention (and many years after the intervention). For example, IEG’s ongoing evaluation of Bank Group support to Municipal Solid Waste Management is using geospatial data to assess the sustainability of such support regarding the intended and unintended environmental and land use impact around supported landfill sites long after the respective projects have ended. Until recently, such analyses typically used to be beyond the scope (and feasibility) of a conventional (project) evaluation. The use of geospatial data is, however, no silver bullet. Whether evaluations can apply geospatial data depends on the nature of the evaluand (e.g. the sector and type of intervention to be evaluated) as well as the analytical skills of the evaluators. Moreover, the objective and rigorous assessment of effectiveness using geospatial analysis is not sufficient on its own to assess why interventions are effective (or not). For example, it remains difficult to proxy political economy and human behavior aspects from available geospatial data. Ideally, any geospatial analysis requires some type of verification and triangulation ‘on the ground’. One of the most challenging constraints regarding the use of geospatial data in many multilateral and bilateral international development agencies (as well as some other organizations) is the disconnect between operations (which focuses on design and implementation) and evaluation. Independent evaluation functions are not directly involved in the intervention cycle (especially project design and implementation). Evaluators, therefore, have relatively little influence on the M&E frameworks of the interventions financed by their organization. A well-known consequence is that public and private investments often lack granular information on project implementation which complicates the use of geospatial data in evaluation afterwards. Going forward, how can evaluation functions like IEG enhance their use of geospatial data? The first step is to focus on some of the low-hanging fruits. In the examples mentioned above, IEG has applied geospatial analysis to a particular set of interventions with a clear temporal and spatial nature. The analyses have been facilitated by the availability of numerous data portals with open access and ready-to-use geospatial data on a wide range of economic, environmental, and agroecological indicators. In some cases, evaluations have benefited from collaborative efforts with research colleagues. These examples have not only generated interesting and useful findings, they also provide useful lessons on the potential feasibility and desirability for conducting geospatial analysis in the framework of an evaluation. To better understand which geospatial data are useful and for which purposes, piloting new methods in the framework of different evaluation modalities should be encouraged. Investments in staff capacity development, hiring external experts, computing capacity and specialized software should be weighed against the results of these pilots. The next step constitutes an organizational dialogue on the integration of geospatial data (collection and analysis) in the design and implementation of interventions. Evaluators can help make a stronger case for informed investments in geospatial data collection and analysis, leveraging the support from like-minded champions in research and operations departments. For example, the World Bank has launched two mobile applications (the Geo-Enabling Initiative for Monitoring and Supervision, GEMS, and the Smart Supervision App, SSA) that precisely register project locations and collect information that feeds into remote and real-time M&E systems. The GEMS initiative and the Geospatial Operations Support Team (GOST) also provide trainings and advice to build the capacity of clients and World Bank staff for remote project monitoring and supervision. This fits in the World Bank’s broader strategy to support client countries in developing the infrastructure, legal framework, and human capacity needed for the management and utilization of geospatial data. After the necessary ‘proof of concept’ experiences, a concerted organizational effort is needed to unleash the potential of geospatial data for better intervention design, implementation, and M&E.

Evaluation of International Development Interventions

Web Resource
Evaluation of International Development Interventions
This guide provides an overview of evaluation approaches and methods that have been used in the field of international development evaluation. This guide provides an overview of evaluation approaches and methods that have been used in the field of international development evaluation.

Building on what is already there: Africa launch of the Global Evaluation Initiative

Web Resource
Participants of the Virtual Africa launch of the Global Evaluation Initiative
“The monitoring and evaluation process is essential for governance and the development of evidence-based policies,” announced Mozambique Vice Minster of Finance Dr. Carla Alexandra Oreste do Rosário Fernandes Louveir at the start of a virtual conference focused on the launch of the Global Evaluation Initiative and its potential role in Africa. “Monitoring and evaluation are powerful decision- Show More“The monitoring and evaluation process is essential for governance and the development of evidence-based policies,” announced Mozambique Vice Minster of Finance Dr. Carla Alexandra Oreste do Rosário Fernandes Louveir at the start of a virtual conference focused on the launch of the Global Evaluation Initiative and its potential role in Africa. “Monitoring and evaluation are powerful decision-making tools for managers,” added the Vice Minister. The Global Evaluation Initiative (GEI) brings together a broad and inclusive coalition of governments, citizens and experts to close a global gap in monitoring and evaluation capacities. According to a recent report, more than half of all countries now have high-quality national development strategies and almost all of them are aligned with the Sustainable Development Goals (SDGs), but only one third of them have data and systems to track implementation of their policies. Responding to an urgent need A sense of urgency was palpable during the launch event, “Evidence and the Road to 2030,” which brought together government and independent evaluation experts from across Africa, as well as representatives of international organizations and donor countries. The Sustainable Development Goals were already off track as the 10-year countdown to the 2030 Agenda began this year, and the COVID-19 pandemic is derailing them further, threatening to push millions into poverty. “We need reliable evidence to guide our path to economic recovery, “ said event participant Godfrey Mashamba, South Africa’s Deputy Director General of the Department of Planning and Monitoring and Evaluation. Monitoring and evaluation (M&E) can deliver invaluable evidence on what is working and what is not. They can be a powerful compass, helping governments learn from experience and use the lessons to adjust course, scale or target public policies more effectively. There is strong demand from countries in Africa and around the world to strengthen their M&E capacities as critical for rebuilding better from the pandemic and getting back on track to the SDGs. Linking national, regional and global knowledge GEI aims to respond to the demand by building on the capacities, experiences and knowledge of local actors and matching these with coordinated support and global knowledge, according to representatives of two organizations at the heart of the  partnership who spoke during the event: Alison Evans, Director-General of Evaluation at the World Bank Group, and Oscar A. Garcia, Director of the Independent Evaluation Office of the United Nations Development Programme (UNDP). For example, Cabo Verde, a small country of 10 islands off Western Africa, has many sources of data but does not yet have the institutional capacity to process and take advantage of this information, according to panelist Gilson Pina, National Director of Planning at the Cabo Verde Ministry of Finance. GEI will help train and build the capacities of staff to use the data of a new M&E platform that his planning department is building, Pina said. “The initiative’s goals coincide with our own goals,” Pina said during his intervention at the conference. Beyond building basic capacity, GEI should serve to “support a culture of evaluation” and encourage shared progress for African countries, said panelist Abdoulaye Gounou, Head of Benin’s Office for the Evaluation of Public Policies and Analysis of Government Action. “The partnership on evaluation is a very powerful tool for institutionalizing and promoting evaluation in our countries,” Gounou said. “We are learning through our peers and we are progressing together.” The partnership on evaluation is a very powerful tool for institutionalizing and promoting evaluation in our countries. We are learning through our peers and we are progressing together. Abdoulaye Gounou, Head of Benin’s Office for the Evaluation of Public Policies and Analysis of Government Action. Independent evaluators in Africa can also play a role in expanding M&E capacities in the region and developing African approaches to evaluation, said panelist Fazeela Hoosen, co-chair of the Young and Emerging Evaluators Network of the African Evaluation Association. As the next generation of evaluators, “Young and emerging evaluators are the ground or fertile soil to embed the seeds of what we have been talking about,” she said. Watch a re-play of the Africa Launch of the Global Evaluation Initiative {"preview_thumbnail":"/sites/default/files/Data/styles/video_embed_wysiwyg_preview/public/video_thumbnails/lA9WZ9aWxAg.jpg?itok=bALL_eQR","video_url":"https://youtu.be/lA9WZ9aWxAg?t=3734","settings":{"responsive":0,"width":"854","height":"480","autoplay":0},"settings_summary":["Embedded Video (854x480)."]}   Regional and global cooperation GEI aims to collaborate with the diverse set of stakeholders involved in the development of M&E capacity, from government officials to independent evaluators, and to serve as a global platform for the curating and sharing of local and global M&E knowledge. The partnership has already been in touch with the African Evaluation Association (AfrEA), an umbrella organization for individual evaluators in countries that lack national evaluation associations, said panelist Rosetti Nabbumba of Uganda, president of AfrEA’s Board of Directors. GEI has begun talking with AfrEA about working together to compile a database of African evaluators. “I’m looking forward to a very collaborative arrangement,” with GEI, Nabbumba said. The Swedish International Development Agency (Sida) also has high expectations for GEI, with the hope that it will take the evaluation capacity agenda forward with a focus on country-led efforts according to panelist Pernilla Rafiqui, Sida’s Senior Program Manager for Capacity Development. Rafiqui described GEI as “both timely and well conceptualized,” and added, “we expect GEI to contribute toward better policies for better lives for poor and vulnerable people, and we very much look forward to making this happen.” In closing remarks, Dr. Anthony Akoto Osei, Ghana’s Minister for Monitoring and Evaluation, emphasized the urgent need in the uncertain context of COVID-19 for robust M&E systems that enhance the chances of success of government policies. “The need for and use of evidence in support of decision-making have never been as important as it is now,” Dr. Osei said. “The required improvements in M&E need to be underpinned by what is already in place. The GEI is an example of that kind of strategy, in that it seeks to build on what is already there, and to operate collaboratively and through partnerships.” Learn more about GEI Watch the re-play of the launch event in English, French, or Portuguese.