Geospatial data encompass all information that is ‘geotagged’ to an exact geographical location on earth. This information can be remotely sensed from space—i.e. satellite imagery—but can also be collected from databases, surveys, project documents, and Monitoring & Evaluation (M&E) systems. The use of geospatial data on project variables has become an attractive solution to fill the void of field missions during the Covid-19 pandemic. Evaluators, however, were using geospatial data in evaluation even before travel was restricted. There is now an incredible opportunity for evaluators to use geospatial data more effectively and efficiently.
The last decade has seen rapid advances in all aspects of geospatial data, especially remote sensing data. First, satellite imagery has become more readily available, at lower (or zero) cost, and with higher quality. Terabytes of free and high-resolution raw data are created every single day. But more importantly, this raw satellite imagery is now rapidly processed into meaningful geospatial data by using machine learning. For example, raw images from the MODIS satellite are daily processed into geospatial data on land cover and forest fires that evaluators can use directly. Second, along with the revolution in big data, many data collection efforts—ranging from open-sourced platforms to household surveys—record the location of their observations. Similarly, more projects report on the geographical targeting of project investments. As a result, all of this geotagged information can be combined into one geospatial dataset. Finally, the analysis of geospatial data has become more efficient and user-friendly through open-source statistical programs.
Innovative geospatial data and software provide evaluators at the Independent Evaluation Group (and other evaluation functions) with unique tools to better address evaluation questions around the relevance and effectiveness of World Bank Group interventions. To assess the relevance of development interventions, evaluators can compare the spatial variation in a project variable with the spatial targeting of development interventions. For example, in a recent Country Program Evaluation for Mexico, IEG assessed whether investments to reduce poverty were directed towards areas with the highest poverty levels. Using regression analysis, and controlling for relevant exogenous variation, the analysis showed that World Bank support at the state level is positively correlated with the presence of the poorest 40% and is fairly independent of national public spending. To assess the effectiveness of development interventions, evaluators can use geospatial time-series to proxy changes in the project outcome indicators and construct a spatial counterfactual. A previous blog elaborated on how IEG exploits the spatial and temporal aspects of geospatial data in a robust impact assessment of World Bank projects in Mozambique, India, Ethiopia, and Madagascar.
Geospatial data also help overcome some of the methodological challenges to rigorously assess the sustainability of project impacts. Remote sensing data (e.g. satellite imagery) can provide unbiased and objective information on project outcomes on a granular level in every part of the globe. The availability of such data over time enables us to understand the evolution of particular variables over the entire life span of an intervention (and many years after the intervention). For example, IEG’s ongoing evaluation of Bank Group support to Municipal Solid Waste Management is using geospatial data to assess the sustainability of such support regarding the intended and unintended environmental and land use impact around supported landfill sites long after the respective projects have ended. Until recently, such analyses typically used to be beyond the scope (and feasibility) of a conventional (project) evaluation.
The use of geospatial data is, however, no silver bullet. Whether evaluations can apply geospatial data depends on the nature of the evaluand (e.g. the sector and type of intervention to be evaluated) as well as the analytical skills of the evaluators. Moreover, the objective and rigorous assessment of effectiveness using geospatial analysis is not sufficient on its own to assess why interventions are effective (or not). For example, it remains difficult to proxy political economy and human behavior aspects from available geospatial data. Ideally, any geospatial analysis requires some type of verification and triangulation ‘on the ground’. One of the most challenging constraints regarding the use of geospatial data in many multilateral and bilateral international development agencies (as well as some other organizations) is the disconnect between operations (which focuses on design and implementation) and evaluation. Independent evaluation functions are not directly involved in the intervention cycle (especially project design and implementation). Evaluators, therefore, have relatively little influence on the M&E frameworks of the interventions financed by their organization. A well-known consequence is that public and private investments often lack granular information on project implementation which complicates the use of geospatial data in evaluation afterwards.
Going forward, how can evaluation functions like IEG enhance their use of geospatial data? The first step is to focus on some of the low-hanging fruits. In the examples mentioned above, IEG has applied geospatial analysis to a particular set of interventions with a clear temporal and spatial nature. The analyses have been facilitated by the availability of numerous data portals with open access and ready-to-use geospatial data on a wide range of economic, environmental, and agroecological indicators. In some cases, evaluations have benefited from collaborative efforts with research colleagues. These examples have not only generated interesting and useful findings, they also provide useful lessons on the potential feasibility and desirability for conducting geospatial analysis in the framework of an evaluation. To better understand which geospatial data are useful and for which purposes, piloting new methods in the framework of different evaluation modalities should be encouraged. Investments in staff capacity development, hiring external experts, computing capacity and specialized software should be weighed against the results of these pilots.
The next step constitutes an organizational dialogue on the integration of geospatial data (collection and analysis) in the design and implementation of interventions. Evaluators can help make a stronger case for informed investments in geospatial data collection and analysis, leveraging the support from like-minded champions in research and operations departments. For example, the World Bank has launched two mobile applications (the Geo-Enabling Initiative for Monitoring and Supervision, GEMS, and the Smart Supervision App, SSA) that precisely register project locations and collect information that feeds into remote and real-time M&E systems. The GEMS initiative and the Geospatial Operations Support Team (GOST) also provide trainings and advice to build the capacity of clients and World Bank staff for remote project monitoring and supervision. This fits in the World Bank’s broader strategy to support client countries in developing the infrastructure, legal framework, and human capacity needed for the management and utilization of geospatial data. After the necessary ‘proof of concept’ experiences, a concerted organizational effort is needed to unleash the potential of geospatial data for better intervention design, implementation, and M&E.
About the Author
Evaluation Officer, IEG Sustainable Development Unit
Former Hiroyuki (Hiro) Yokoi is a former Evaluation Officer in IEG's Sustainable Development Unit.
Evaluation Advisor
Jos Vaessen (PhD, Maastricht University) joined the Independent Evaluation Group in 2016 as adviser on evaluation methods.
Comments
Thank you for sharing these…
Thank you for sharing these great examples of geospatial data collection and analysis for evaluation purposes. We have also benefited from utilizing geospatial tools, including the Collect Earth (https://collect.earth/) in evaluating the results of the support provided by the UN Food and Agriculture Organization in hard-to-reach areas and crisis-affected countries. These benefits included low-cost of data collection, opportunities for obtaining data from restricted and hard-to-reach areas, re-constructing baselines and and ability to compare data related to different stages of projects' implementation.
The results from these experiments were quite interesting and practically useful for evaluations we have undertaken. At the same time, we believe that geospatial tools could offer greater values, if these are used effectively in contributing data for context analysis, development projects' design, oversight and monitoring of projects' implementation, and ultimate evaluation or assessments of the results achieved. National governments and local communities could effectively utilize these tools for greater transparency, adaptive programme management and corrective adjustments, and enhancing accountability for results.
Just a reflection.
Kindest regards,
Serdar Bayryyev
Senior Evaluation Officer, UN FAO
Interesting blog post, many…
Interesting blog post, many thanks. I am wondering if you have any projects with an agricultural focus that you are applying RS techniques to in planning, monitoring, or evaluation? And if so, how?
Thanks for sharing these…
Thanks for sharing these reflections, which I fully agree with.
Along similar lines, we discussed advantages and intrinsic limitations of geospatial in the final Call-to-Action paper published in early 2020 following the 2019 initiative Evaluation in Hard-to-Reach Areas (available here https://europa.eu/capacity4dev/devco-ess/documents/evaluation-hard-reac…).
Now that -following the C-19 crisis- the entire world has become a Hard-to-Reach Area, the challenges to do triangulation fieldwork are more acute than ever and evaluators are called to do extra efforts to envisage safe, creative but rigorous solutions to validate their assumptions.
That’s a big challenge for the entire evaluation community, and learning from each other is paramount.
How upsetting question "Why…
How upsetting question "Why evaluators should... " THEY WOULD NOT ....
The answer is very simple: World Bank Group and other international development agency that manage hundreds of Millions of dollars DO NOT WANT TO HAVE MEANINGFUL EVALUATION TOOLS! When displayed on map their achievements for the indicated dollars injected, the results are clear - "FAILURE" persistent poverty, astonishing natural resources degradation, very limited spatial coverage for interventions, very limited number of beneficiaries, no productive investments to improve low productivity, agricultural infrastructures etc... ...
You show the case of Madagascar a very limited part of Madagascar. But already, your illustration demonstrates:
i) clearly how limited in the World Bank intervention is;
ii) then you should ask for what, for whom, and how exactly would the Bank choose the location? as the delimitation do not show forest at all and no population?
iii) further on, if one ask exactly the amount of assistance involved? and you map there the administrative delimitation, you would see that more and more the interventions of the Bank is very limited if not tiny but the fund involved is not negligible.
I was the monitoring and Evaluation officer of USAID - Madagascar Bureau of Environment and Rural Development from 1997 to 2004. The international development agencies involved in the Environment Program as USAID and the World Bank were so proud that they were working for the preservation of this " unique top-5 biodiversity hotspot" that Malagasy biodiversity is mostly found in its tropical forests and diverse forest-biomes. Experts were full of it! But if one would ask, how would you ensure that your actions have significant impacts on this "unique world treasure" and demonstrate that your impacts are worth the bucks... they would be speechless and quick to dodge the question and say, you know, "unlike Africa - with their "big flagship species" Madagascar is not endowed with these types of keystones species to preserve. Donors would mostly focus on these biodiversity rich forest corridors.
I challenged all of those experts - how could they know that their actions were on course to effectively protect these "biodiversity rich forest corridors" While I requested to have a Geographic Information Based Monitoring and Evaluation System, World Bank Group and USAID's experts refused to abide to it...
THAT WOULD KILL THEM??? !!!!? WHY???? BECAUSE EVERYBODY KNEW THAT THE MONEYS FROM THESE INSTITUTIONS WHERE MOSTLY USED TO PAY THEIR EXPERTS NOT SO MUCH TO ASSIST THE INDIGENOUS POPULATION TO ACHIEVE MEANINGFUL LIVEHOOD AND BE A PART OF CONSERVATION.
IN 2 000, I WENT TO USGS - EROS DATA CENTER TO ACQUIRE 2 SETS OF LANDSAND 7 COMPLETE BANDS AND THESE FIRE DATA THAT YOU SHOW THERE FOR MADAGASCAR TO TRACK DEFORESTATION AND FIRE USAGES. ALL OF THE SO CALLED EXPERTS CONSPIRED TO DELAY THE FOREST COVER CHANGE ANALYSES! AS THEY HAD DIFFICULTY TO MAKE THEIR CASE, THEY FINALLY DECIDED TO HAVE CONSERVATION INTERNATIONAL DO THE ANALYSES AND HAVE CONTROLE ON THE NARRATIVE!!!!!
BUT THE RESULTS WERE STUBBURNED, THEY SHOWED THAT EVEN INSIDE THESE "VALUABLE FOREST CORRIDORS - WHERE ONE HECTARE LOSS IS A TREMENDOUS LOSS FOR THE WORLD, FOREST DEGRADATION ET DEFORESTATION WERE UNSUSTAINABLE... BUT DONORS AND WORLD BANK KEPT FUNDING THE ENVIRONMENT PROGRAM!
MISERIES ON MISERIES - WHEN ASSUMING WITHOUT GIS BASED DATA THAT POVERTY WERE THE MAIN ISSUE, AND CHALLENGED TO ADDRESS THE POVERTY BUT NOT ENVIRONMENT OR BIODIVERSITY PRESERVATION... THE DECISION WAS SIMPLE - DENIGRATE THE PERSON THAT BRING ON THE CHALLENGE - RETALIATE! HOWEVER FACTS ON MAPS DID LIE. THE ENVIRONMENT PROGRAM WAS A TOTAL FAILURE... COMPARE THE MAPS IN THE PROJECT PERFORMANCE ASSESSMENT REPORT : https://ieg.worldbankgroup.org/sites/default/files/Data/reports/ppar_ma…
In addition, the multitude restructuration environmental program - 10 in total - should have alerted responsible decision-makers.
It did not! To the extent possible these decision-makers simply avoided it. As I say earlier, given the skewed powers between Malagasy professionals and World Bank or USAID decision-makers coverups were very easy.
All the horrors of illicit and illegal traffics to the point of massacre of the Madagascar precious woods - Rosewoods, Ironwoods, palissanders as well as endemic species are few manifestations of failure of the environmental program... But how could the program managers know and how could the evaluators pinpoint what worked and what failed without powerful tools such as GIS based Monitoring and Evaluation System??? But how could these tools be allowed, if they clearly pinpoint the degree of failure of World Bank and other donors' assistance?
But rest assure, this need to be an upstream battle to get implemented... The money talk to much and World Bank impacts are not only weak but disastrous (feeding more the corrupt political systems even suspiciously within the World Bank) , it would not allow the use of tools that would show that it does not have impacts!
Hear this:
Catherine A. Corson (August 23, 2016)
Corridors of Power: The Politics of Environmental Aid to Madagascar (Yale Agrarian Studies Series) Since the 1970s, the U.S. Agency for International Development (and the World Bank at the same level) has spent millions of dollars to preserve Madagascar’s rich biological diversity. Yet the island nation's habitats are still in decline. In this important ethnographic study, Catherine Corson illustrates how the effort to attract high-level political attention to conservation by isolating the environment in national parks and blaming impoverished Malagasy farmers has avoided challenging key drivers of Madagascar's deforestation.
Add new comment