Estelle Raimondo: There's an old saying that a picture is worth a thousand words. So what is the value of the more than trillion pictures that are taken every year by satellites orbiting the earth, cameras on street corners? That's a $1 million dollar question.
Geospatial analysis allows us to transform these raw images into valuable data, offering a new lens through which we can understand the world. For evaluation, this offers, a new set of methods for analyzing change over time, across space, and for judging what works, what doesn't, under what circumstances.
Welcome to What Have We Learned? The evaluation podcast. I'm Estelle Raimondo, the head of Methods at the Independent Evaluation Group, IEG, and I will be your host today. In today's episode, we will explore the potential of geospatial analysis for converting pixels into evaluative insights for better development outcomes.
We will have the pleasure of talking with Anupam Anand, senior evaluation officer at the Global Environment Facilities Independent Evaluation Office. Anupam leads their work on using satellite data, geographic information systems, and other innovative methods to enhance evaluative evidence.
Welcome Anupam.
Anupam Anand: Thank you, Estelle.
Estelle Raimondo: We also have today with us Virginia Ziulu, who is a data scientist in the methods team at IEG. At IEG, Virginia leads the use of imagery analysis in our evaluation portfolio, which includes modeling urban indicators from features extracted from digital images, as well as pushing the boundaries of remote sensing use and more broadly geospatial phenomena and how we analyze them. Welcome Virginia.
Virginia Ziulu: Thank you.
Estelle Raimondo: Virginia, it would be wonderful if you could help us lay the groundwork with a pretty brief overview of these fundamental concepts. I'm talking about geographic information systems, also known as GIS, remote sensing, and computer vision, among other terms that we might need today to understand what we're talking about.
Virginia Ziulu: Yes, thank you for the question, and I think it's very important to clarify this as they are very related concepts, but they are still quite distinct. The first one, as you said, GIS, or geographic information systems, are software or systems used to store and process different layers of geospatial data.
Remote sensing, it's the science of measuring or inferring the physical properties of an object from a distance. Typically, this is done by different sensors that capture properties of electromagnetic radiation from the earth and can be carried along different instruments such as drones, aircraft, satellites, etc.
The third important concept is a computer vision. And computer vision is a field of computer science and artificial intelligence that studies and tries to get meaning and understand visual inputs such as images and videos, and for that we look at remote sensing images such as those from satellites. We also look at digital photos taken from phones or user cameras that can tell us something about the change on specific location.
Estelle Raimondo: Sounds good. Thank you so much for laying this conceptual groundwork. Now let's go to practice. I would like to turn to Anupam to ask you if you could give us some kind of vivid examples of how you've integrated geospatial analysis into your evaluation work to date.
Anupam Anand: Thanks, Estelle. It's a pleasure joining you both today. We integrate geospatial analysis across all stages of the evaluation cycle. At the design stage, it helps us refine the theory of change and gather contextual information about project location, environment, physical aspects, and the socioeconomic and demographic aspects of the intervention. We apply geospatial techniques to answer questions on multiple evaluation criteria. For instance, to assess the relevance of the interventions and whether they are being targeted in the right geography, we use multi-criteria overlay analysis. Similarly, for impact assessment, we use data from satellite remote sensing and geolocated demographic and health survey data to run quasi-experimental methods such as propensity score matching or difference-in-differences.
Estelle Raimondo: Can you give us a particular example of an evaluation that you carried out that you thought was really enhanced by using geospatial analysis? Maybe give us a bit of a flavor of what kind of data you used. I know that you are a fan of drones, for instance. Could you give us an example of how you might have used it in a recent evaluation?
Anupam Anand: Recently, there was an evaluation on sustainable forest management where we looked at the GEF impact on the areas supported by the GEF. We examined the protected areas and the project areas that were supported, and also the areas that were not supported. Geospatial analysis not only provided us with data on effectiveness and impact, but it also helped us find the right control. We cannot compare one area of intervention with another without contextual data such as urbanization, distance from the road, and geographic conditions to make an apple-to-apple comparison.
Estelle Raimondo: Thank you so much for this example. Virginia, can I turn to you? How do you see geospatial analysis fitting into the design of those particularly complex evaluations that are inherently multi-site and look at multiple outcomes? Both our institutions carry out this type of evaluation. How do they relate to other information sources? I'm interested in how you combine different sources of data to make sense of variables that are not inherently physical, such as economic and social aspects.
Virginia Ziulu: In most cases, we combine quite different data sources within the same analysis. We typically do this by building a grid that represents the area of analysis and where all the different data sources can be combined using the same unit of analysis. For instance, we use project location data, which would be points with latitude and longitude coordinates for each project site. Then we get image data from satellite images or digital photos, from which we can extract different types of information. We also use non-image-based data sources, such as traditional census and survey data when it is disaggregated at the level we're interested in. Additionally, we use feedback from beneficiaries if it includes location information. This approach is used both for relevance, such as assessing if the right areas are being supported by projects, and for effectiveness analysis. For example, in an energy efficiency evaluation, we combined LSMS survey data, nighttime lights data, and project location data to answer questions about the effectiveness of the project.
Estelle Raimondo: Anupam, I just wanted to ask you if there are other challenges, other limitations, there is no silver bullet in evaluation ever, and geospatial is not one of them either. So just curious to know what's your take on how to handle these tools and these approaches well?
Anupam Anand: When it comes to practice, you'd be surprised to know that when we wanted to conduct our first geospatial analysis on the evaluation of GEF support to combat desertification and land degradation, we could only locate 5% of the projects precisely. This precise location is fundamental to any geospatial analysis. We addressed this challenge by manual coding and relying on multiple existing data sources. However, it's still a daunting task. What we have done is turn our frustrations into findings. In multiple evaluations, we recommended that projects need to collect precise information. Because of that, it is now a requirement within the GEF to collect precise information, which opens a whole new world for the analyses we always wanted to carry out.
Apart from that, there are challenges related to capacity, financial, and human resources. Evaluation units are often not equipped with the infrastructure or human resources needed to conduct these kinds of analyses. We overcome this challenge by using open-source tools and datasets, and by forging collaborations with universities and institutions so that we are not reinventing the wheel. I am also very happy to share that in a few weeks, we will be launching an online course with a dedicated module on how to integrate geospatial thinking into evaluations. This course will be free for all.
Estelle Raimondo: That sounds amazing. Thank you, Anupam. Our audience cannot see you, but I can, and I can tell you that there were sparkles in Anupam's eyes when he said we were turning our frustration into findings and finally getting very precise geolocation for our projects. That's amazing. I would like to turn back to you, Virginia, to ask you a little bit more about how the analysis you have carried out has been informative for the evaluations and perhaps also for the operational teams in their learning process. Do you have some examples where there was a learning engagement between the teams managing operations and our team who carried out the experiments?
Virginia Ziulu: We have worked through learning engagements with the operations team, or sometimes there is a retrofitting of the findings back to the original team that worked on the project. One key finding, as Anupam mentioned, was the difficulty in identifying the specific project location site. This was a main challenge for that particular learning engagement, compounded by the fact that country boundaries had changed. The project was quite old—about 20 years ago—and the location was very small. We had to work closely with the initial team that developed the project and with consultants from the country to locate the specific area.
We also had some technical findings. This was a transport project, and we were looking at the transformation in the area where the road was built. Many specific findings in terms of land use mix and spatial transformation were transmitted back to the team to keep in mind for future operations.
Estelle Raimondo: Great. Thank you for that. Anupam, I heard you mention a bridging process between evaluators who do not have expertise in geospatial analysis and geospatial experts who might not always be conversant in evaluation. This kind of bridging and translation process is very important. I also heard you say we should avoid reinventing the wheel. Do you see a community of practice emerging as different organizations invest more in leveraging geospatial analysis? Any thoughts on how our listeners can contribute or benefit from this?
Anupam Anand: Absolutely, Estelle. It's encouraging to witness the increasing awareness and interest in geospatial methods within the evaluation community. When I joined the GEF evaluation office 10 years ago, there were hardly any peers applying these methods. Today, we see a vibrant and expanding community. In my opinion, this is driven by several factors. The first is the need for robust evidence in light of the challenges facing the world today.
There is also an opportunity due to the phenomenal flow of geospatial data that has not been fully exploited by our evaluation community. This is coupled with the computational availability through cloud-based infrastructure and the advent of data-hungry machine learning and AI algorithms needed to analyze big geospatial data. These resources have been democratized and are relatively cheaper compared to setting up your own infrastructure in-house.
Listeners from this podcast or the evaluation community can contribute to and benefit from this growing community by engaging in interdisciplinary collaborations. They can help bridge the gap between evaluation and spatial science. For instance, evaluators can frame the evaluation questions and indicators with the help of subject matter experts, while geospatial scientists can choose the appropriate geospatial methods. These multidisciplinary collaborations provide ample opportunities for individuals from various fields to contribute and benefit from shared knowledge and insights.
Estelle Raimondo: Thank you very much. Looking ahead, Virginia, what do you see as the potential future applications that would be a great fit for geospatial analysis in evaluation practice? What excites you the most, and perhaps what concerns you that we need to keep an eye on as we grow this field?
Virginia Ziulu: First, what excites me is the increased availability of geographically disaggregated data that can be used for evaluations. This includes not only remote sensing data but also digital photos, which can be very useful for granular analysis of urban spaces, for example. There's also an increase in higher resolution imagery available that allows for much more nuanced analysis. This is important because, in evaluations, we sometimes need to see what happened in a very specific location where an intervention took place, rather than looking at an entire country.
In terms of areas of concern, while images are useful, they can also be misused. For instance, you can use image data to identify beneficiaries for development interventions, but it can also be used to identify defectors of an authoritarian regime. There are always concerns about how images are used. Ethical considerations are crucial, especially as the resolution of images increases, which can violate the privacy of individuals by providing specific geolocations of individuals or households.
Another concern is the verification of the validity of images. While official providers like NASA or the European Space Agency offer reliable satellite images, there is a growing issue with deep fake satellite images and digital photos. This isn't a huge problem for geospatial analysis yet, but it is becoming more of an issue and an active area of research.
Estelle Raimondo: Absolutely. Anupam, are there any promises or perils that Virginia hasn't mentioned that we should consider?
Anupam Anand: I think it's a very exciting time for anyone working with data. The advent of generative AI has opened many possibilities. In geospatial analysis, a lot of time is spent on cleaning, curating, and collating data. These processes are becoming much easier, allowing us to focus on the actual analysis. I'm really excited about the possibilities of what we call geo AI. This is a new area for us to explore and look forward to.
Estelle Raimondo: That sounds very exciting, and we should do this together. Absolutely. We have reached the end of the episode. I would like to thank you, Anupam and Virginia, for sharing your expertise, your know-how, and your warnings when it comes to using geospatial analysis for evaluative insights. I invite our listeners to visit our website for additional relevant resources from the GEF and from IEG. This has been "What Have We Learned?" The evaluation podcast. Please stay tuned for the next episode.
