Field missions are at the very core of project evaluation. An evaluator will start with a desk-based review of available project information and prepare a methodology to assess the effectiveness of a project. However, it is only by interacting with policymakers, implementing agencies, and project beneficiaries that the evaluator gets a better understanding of the reality affecting the design and implementation of projects. This ‘reality-check’ stimulates learning and allows the evaluator to fine-tune their questions and methodology.
The current COVID-19 travel restrictions pose significant challenges to field-based assessments of project effectiveness. So, what can evaluators do when they can’t get in the field?
One possibility is to observe project impacts from space. Geospatial data is information collected by satellites pinpointed to an exact geographical location on earth. It is often freely available, covers several time periods, and offers a wide range of interesting indicators. Popular geospatial data are indicators of market accessibility, agroecology, and the environment. A geospatial dataset can thus be constructed by linking multiple geospatial data points with the geographical location of project activities and their surroundings.
The possibility to construct a geospatial dataset for evaluating a project provides a unique opportunity for a robust quantitative assessment of project effectiveness. Beyond effectiveness, geospatial data can also provide a wealth of descriptive information that allows evaluators to better understand the local context. Even if visiting a project site is no longer possible due to COVID-19 related travel restrictions, evaluators can get a detailed picture of what is happening where in the project area by observing from space.
IEG is analyzing geospatial datasets in several of its ongoing evaluations including an urban transport project in Mozambique, a sustainable land- and water-management project in Ethiopia, and a biodiversity project in Madagascar.
Geospatial analysis usually involves two steps. First, geospatial data is used to precisely and accurately measure an indicator of project effectiveness. When a chronological series of geospatial data is available, changes in the indicator can be calculated using different measurements over time. The geospatial data on land use and road infrastructure are of particular interest to IEG’s evaluations. The ‘vegetation greenness’ of the land in Ethiopia is measured by looking at changes over time in the coverage of land with green vegetation. Similarly, deforestation rates in Madagascar are measured as the change in forest coverage of the land surface over time. In Mozambique and India, the density of social and economic activities is measured by the travel distance to urban amenities using roads.
Second, as geospatial information is available for locations beyond the project boundaries, a proper ‘counterfactual’ can be constructed. The counterfactual illustrates a ‘with and without’ scenario - what would have happened at the project location if project activities were not implemented there. Combining the temporal and spatial variation in geospatial data provides a very robust ‘difference-in-difference’ assessment of project effectiveness. The temporal variation identifies the ‘before-and-after’ difference, and the spatial variation identifies the ‘with-and-without’ difference.
The ‘difference-in-difference’ assessment of project effectiveness is applied as follows. In Mozambique and India, IEG compares changes in economic activity between urban areas that were either adjacent to a road improved by the project or adjacent to a nearby but non-improved road. Similarly, long-term changes in vegetation cover in Ethiopia are compared between land parcels in treated watersheds with similar parcels in untreated watersheds within a reasonable distance from the project site. Finally, IEG compares changes in deforestation rates between patches of forests on either side of the border of conservation areas in Madagascar. Then, these changes are compared between conservation areas supported by the World Bank and areas without project support. In each of these scenarios, the analysis informs the broader question of ‘what difference did the project make?’.
However, not all projects allow for a geospatial analysis of effectiveness. The availability of geospatial data to measure project indicators depends on the sector, the type of project, and the nature of activities. Projects without a specific geographic location, such as projects supporting a development policy at the national level, do not lend themselves to a geospatial analysis.
But even if a quantitative geospatial analysis is possible, asking whether a project was effective might not be the most important question for the project evaluation. The more interesting evaluation questions are often those looking at the factors limiting the project’s impact. These factors are often highly contextual and linked with human behavior, which is much more difficult to measure from space. So, the quantitative geospatial analysis is an important first step to assess project effectiveness, but evaluations need to go further and understand why the project has been effective or not.
But geospatial data can have an important contribution here as well. The geospatial information on contextual factors, such as the cover of the land or travel time to reach a given location, can help to identify different levels of project effectiveness and understand the role of underlying drivers in explaining the observed differences. In a follow-up blog, we will elaborate on how geospatial analysis can guide the design of a qualitative data collection method.