The Rigor of Case-Based Causal Analysis
Introduction
Over the past decade, the debate on the rigor of various approaches to impact evaluation and causal analysis has made significant strides. There is increasing consensus that questions must drive the methodology for this type of evaluation and not the reverse (Ravallion 2020), that some types of causal questions about interventions that are being evaluated simply do not lend themselves to (quasi-)experimental designs (Stern et al. 2012), and that various methodologies have different comparative advantages for answering various types of causal questions (AFD 2022; Befani 2012; Quadrant Conseil 2017). Epistemologically, some common ground is emerging regarding the redefinition of standards of rigor (for example, Jimenez et al. 2018; Johnson and Rasulova 2017), moving away from the idea that any one methodology is the gold standard for all quality criteria for research.
In another institutional movement, clearinghouses created to promote (quasi-)experimental impact evaluations are now exploring avenues for incorporating other approaches (Dixon and Bamberger 2022). The number of articles on how to integrate qualitative methods into impact assessments to complement quantitative methods has increased dramatically over the past few years. However, the level of rigor with which quantitatively driven impact evaluations incorporate qualitative approaches remains weak, as a recently conducted review of impact evaluations shows (Jimenez et al. 2018).
Recognition of the need to expand the range of evaluation methods used for causal analysis is also increasing (Stern et al. 2012; Jimenez et al. 2018). As a community of practice, evaluators have started to experiment with various methods of causal analysis, borrowing from other disciplines of the social sciences and adapting to real-world evaluation constraints (Schmitt 2020). The literature on applying alternative approaches to impact assessments has thus flourished, predominantly regarding combining theory-based approaches that use case studies as their primary empirical material, such as contribution analysis (for example, Delahais and Toulemonde 2012; Kane et al. 2021; Ton et al. 2019); realist evaluation approaches (Kazi 2003); process tracing (Befani 2021; Raimondo 2020; Rothgang and Lageman 2021); and qualitative comparative analysis (Befani 2016; Hanckel et al. 2021). Yet this literature remains emergent, and the applicability, quality, and usefulness of these various approaches require more testing.
In addition, a few myths and misunderstandings linger, despite several attempts at busting them (Flyvbjerg 2006; Widner, Woolcock, and Ortega Nieto 2022), and hinder wider adoption of case-based approaches to causal analysis in evaluation practice. The first of these relates to the inferential power of case-based approaches to such analysis (Cartwright 2022). There is a common misunderstanding that causal claims can be built only on approaches involving analysis of large numbers of observations using counterfactual thinking. The second has to do with the generalizability (external validity) of causal claims built through case-based methods. The misconception here is that evidence generated through analysis of cases is necessarily anecdotal and case specific and cannot be transferred to other cases. Taken together, these two myths fuel the argument that case-based approaches to causal analysis fail to generate findings and evidence that are useful for informing policies and thus lack value in practice.
This paper busts these myths by demonstrating that (i) the application of case-based and theory-based methods can lead to robust causal inferences and explanations and hence fill important knowledge gaps on the impact of development interventions, and (ii) it is possible to generalize from case studies, and in so doing, to generate practical and useful information on the inner workings of complex interventions and the conditions under which interventions are more or less successful.
The demonstration presented here is based on a causal analysis of the World Bank’s support to carbon finance through the development of Emission Reduction Purchase Agreements between 1999 and 2012. Carbon finance mechanisms enable high-polluting countries to offset their carbon emissions; the World Bank has supported increasing use of these mechanisms through its support for projects that introduce new technologies in developing countries, such as renewable energy projects, reforestation projects, or waste capture projects, generating reductions in greenhouse gas emissions that high-polluting countries can in turn buy as part of their carbon-offsetting efforts. An Emission Reduction Purchase Agreement is a contractual vehicle through which such a system can work. The World Bank plays the role of trustee of a carbon fund, agreeing to pay for the purchase of a certain quantity of an asset (emission reduction); payment takes place on delivery. The evaluation question that motivated the causal analysis undertaken in this paper is, How effective have the main World Bank Group carbon finance interventions been in (i) catalyzing and developing carbon markets and leveraging private investments, (ii) reducing greenhouse gas emissions, and (iii) generating demonstration effects for technologies and carbon finance?
The demonstration is organized in three sections. The first section lays out the methodology used in the causal analysis, emphasizing principles in the design of the analysis that address causal inference and generalizability. The second section shows how within-case causal analysis was conducted; exposition of the application of cross-case causal analysis follows. The third section shows how the findings can be illustrated and integrated into other analyses. The paper concludes with a discussion on the applicability of case-based approaches, their relative strengths, and their limitations.