Add new comment

Jeff Tanner
Embrace all APPROPRIATE evaluation methods. The challenge with inclusiveness is that it can lead to a watered-down inability to make any useful conclusions: When all methods are equally valid, we are left with no way to discern between competing claims. Rather, we should endeavor to apply the right method to the right question. In doing so we must be clear about the explicit identification assumptions as well as the too-often implicit ontological and epistemological assumptions behind different methods of evaluation. Although there is no hierarchy of evaluation methods generally, there is for specific questions. So while an experimental or quasi-experimental design is probably not the right method to answer the question about how stakeholders viewed the rollout process of a particular type of intervention, it is the right method to answer the question of the causal, attributable effect of a defined intervention in raising a defined construct of welfare for a defined population at a particular place and time. The question of external validity is a valid one. As Lant points out, there are few “invariance laws” for social science. In practice, therefore, the general principle of “external validity” is not likely to be accurate—there is no such thing as general generalizability in social sciences. Moreover, the question of out of sample prediction, which is what critics usually mean by external validity, is not limited to any particular type of evaluation method; all have the same challenge. Rather, we must have a particular, defined context in mind when discussing the transferability of programmatic findings, regardless of the evaluation method that generated those findings. The advantage of systematic reviews is not that they prescribe universal policy, but that they present those interventions that tend to be robust to the many ways things can go wrong and describe the contexts in which they worked to allow policy makers to form their own judgments on the appropriateness of local application. Beyond the oft-repeated and as often agreed to bit of useless wisdom that “context matters”, the challenge for evaluation—of all stripes—moving forward will be understanding which are the important elements of context that matter. While the applicability of a finding from one intervention to another place, time, or scale is never 100%, neither is it zero. Discovering the key elements upon which transferability is likely to hinge may well rely on both social science theory and the arsenal of evaluation methods, appropriately applied.
Note: Inappropriate comments or requests will not be posted.
The content of this field is kept private and will not be shown publicly.
By submitting this form, you accept the Mollom privacy policy.