Rethinking Evaluation - Assessing Design Quality
Will asking if an intervention's "intentions were right" be enough as we look toward the future?
Intervention design must consider how trade-offs were weighed and influenced
Intervention design must outline how to track and quickly respond to changing contexts
Intervention design must explore effects on consumption levels according to SDG12
An assessment of the design quality of an intervention asks whether the “intentions were right."...But, will this be enough as we prepare for the future?
Evaluators in international development typically assess the design quality of an intervention. The reason is simple: the intervention design provides the yardstick to determine success and failure. Achievements are measured by comparing what was planned with what has actually been achieved. In addition, we find that “quality at entry” (in other words: the quality of intervention design) is a good predicator of the intervention’s outcomes.
An assessment of the design quality of an intervention asks whether the “intentions were right”. Questions we evaluators use to determine that include whether:
But, will this be enough as we prepare for the future?
Previous blogs in this Rethinking Evaluation series discussed issues that need to be reflected upon in both intervention design and its evaluation.
For example, in an earlier blog on effectiveness, we argued that a different approach to managing complexity might lead to a different way to define objectives; one where it matters to understand the web of interrelated factors to identify entry points that will amplify possible impacts and be cognizant of what is until now unintended consequences. Under these circumstances, is it enough to ask whether objectives are realistic?
Or take the Relevance of Relevance blog where I argue that the simple question as to whether something is relevant in a complex network raises additional questions about approaches that have depended on a rather linear interpretation of reality. Logical models and results chains, if actually used in all sincerity, have far more often than not been simplified and linear. Question 2 above is clearly aligned with that questions of internal logic.
There are additional challenges that intervention design needs to take into account in light of the Sustainable Development Goals (SDGs).
For instance, tensions can and will arise especially between goals that require tough trade-offs; a challenge that is embedded in the SDGs that want a better life for all, but in environmentally sustainable ways. In an ideal scenario, the tensions between goals will stimulate innovation and lead to better solutions. Say, lowering of costs of alternative sources of energy to ensure we can meet goals to give equal access to electricity to all without, however, further depleting the earth’s natural resources. But, these ideals might be hard to get by. Evaluating whether and how trade-offs were weighed and influenced ultimate intervention design will add a much better understanding whether the right decisions were made.
And then there is the need to evaluate whether
These factors (and more) need to be reflected in design quality and its assessment, whether through internal quality assurance processes and evaluation. If we do not start now, necessary evidence will not be generated in time to learn from experience and make course-corrections as they are needed.