The Influence and Power of Evaluation
I had the opportunity to participate in an exciting panel discussion at the recent Canadian Evaluation Society conference in Montreal, which focused on the influence and power of evaluation to build a better world.
So many professionals begin their careers with the hope of changing the world for the better. And, in my view, influencing change is at the core of the evaluation profession. I have written a number of times about this subject on the IEG What Works Blog. We evaluate to draw lessons from the past and apply them to the future -- a future that is hopefully better than the past. Underpinning the evaluation endeavor is the proposition that we can avoid repeating mistakes, and that we can translate examples of strong performance and results into greater success.
But, how does that happen?
That very question stimulated our debate at the CES conference. What are the factors, internal to an evaluation, that help or hinder its influence? And what are the external factors that stymie or encourage change?
Increasing Influence: Stack the Deck in Your Favor
Ever hear of "stacking the deck in your favor?" There are many factors that help increase the potential influence of an evaluation that are internal to the evaluation process itself:
- Team leadership and composition in terms of technical competence, analytical and evaluation skills, political savvy and inter-personal skills;
- Clear communication with stakeholders, as well as a strong appreciation of stakeholder interests and possible tensions;
- Design features of the evaluation that increase its credibility among, and relevance to stakeholders, whether drawing the scope of the evaluation, using methods that ensure triangulation from different sources, or benchmarking with industry standards;
- Process features, including stakeholder engagement, that promote greater understanding and learning during the process of the evaluation, while not adversely affecting its independence;
- Tailored presentation, packaging, and outreach activities that âspeakâ to the reportâs audiences;
- Quality recommendations that are clear without being prescriptive, and that have the potential to impact performance and outcomes;
- Follow-up actions, including management action record and follow-up statements to the governing body.
Factors external to the evaluation that favor its influence include:
- Strong stakeholder interest that increases potential influence, including openness to criticism, interest in learning from the past, or existing reform processes that are informed by evaluation findings;
- Events external to the evaluation that support its course of action, such as funders or policy-makers deciding to wait for evaluation feedback before proceeding, or major changes in context that force a change of direction in a program that the evaluation can inform.
Hindering Influence: Don't be Your Own Worst Enemy
The evaluation process, and decisions taken by the evaluation team, can generate a number of factors that may be stacked against the potential to influence change. These may include:
- Attempting to take on issues that are too large, complex, or ill-defined;
- Wrong timing, where decisions the evaluation was designed to influence have been made by the time results come out, or where there is a lack of political readiness to engage in a change process;
- Errors in the evaluation itself (faulty design, process, reporting, or recommendations) that undermine its credibility and potential influence;
- Constraints (data, time, resources) that were not addressed in the evaluation design;
- Limitations to follow-up actions due to the absence of follow-up systems.
And, finally, a number of factors external to the evaluation can hinder its influence:
- Strong stakeholder interest and/or tensions that undermine an evaluation (sometimes from the start) and that result in findings being questioned or undermined;
- External events that occur during the evaluation and that impact the extent to which findings are accepted, understood and implemented;
- Limited follow-up action on the part of the program designers and/or implementers, because they are not required to observe recommendations or because other pressures focus attention elsewhere.
There are, of course, many more factors that evaluators can manage and address to maximize the opportunity to influence change. And yet, no matter how expertly handled, there are always imponderables about what actually happened by way of follow-up and change that can actually be attributed to evaluation findings and recommendations. After all: many other factors are taken into account in the decision-making process. Evidence from evaluation is a small, but important part of the puzzle, and evidence on its influence is still limited. But, as recently noted in my blog, itâs a young profession, and we are continuing to seek answers to this and other parts of the puzzle. That is, after all, what we do!
Comments
Add new comment