The Influence and Power of Evaluation

I had the opportunity to participate in an exciting panel discussion at the recent Canadian Evaluation Society conference in Montreal, which focused on the influence and power of evaluation to build a better world.

So many professionals begin their careers with the hope of changing the world for the better. And, in my view, influencing change is at the core of the evaluation profession. I have written a number of times about this subject on the IEG What Works Blog. We evaluate to draw lessons from the past and apply them to the future -- a future that is hopefully better than the past. Underpinning the evaluation endeavor is the proposition that we can avoid repeating mistakes, and that we can translate examples of strong performance and results into greater success.

But, how does that happen?

That very question stimulated our debate at the CES conference. What are the factors, internal to an evaluation, that help or hinder its influence? And what are the external factors that stymie or encourage change?

Increasing Influence: Stack the Deck in Your Favor

Ever hear of "stacking the deck in your favor?"€ There are many factors that help increase the potential influence of an evaluation that are internal to the evaluation process itself:

  • Team leadership and composition in terms of technical competence, analytical and evaluation skills, political savvy and inter-personal skills;
  • Clear communication with stakeholders, as well as a strong appreciation of stakeholder interests and possible tensions;
  • Design features of the evaluation that increase its credibility among, and relevance to stakeholders, whether drawing the scope of the evaluation, using methods that ensure triangulation from different sources, or benchmarking with industry standards;
  • Process features, including stakeholder engagement, that promote greater understanding and learning during the process of the evaluation, while not adversely affecting its independence;
  • Tailored presentation, packaging, and outreach activities that ‘speak’ to the report’s audiences;
  • Quality recommendations that are clear without being prescriptive, and that have the potential to impact performance and outcomes;
  • Follow-up actions, including management action record and follow-up statements to the governing body.

Factors external to the evaluation that favor its influence include:

  • Strong stakeholder interest that increases potential influence, including openness to criticism, interest in learning from the past, or existing reform processes that are informed by evaluation findings;
  • Events external to the evaluation that support its course of action, such as funders or policy-makers deciding to wait for evaluation feedback before proceeding, or major changes in context that force a change of direction in a program that the evaluation can inform.

Hindering Influence: Don'€™t be Your Own Worst Enemy

The evaluation process, and decisions taken by the evaluation team, can generate a number of factors that may be stacked against the potential to influence change. These may include:

  • Attempting to take on issues that are too large, complex, or ill-defined;
  • Wrong timing, where decisions the evaluation was designed to influence have been made by the time results come out, or where there is a lack of political readiness to engage in a change process;
  • Errors in the evaluation itself (faulty design, process, reporting, or recommendations) that undermine its credibility and potential influence;
  • Constraints (data, time, resources) that were not addressed in the evaluation design;
  • Limitations to follow-up actions due to the absence of follow-up systems.

And, finally, a number of factors external to the evaluation can hinder its influence:

  • Strong stakeholder interest and/or tensions that undermine an evaluation (sometimes from the start) and that result in findings being questioned or undermined;
  • External events that occur during the evaluation and that impact the extent to which findings are accepted, understood and implemented;
  • Limited follow-up action on the part of the program designers and/or implementers, because they are not required to observe recommendations or because other pressures focus attention elsewhere.

There are, of course, many more factors that evaluators can manage and address to maximize the opportunity to influence change. And yet, no matter how expertly handled, there are always imponderables about what actually happened by way of follow-up and change that can actually be attributed to evaluation findings and recommendations. After all: many other factors are taken into account in the decision-making process. Evidence from evaluation is a small, but important part of the puzzle, and evidence on its influence is still limited. But, as recently noted in my blog, it’s a young profession, and we are continuing to seek answers to this and other parts of the puzzle. That is, after all, what we do!
 

Comments

Submitted by Tessie Catsambas on Tue, 06/30/2015 - 00:19

Permalink
Dear Caroline, Thank you for this summary of practical considerations for amplifying the influence of evaluation, and removing hindrances to such influence. In reviewing the factors you list on your blog, I see, once again the importance of good facilitation of the evaluation process, an appreciation of the political nature of evaluation, and underscoring the importance of a good quality evaluation. This blog focuses on the influence of evaluation, while the title of the blog is using evaluation to build a better world. I am inspired by the title, which goes beyond practical considerations for influence, and would like to add three additional considerations if we want our evaluations to be geared toward building a better world: (1) Intention: no matter the evaluation, it is important for the evaluator to have a commitment to conducting an evaluation that helps to build a better world. In EvalPartners, http://www.mymande.org/evalpartners, we support evaluation that aims to promote social justice, equity and gender equality. That makes a difference in who we include, our methods, and the issues we raise. I believe, we are all more likely to build a better world for evaluation, if that is our explicit intention to promote and be mindful of what that means in the context of the evaluation at hand. (2) Ethics: our professional communities are closely knit and interdependent whether we are evaluators, managers or funders of evaluation. Ethics, practically speaking, is about self-management and presence, possibly by talking things through with a professional who is not directly involved in the evaluation, to help see the blind spots and implicit assumptions we carry, and also to help us decide how to handle sensitive issues. We want to be honest and unbiased, and also constructive and helpful. That can be a fine line. (3) Institutional Culture: an additional contextual factor to your good list is the "culture of institutions where evaluations are conceived, planned and managed," to borrow a phrase from Dr. Maria Bustelo's plenary presentation at the UNDP-sponsored National Evaluation Capacities (NEC) conference in Sao Paulo, 2013. Her full paper and presentation can be accessed here: https://www.unteamworks.org/bitcache/acc7961d2c6b204cdfcbc6714bbb9d538bddcb90?vid=402743&nid=369568&parent_vid=396080 The institutional culture factor is particularly important for visible and influential institutions such as the World Bank, UNDP, UN Women, UNICEF, and others that have strong evaluation offices. So, from me to you: it is good to have you and your managers at the helm at IEG, and we in the evaluation community want to know more about your stewardship of a strong evaluation culture that uses evaluation to build a better world at the World Bank.

Add new comment