As with development practitioners, evaluators frequently ask themselves: are we making a difference and if we are, in what way? 

In my more than 25 years of practice I’ve found that these questions can be addressed by embedding influence throughout the process: from choosing what to evaluate, how to design and undertake the evaluation, to outreach and follow-up.

Making strategic choices

What to evaluate when and why? More often than not the response is that it’s what the donor wants.  Fair enough, all donors require accountability, as they themselves must report back to their tax payers.

My own sense is that as evaluators we have to bring together two dimensions. The first is “coverage” that allows us to give a verdict on the health of a portfolio or an institution. The power of these evaluations, comprised of evaluations of a representative sample of individual interventions, derives from its aggregation of trend analysis, patterns, and overall assessment. Our Results and Performance Report is a powerful example of this type of work. The strategic choice is to maintain the individual evaluations at such a level that aggregation is possible.

The other dimension is that of strategic choice, finding those game changer subjects where the evaluation can make a contribution to solving a larger problem. These are sometimes demanded by stakeholders, but many times these evaluations touch on critical issues that at least some stakeholders would rather leave uncovered. The notion of “readiness for evaluation findings” is often thought to be essential if an evaluation is to stick. That may be true, but our efforts might also be better spent on a harder topic.

The difficulty is in identifying those topics or issues. We did that in my previous job and the evaluations were finished just as the institution was taking a hard look at the changes in strategic direction it had taken two years earlier. This is more complicated in an institution such as the World Bank Group, given the many dimensions it addresses with its services. In IEG, we are using our results framework to make those strategic choices.

How evaluations get done

The design and process of an evaluation also contribute to its influence.  The evaluation team itself must be credible, as must the process and interactions with stakeholders, and the data collection and analysis.  An evaluation is unlikely to be influential is it’s not credible.

Methods and evaluation questions are also integral parts of creating credibility and trust in the evaluation. Transparency around what questions are being researched, what methods and data will be used, and which stakeholders will be consulted will influence the quality of the evaluation and receptivity of stakeholders to its messages.

An influential evaluation also requires engagement with stakeholders. Independence should not be confused with isolation which can result in a poorer understanding of the intervention being evaluated.

Outreach and follow-through

Finally, there is the importance of outreach and follow-through.  Being part of the World Bank Group, many look to IEG for our findings, whether it is because of the central role the institutions play in the development debate, or because of the span of issues we have covered.

And, we are also fortunate to have the resources to fund activities that range from internal launches aimed at Bank Group staff, to events at the Bank’s Annual and Spring Meetings, to participating in international, regional and national conferences. We develop learning products and integrate lessons from our evaluations into training courses.

But it’s not just about quantity. Making a difference involves tailoring the right product with the right engagement process.   Experts on a particular issue are more likely to engage with the knowledge gained from an IEG evaluation while fellow evaluators may be more concerned with methods and approaches. Likewise, getting attention at a senior level is important for policy buy-in, but engagement at an operational level is essential to making a difference on the ground.

Institutional Factors

The organizational culture of an institution also plays an important role. Our ongoing evaluation of Learning and Results in World Bank Group Lending will shed light on institutional factors that enhance or hinder learning. As we did with the first phase of this evaluation, we’ll be sharing our knowledge through various channels, so please stay tuned.

Are these experiences proof that evaluation is making a difference? We track where and how our evaluations are referenced and used, follow their implementation through the Management Action Record, get feedback through our client survey and have done follow-up studies on select evaluations. Still, we wanted more evidence of influence. 

We have now included indicators in our results framework that will allow us to plan more deliberately for the influence we want to exercise and to measure in the future what we have achieved.