Thanks Caroline for the informative blog and perspective. System thinking was well embedded in those key points through taking into account of diverse perspectives and connections/links etc. In reality and in system thinking terms, how to then make those boundary choices to determine what is relevant and what is not? Perhaps an actor-centered thinking or theory of change might add some value by making explicit major actors involved, their inter-relationship and domain of interests to help identify the focus and priority? For instance, like the actor centered logic of Outcome Mapping -- differentiating various groups of project actors locating in varied project domains (sphere of control, sphere of influence and interest).

As evaluators, our focus should also span across the evolution and progress of an intervention's implementation, over a specified duration of time. It signifies that, along with the application of relevance, effectiveness, impact and sustainability as evaluation criteria, there would be a requirement of an addition of "Evolution/ progress" as one more evaluation criteria. This progress would be calculated and measured at the country level or regional level, on the basis of year - on - year increase or decrease in the implementation of the intervention under focus. It would be calculated with the utilization of the mathematical formula, as mentioned below :-

Evolution / Progress = { [ Year n's contribution - Year (n - 1)'s contribution ] / [ Year (n - 1)'s contribution ] } * 100%

This contribution would be the contribution of the nation or the measured collective contribution of the region, for that particular year (n) and the year (n - 1).

Then based on the above deduced values of Evolution/ Progress, countries as well as regions across the world would be ranked. Thus leading to the creation of Rankings based on the evaluation criteria "Evolution/ Progress" !

Dear Caroline,
I fully agree with you and would like to add that for an intervention to be relevant it must make a difference in the development problem we would like to address. To paraphrase a former colleague, all of our projects do good things but whether they have an impact is an entirely different question. Seen from that perspective, relevance must be linked to impact. In fact, your idea about the need to address the key entry points is important.

Dear IEG and Caroline- With all due respect, "After nearly 15 years of applying relevance, effectiveness, efficiency, impact, and sustainability as our foundational evaluation criteria" and the Bank still cannot tell us what project outcomes and impacts were standing or newly emerged after the Bank ended its project and programme funding?! We seem to have found that you rarely consult the very participants your President extolls (see http://valuingvoices.com/ieg-blog-series-part-ii-theory-vs-practice-at-…) and you want to shift the focus from evaluating sustained impact (done on less than 1% of all projects) rather than doing far more which is true accountability to those you serve? Help me to understand!

Interested to read the blog but can not really comment until I have - so will only comment if I can read it

I liked the vision of adding a LOT MORE to 'relevance' - I'll be interested to follow up on some of the new lines of thinking in evaluation opened up by the article - as a market system development specialist - we have been, for a number of years, been interested in identifying the nodes of intervention that have the biggest chance of influencing whole value chains or inducing systemic changes that create the biggest impact. I agree - we all need to make what we do more relevant.

I like the thoughts about "is relevance still relevant?". the paradox is: if a project had been relevant for whoever at the start and if it has been successful, it would ideally become obsolete or "irrelevant". how are we to assess that the project is still relevant after it supposedly achieved its objectives?

Thanks for sharing your thoughts on this- I have been thinking about this criterion so much, particularly lately. I am inclined to say that, as it stands now (and with usual questions used within it)- relevance is obsolete and does not significantly contribute to the evaluation. This is due to the fact that, in developmental context (even in case government policies are not there or not elaborated), all interventions are 'relevant' as there are so many needs and so many gaps. From that point of view, your suggested points are going in the right direction, though I would for start call it differently to highlight the different angle that what we actually want to assess under relevance. In most cases, even if an evaluation findings point that an intervention has diminished relevance (due to many reasons, including also efficiency of process from programming to contracting to implementation), this cannot be well reflected, due to way the questions are set. Even when evaluators are in charge of setting evaluation questions, there is a tendency to 'go with a flow' and pose the same old questions, meaning that we will get the same set of answers. This results in the fact that, for vast majority of evaluation reports, relevance is always positive. Review of ROM monitoring reports (EC monitoring tool) also shows that if not all, majority of projects are rated with A or (soft) B. This, to me, is a good indicator that something needs to change quickly or we will just continue to do 'lip service' to our clients.

1. Looking at relevance at the evaluation stage is too late. The focus should be on if and how the intervention (project, Program or whatever) was monitoring what was happening its context that should be measured.
2. Relevance is important in more in design and less in implementation and much less important in evaluation. Analysis of relevance should be done in design and included in the monitoring systems and processes.