DAC

Consulting on the “Big 5” Evaluation Criteria - What got us here?

mpadrino — Tue, 10 Jul 2018 16:06:32 +0000

Consulting on the “Big 5” Evaluation Criteria - What got us here? mpadrino Tue, 07/10/2018 - 12:06

Add new comment

Rethinking Evaluation- Tracing the Origins of the DAC Evaluation Criteria

kmilhollin — Tue, 27 Mar 2018 15:15:10 +0000

Rethinking Evaluation- Tracing the Origins of the DAC Evaluation Criteria kmilhollin Tue, 03/27/2018 - 11:15

Add new comment

Rethinking Evaluation - Where to Next?

kmilhollin — Tue, 05 Dec 2017 16:58:51 +0000

Rethinking Evaluation - Where to Next? kmilhollin Tue, 12/05/2017 - 11:58

Comments

Dear iIEG- thank you so much…

Dear iIEG- thank you so much for this discussion. It is wonderful that an update of the DAC criteria is being contemplated and possibly even undertaken. I would encourage there to be a deeper look at sustainability definitions, expectations, and methods, especially in light of evaluating the SDGs and post project impacts. I look forward to fora that are open to developing country input especially valuing voices of our participants and partners in shaping such evaluation criteria.

Dear Caroline, thank you…

Dear Caroline, thank you very much for this blog.
This summer I was facilitating consultations on the Sri Lanka Equity-Focused and Gender-Responsive Evaluation Guidelines. And we had a very interesting discussion about DAC criteria. The consensus that emerged from this discussion was that DAC criteria are still relevant, but they should be adapted by infusing into them the file pillars of the SDGs Agenda: people, planet, peace, prosperity and partinership.
So we came up with revised definitions (applicabe in Sri Lanka) which are as follows:

Relevance:
Extent to which the intended and actual outcomes of an intervention are consistent with:
• the needs of all sub-groups of beneficiaries, including those most vulnerable and prone to marginalization;
• Sri Lanka national SDGs related to all five pillars of the 2030 Agenda (people, planet, prosperity, peace and partnership);
• Sri Lanka national policies;
• International human rights and gender equality treaties and conventions ratified by Sri Lanka.

Effectiveness:
Extent to which an intervention:
• has achieved similar levels of intended outcomes for all sub-groups of beneficiaries, including those most vulnerable and prone to marginalization;
• has ensured that none of the target beneficiaries was left behind.

Efficiency:
Extent to which intervention has ensured that all resources/inputs (funds, expertise, time, etc.) were allocated and used economically and in timely manner to achieve similar levels of intended outcomes for all sub-groups of beneficiaries, including those most vulnerable and prone to marginalization.

Sustainability:
Extent to which the benefits from an intervention for all sub-groups of beneficiaries, and especially for those most vulnerable and prone to marginalization, are likely to continue after all intervention activities have been completed.

Impact:
Extent to which an intervention has contributed to or constrained the progress towards Sri Lanka national SDGs related to all five pillars of the 2030 Agenda (people, planet, prosperity, peace and partnership).

Evaluation of synergies,…

Evaluation of synergies, based on Leopold matrix, translated into an input-output matrix, then reworked into a correlation matrix. The procedure developed by Slovenian Evaluation Society (2006-2017). Our paper: "Apples and Oranges: Synthesis
without a common denominator", 1/7(2014), https://www.dlib.si/details/URN:NBN:SI:DOC-LEB7LI9O/?query=%27contribut…. New version of the text available only on demand (bradej at gmaildotcom).

Thank you for sharing your…

Thank you for sharing your key considerations.
One aspect that could be considered for the next refined version of evaluation criteria is the "outside world". I mean that this material can be (and should be more) used by the community that is not in international development and not directly in evaluation. Keeping criteria as simple and clear as possible will allow you to share them with non-international project managers (who are following PMI or PRINCE2 guidance, not DAC) to encourage them to have more assessment as part of a normal process (in the planning, executing or closing phases).
All this simplification/clarification of evaluation criteria should be communicated to encourage more communities/industries to use them (at the very least in planning their work, and ideally in assessing it). In the end, this could at least add some integrity in their approach.
Please use the opportunity of this renewed/refined version of the criteria to communicate them to non-specialists.
Wishing you success with the next steps.

Dear IEG and Ms. Heider -…

Dear IEG and Ms. Heider - Regarding the DAC criteria revisions, I would be delighted if as an industry we re-examined and re-prioritized 'sustainability'.
How I long to have substantive data on #WhatWorks And #ForHowLong :)!
If you look at the results here on post project sustained impact evaluation: http://valuingvoices.com/cant-wait-to-learn-from-post-project-sustainab… and here on how hard it is to find methodologically clear results post project (Note: finding only 1 World Bank substantive evaluation in your PPARs that asked participants what still stood after closeout, much less lessons learned): http://valuingvoices.com/building-the-evidence-base-for-post-project-ev…. Otherwise it all seems like beautiful #WhatWorks words without substance to our participants and partners who dream of sustained impacts long after we all leave. Thoughts? Jindra

Add new comment

Conversations: the Future of Development Evaluation

kmilhollin — Thu, 22 Jun 2017 19:28:01 +0000

Comments

Thanks for sharing such a…

Thanks for sharing such a great conversation! I am not a very pessimistic person, but I have got a huge doubt on the effectiveness of development cooperation. I think before we think about the effectiness of DAC evaluation framework, I am convinced that we need to rethink the whole development cooperation delivery mechanism. This is because at least during my age, I have not seen any country that has been transformed as result of development cooperation. Development cooperation however has become part of the problem particularly in creating vicious circle of dependency, endemic corruption and a sophisticated beaurocracy that sucks the money of hard work tax payers. Many of our problems are entirely political and I think we need to judge political and social commitments before we think of engaging in development cooperation with any country!

Over some 45 years with the…

Over some 45 years with the Bank, and I forget how many with IEG, the main change I have seen in projects is a shift from top-down simple planned projects, setting objectives and ensuring that activities are designed to meet them efficiently, to messier projects which attempted to follow a more organic approach. Community development projects are the obvious case. The need for such a shift was best summed up in a book called Redesigning Rural Development written long ago in 1982 by Johnston and Clark. They believed that rural development projects should be handled more like an Eskimo/Inuit whittling a piece of bone. "He carves a little not quite knowing what he is making, exploring the bone for its texture, faults and potential. He pauses, examines, carves a bit more ... Finally, a smile of recognition: 'Hello seal, I wondered if it might be you'. Problem posed and resolved all in the same process ..." This was a fine concept but how to make it work? And how would one evaluate?
I see no merit in abandoning the three main criteria of relevance, efficacy and efficiency. But the way Bank operations plan and implement against those criteria, and the way we evaluate them, needs to accommodate such iterative, interactive, approaches and be more generally flexible. But this is easier set out in such platitudes than turned into a manageable practice. One cannot have criteria that are so malleable through alternative interpretations that nobody can be held accountable.
We also need to focus on the highest priorities in evaluation. I my view, the biggest weakness in evaluation in all agencies is, and always has been, the tricky process of turning the evidence into useful written lessons. And a part of that problem is that we have never developed any sets of evolving lessons for different sectors that might be built upon with the findings of each new evaluated project and used as a measuring stick. We rarely, if ever, say, "This is what was found in this project case and it is similar to projects x, y and z and in countries of types a and b under circumstances 1, 2 and 3." In other words, we have no constructed learning pyramids, just a flat expanse of plain onto which we randomly throw the odd interesting rock or two.

I was involved in several…

I was involved in several recent exercises to review the OECD-DAC criteria and their applicability to economic and social development endeavors. These adaptations were timely and needed. Note that I use the word "endeavors" to encompass not only the donor-financed "traditional projects" for which the OECD-DAC criteria were originally designed, but also longer term humanitarian efforts, multidonor partnership programs, social entrepreneurship ventures, advocacy campaigns, networking efforts and communities of practice, etc. The adapting of the original OECD-DAC criteria to humanitarian efforts was needed, but some of them are still hard to apply (those related to human rights). More work can be done there. The adaptation of the OECD-DAC criteria to global and regional partnership programs (GRPPs), in which I was involved with a team at World Bank Independent Evaluation Group (IEG) in 2007, was an important addition, adding the importance of evaluating governance, and refining the "sustainability" concept which is too limiting, addressing as it does mainly traditional donor-funded projects, and leaving out other legitimate long-term objectives such as "sustainability plus, or scaleup" , replicability in different contexts, "building back better to incorporate environmental improvements or resilience, scale-down (if program not working) or devolution of implementation responsibility to local implementers, etc. The other exercise in which I was involved was funded by the Faster Forward Fund-- to look at both general critiques of the OECD-DAC criteria with a fresh eye-- and -- how they might need to be adapted to apply to social entrepreneurship ventures. My findings and suggestions are available on request. The critique captures some of Caroline Heider's points about complexity-- social entrepreneurship ventures are dynamic enterprises, with often conflicting dual objectives-- of profit or financial sustainability and social purpose-- which means the optimal economic equilibrium may be shifting over time. How can this dynamic be captured? To sum, I believe the OECD-DAC criteria served evaluations well for awhile, serving as they did mainly as a guide for more in-depth and flexible evaluation questions-- but there are flaws that, when addressed, would strengthen them further. One of the most obvious-- and that most evaluations just naturally overcame by adding the appropriate evaluation question-- was "Reach" and "Results" -- there was no requirement for a description of actual results -- whether outputs or outcomes--or the factors that improve results and subtract from them. (You could say that impact covered this, but in fact, there were many cases where intermediate outcomes were the most visible and measurable). The other flaw often mentioned is that one doesn't have to base an evaluation on pre-determined objectives (as efficacy calls for). One can have a "goal-less evaluation". A similar flaw is that many times different stakeholders have different objectives, or "stakes" in the program. With the increased emphasis on stakeholder participation, the assessment of "efficacy" (and the other criteria which implicitly also rely on objectives) is limiting. Finally, the concept of sustainability was often misunderstsood, even when applied correctly in the context for which it was meant-- donor-financed projects. The correct interpretation was "sustainability of outcomes beyond the project period after donor financing ceased". (Many mistakenly thought it referred to sustainability of the project activities). But even when applied correctly- the time frame was not well-defined, and the most important question-- how is exit to be managed?-- was not always asked.. Nowadays, with so much emphasis on environmental sustainability, the term simply creates more confusion than is worthwhile. And for the many other "endeavors" we find in the world of economic development today-- sustainability is indeed, only one of many long term objectives.

It is a very good discussion…

It is a very good discussion .. with many elements of reflection ... the challenge is how people take advantage of the tools, as a means and not an end .. change the view of the use of criteria CAD and reflect on their improvement, always Is necessary .. since in some cases the same projects use the same criteria and evaluations as a school qualification list and the evaluation itself does not see it as an opportunity for improvement .... but as a requirement within the project. ... and are not giving adequate use to the results of an evaluation and do not make the adjustments that must do or how to enhance the changes generated positively by the intervention and scale in the model ... Another important point is how involving to different authors .. example independent consultants who are applying the evaluations. And how to move these discussions to different regions and in different languages.

Add new comment

Rethinking Evaluation: Agility and Responsiveness are Key to Success

kmilhollin — Tue, 07 Feb 2017 19:10:10 +0000

Rethinking Evaluation: Agility and Responsiveness are Key to Success kmilhollin Tue, 02/07/2017 - 14:10

Comments

Incorporating the new…

Incorporating the new criterion into evaluations will add complexity to the art and science of evaluation. For example, while responsiveness and agility may be seen by a subset of the team as more than less of an imperative to achieving success, it may be that key stakeholders disagree on the what, how and when. Also, it would seem natural that a leader be empowered to change course on policies and programs in a timely manner, though there are different incentives and numerous approaches to leadership that will need to also be considered. This implies a need for additional training for evaluators, as well as new requirements for self-evaluation to capture some of the context/thinking while still in the moment.

Entirely right, it would…

Entirely right, it would take new skills and tools to do this, and to do it in ways to identify where and when systems limit the capacity of leaders to make necessary course corrections, as well as understand when wrong choices were exercised to avoid making the same mistakes over.

Dear Caroline, agility and…

Dear Caroline, agility and responsiveness are indeed very important, especially in humanitarian and peacekeeping contexts. I find that the argument that development practitioners often advance about ‘objectives no longer valid’ is a bit prosaic, as it very much depends at what level objectives are defined. Expected accomplishments and longer-term goals might remain the same, even in case of a change in environment, while immediate outcomes could significantly vary. That being said, I think it is very important, as you very well put it, that evaluators maintain an equal level of flexibility in assessing the relevance, efficiency, and effectiveness of programs, and acknowledge managers' capacity to adapt to change when it occurred. Equally important, I think that there is room for evaluators to further assess any resistance or 'immunity to change', not only at systemic and institutional level but also at team and individual level. This is where evaluators could join efforts with organizational psychologists to understand better the internal dynamics and inform more sustainable patterns of change.

Many thanks, Anna, great…

Many thanks, Anna, great suggestions.

Thanks Caroline for raising…

Thanks Caroline for raising this very important but challenging aspect. This resonates with the previous topic about relevance (in the evaluation criteria chain): being responsive and agile in order to remain relevant? This needs an enabling evaluative culture and environment, both at institutional and individual level, as well as methodological diversity and flexibility. However, current donor environment shows apparent preference and requirement for rigid tools such as logframes. A balance needed on manageable degree of responsiveness and agility, which perhaps is most challenging and needs some underlying transformation. Even though challenging as it is, it still needs us to face, discuss and better our responses.

Thank you Ting for your…

Thank you Ting for your comments. I agree that the discussion is needed beyond the evaluation community, but am focused on the professional group where I have the greatest responsibility and maybe some influence. Wouldn't it be great if the discussion among evaluators would influence practitioners, including donors, to adopt new ways of working?

Thanks Caroline for your…

Thanks Caroline for your prompt reply. Yes, true. People tend to stay in the familiar and comfort zone, it's great to have such thought-provoking discussions initiated and led by IEG, which certainly benefits not only the evaluation community but also wider range of groups of development practitioners.

I have recently worked on a…

I have recently worked on a number of assignments on evaluation of humanitarian programmes. After reading this blog post, I feel nowhere the indicators of agility and responsiveness are relevant that humanitarian programmes. Although we try to cover things under the rubric of relevance but the programmes are actually either responsive to the needs or they waste resources. However, in development interventions too. Responsiveness should be measured possibly while understanding "the most significant changes" and while unpacking the logic model. Thanks for the blog and the ideas. I shall try to experiment with it in the next assignment.

Thank you, Zubair. And…

Thank you, Zubair. And remember to share your experience so that we can all learn from it.

Dear Caroline, Many thanks…

Dear Caroline, Many thanks for sharing another perspective of looking at evidence building agenda with another perspective. I will test these additional questions to include in the upcoming evaluation studies. These are very useful points to learn/understand about the programme management and how often programme teams utilised the evidence to inform decision making. on similar lines , we have developed process indicators to ensure the utilisation of evaluation studies. we will gather more evidence on the utility of this system. lets see how these things will work in future.
I must say that you have great contribution to bring in a fresh and interesting perspective in the development sector. I really find your blogs excellent.

Thanks for your support and contribution.

Dear Ehtisham, many thanks…

Dear Ehtisham, many thanks for the feedback and for the details on how you are evolving your evaluation practices. Looking forward to hearing more about it.

Rethinking Evaluation should…

Rethinking Evaluation should be Dynamic Process. Caroline has in the series raised serious issues of serious business that deserve the serious attention of Policy Makers and Decision Makers within and beyond Evaluation. In "Have we had enough of REEIS" Blog article, Hans (DAC OECD) historical insight shed light on priorities and direction moving forward this dialogue in Global Interest.

Hans contribution underline urgent need to take applied history more seriously and to better appreciate that in the first 50 years of International Development Cooperation (1960-2009) the overarching lessons learnt is that no lessons have been learnt hence failure to build bridge between lessons learning and lessons forgetting was re-occurring decimal. As the second 50 years of International Development Cooperation reach 8 years these flaws and failures persist. This is scar on the conscience of relevant authorities - World Leaders, Regional Leaders in each of the 5 Continents and National Leaders in each of 306/193 UN Member States

The SDG which applies equally to all North and South Countries, unlike the MDG which applied to South Countries only underline need for National Leaders and World Leaders on UN Member States Governments: Executive, Legislature, Judiciary at all tiers; UN System: UNO, WBG, IMF Entities including IEG-WBG; CSOs/NGOs; Farmers and Processor Organizations; Private Sector: Micro, Small, Medium, Large, Multinational Enterprises; Academics and Researchers; Internal Consultants and External Consultants to individually and jointly address Rethinking Evaluations real and complex problems on the ground at Community, Sub-national, National, Sub-regional, Regional and Global levels.

2017 is Year 2 of Implementation and 1st quarter end in few days, yet there is no evidence that National Leaders and World Leaders are seriously committed towards addressing Rethinking Evaluation real and complex problems on the ground in each Community, each of 306/193 UN Member States, each of 5 Continents and Worldwide.

Logframe has been identified as part of the problem. Is this really the case? Again going back to history. Hellmut Eggers at EC Evaluation created Project Cycle Management, PCM Benefits focused Approach to Evaluation in 1987. He retired in 1993 before the idea could firmly take root. Although PCM Approach is widely used Worldwide, what is being used is far from Original PCM. Original PCM answers many of the questions that this dialogue is grappling with.

Working independently Lanre Rotimi created Policy, Program, Project Cycle Management and Comprehensive Systems Reform, 3PCM and CSR Benefits focused Approach to Development, M & E, Performance Management (Service Delivery), Procurement and Human Rights in 1993. 3PCM and CSR is Significant Improvement on PCM. In March 2009 Hellmut and Lanre Versions of PCM were merged into the 3PCM Version of PCM.

3PCM has 4 Principles, 4 Instruments / Tools corresponding to each Principle two of which are ToR and Logframe; 4 Practices and a Database. It is pertinent to note that a major Evaluation challenge is an Evaluation having a different ToR from the ToR of the Policy / Program/ Project being Evaluated.

The point made by Hans underline urgent need National Leaders and World Leaders to adopt a One Worldwide Approach to Evaluation that is a Common and Systemic Approach known and practiced at Community to Global levels in each of 306/193 UN Member States and not a one-cap-fit-all Approach.

The points made in Caroline and many contributors to this dialogue underline urgent need for National Leaders and World Leaders to adopt a One Worldwide Approach to Development, Diplomacy, Defense, Democracy, Data and Digitization - Research: Research, Planning, Statistics/Data; Implementation; Evaluation: Monitoring, Evaluation; Success: Learning, Results, Success Policy, Program, Project Interventions, 3PIs and 3PIs Training as One.

It is our hope that this series will not be about Talking and Thinking but Action and Results as the dialogue progress in grappling with Rethinking Evaluation Challenge in our World as is and not as any Stakeholder - no matter How Powerful wish it to be.

Hi Caroline, please find…

Hi Caroline, please find attached link to article moving forward points earlier made by Davis and Lanre
http://developmentchangechampions.blogspot.com.ng/2017/03/global-push-t…

Points raised highlight Acid Test of Credibility of this IEG Initiative - How it help to deliver:-
1. Better Development, Diplomacy, Defense, Democracy, Data and Digitization
2. Better Trade, Aide, Debts, Anti Corruption, Anti Terrorism and Migration
3. Better End Hunger, Malnutrition and Poverty
4. Better Trust, Integrity, Openness and Transparency

2nd quarter 2017 Year of Implementation is racing to an end, yet many fundamental issues that ought to have been settled by end 2nd quarter 2015 Year of Implementation are still outstanding. As long as National Leaders and World Leaders do not face new direction and adopt new priorities, these information, research and knowledge gaps will remain re-occurring decimals. Allowed to occur the ultimate consequences for over 2 billion poor the UN System: UNO, WBG, IMF serve could be catastrophic.

The fundamental issues we consistently raise cannot be wished away. Stakeholders in the 7 blocks identified need to accept to work jointly as they discuss, negotiate and make all necessary arrangements for 2030 Agenda Implementation and Evaluation sustainable success at each specific Community - Global location context.

We have thrown up the gauntlet. There is a need for you, other authors and Contributors to genuinely demonstrate "Walk Your Talk". Will you and others pick up the gauntlet?

Add new comment

Rethinking Evaluation— Is Relevance Still Relevant?

kmilhollin — Tue, 17 Jan 2017 15:54:49 +0000

Rethinking Evaluation— Is Relevance Still Relevant? kmilhollin Tue, 01/17/2017 - 10:54

Comments

Thanks Caroline for the…

Thanks Caroline for the informative blog and perspective. System thinking was well embedded in those key points through taking into account of diverse perspectives and connections/links etc. In reality and in system thinking terms, how to then make those boundary choices to determine what is relevant and what is not? Perhaps an actor-centered thinking or theory of change might add some value by making explicit major actors involved, their inter-relationship and domain of interests to help identify the focus and priority? For instance, like the actor centered logic of Outcome Mapping -- differentiating various groups of project actors locating in varied project domains (sphere of control, sphere of influence and interest).

Many thanks, Ting, for this…

Many thanks, Ting, for this interesting contribution. We will collect a number of suggestions and reflect on what they mean for our work going forward. Until then I hope you keep sharing your ideas.

As evaluators, our focus…

As evaluators, our focus should also span across the evolution and progress of an intervention's implementation, over a specified duration of time. It signifies that, along with the application of relevance, effectiveness, impact and sustainability as evaluation criteria, there would be a requirement of an addition of "Evolution/ progress" as one more evaluation criteria. This progress would be calculated and measured at the country level or regional level, on the basis of year - on - year increase or decrease in the implementation of the intervention under focus. It would be calculated with the utilization of the mathematical formula, as mentioned below :-

Evolution / Progress = { [ Year n's contribution - Year (n - 1)'s contribution ] / [ Year (n - 1)'s contribution ] } * 100%

This contribution would be the contribution of the nation or the measured collective contribution of the region, for that particular year (n) and the year (n - 1).

Then based on the above deduced values of Evolution/ Progress, countries as well as regions across the world would be ranked. Thus leading to the creation of Rankings based on the evaluation criteria "Evolution/ Progress" !

Ashwini, as you can see from…

Ashwini, as you can see from my subsequent blog, I agree with you that we need tools and a focus on assessing change over time, and whether adaptation is timely and responsive. Not sure that this can always be calculated as you suggest. But, even a more qualitative approach will be useful.

Dear Caroline,…

Dear Caroline,
I fully agree with you and would like to add that for an intervention to be relevant it must make a difference in the development problem we would like to address. To paraphrase a former colleague, all of our projects do good things but whether they have an impact is an entirely different question. Seen from that perspective, relevance must be linked to impact. In fact, your idea about the need to address the key entry points is important.

Thanks, Juha, and I think…

Thanks, Juha, and I think this is even more relevant for the green agenda that you and your team are working on at The GEF.

Dear IEG and Caroline- With…

Dear IEG and Caroline- With all due respect, "After nearly 15 years of applying relevance, effectiveness, efficiency, impact, and sustainability as our foundational evaluation criteria" and the Bank still cannot tell us what project outcomes and impacts were standing or newly emerged after the Bank ended its project and programme funding?! We seem to have found that you rarely consult the very participants your President extolls (see http://valuingvoices.com/ieg-blog-series-part-ii-theory-vs-practice-at-…) and you want to shift the focus from evaluating sustained impact (done on less than 1% of all projects) rather than doing far more which is true accountability to those you serve? Help me to understand!

Jindra, I am confused about…

Jindra, I am confused about your comment, as it doesn't seem to relate to the blog. I also think it is a rather sweeping statement that we do not consult stakeholders and civil society at all. Of course there is always room to do more, but as you would know there are many players and those who are not consulted can be very vocal, while those who have might not.

Interested to read the blog…

Interested to read the blog but can not really comment until I have - so will only comment if I can read it

If you came to this place in…

If you came to this place in the blog via a link, try scrolling up to the top of the page to be able to view the whole blog :)

I liked the vision of adding…

I liked the vision of adding a LOT MORE to 'relevance' - I'll be interested to follow up on some of the new lines of thinking in evaluation opened up by the article - as a market system development specialist - we have been, for a number of years, been interested in identifying the nodes of intervention that have the biggest chance of influencing whole value chains or inducing systemic changes that create the biggest impact. I agree - we all need to make what we do more relevant.

Thanks, Kevin, would be…

Thanks, Kevin, would be interesting to hear more about the experience of the private sector in this regard.

I like the thoughts about …

I like the thoughts about "is relevance still relevant?". the paradox is: if a project had been relevant for whoever at the start and if it has been successful, it would ideally become obsolete or "irrelevant". how are we to assess that the project is still relevant after it supposedly achieved its objectives?

Petra, you raise an…

Petra, you raise an important point. I think it illustrates well the difference between a project that builds a service capacity, and the continued provision of services. For instance, if a project develops the capacities to deliver health services, the project to set these up might no longer be needed/relevant, once they are up and running, but the health services themselves continue to be relevant to the needs of the population. One could imagine another example, outdated by now but useful as illustration effect, a project that build capacities for delivering IT services. Let's assume it aim to develop landlines (anyone remember those?), because that was the technology at the time of approval. By the time the evaluation comes through, the objectives -- to provide affordable IT services to people -- might still be relevant, but the technical solution is not any more.

Thanks for sharing your…

Thanks for sharing your thoughts on this- I have been thinking about this criterion so much, particularly lately. I am inclined to say that, as it stands now (and with usual questions used within it)- relevance is obsolete and does not significantly contribute to the evaluation. This is due to the fact that, in developmental context (even in case government policies are not there or not elaborated), all interventions are 'relevant' as there are so many needs and so many gaps. From that point of view, your suggested points are going in the right direction, though I would for start call it differently to highlight the different angle that what we actually want to assess under relevance. In most cases, even if an evaluation findings point that an intervention has diminished relevance (due to many reasons, including also efficiency of process from programming to contracting to implementation), this cannot be well reflected, due to way the questions are set. Even when evaluators are in charge of setting evaluation questions, there is a tendency to 'go with a flow' and pose the same old questions, meaning that we will get the same set of answers. This results in the fact that, for vast majority of evaluation reports, relevance is always positive. Review of ROM monitoring reports (EC monitoring tool) also shows that if not all, majority of projects are rated with A or (soft) B. This, to me, is a good indicator that something needs to change quickly or we will just continue to do 'lip service' to our clients.

1. Looking at relevance at…

1. Looking at relevance at the evaluation stage is too late. The focus should be on if and how the intervention (project, Program or whatever) was monitoring what was happening its context that should be measured.
2. Relevance is important in more in design and less in implementation and much less important in evaluation. Analysis of relevance should be done in design and included in the monitoring systems and processes.

Add new comment

Rethinking Evaluation – Have we had enough of R/E/E/I/S?

kmilhollin — Tue, 10 Jan 2017 17:33:32 +0000

Rethinking Evaluation – Have we had enough of R/E/E/I/S? kmilhollin Tue, 01/10/2017 - 12:33

Comments

A "Copernican" moment in…

A "Copernican" moment in time or the beginning of an Archimedial "Eureka" and a revolutionary movement for change? Are the SDG's an opportunity to reframe the Evaluation dialogue and build the foundations for a more embracing, resilient, inclusive and sustainable world? A time for more innovation and creativity in both approaches and methods and a recognition of the longer term trajectory of change - and dare I say it - behavioural science and economics - it's a messy and risky business we're in!

Thanks, Mark. Good questions…

Thanks, Mark. Good questions. And, yes, it's a messy business, but that's even more a reason to ask the difficult questions and find answers to them.

Thanks Caroline for such…

Thanks Caroline for such thought-provoking message. There's debate and concern on suitability and appropriateness in choosing and in applying evaluation methods in various contexts, perhaps similarly, how those ‘gold standard’ DAC evaluation criteria get used in reality is worth reviewing as well? Are they simply used as ‘checklist’ to fulfil bureaucratic requirements or are they applied to really reflect and capture the change stories? Are those criteria more relevant for some types of programmes/projects than others or are they applicable to most cases? This seems interesting research area.

Thanks, Ting, for your point…

Thanks, Ting, for your point about asking also whether the criteria are being used -- at all, as a checklist, or otherwise. A number of blogs speak to exactly this point. I hope you will comment and add your experience to the discussion.

The R/E/E/I/S framework can…

The R/E/E/I/S framework can capture diversity and unintended consequences if we allow it to. The problem is not R/E/E/I/S but the way we use the Results Framework, like a racehorse's blinkers, to focus our attention on our original intentions.

Thanks, Alex. Yes, there is…

Thanks, Alex. Yes, there is a question about application. Like any tool: it can be designed with the best intention and ideas in mind, but when used badly even the best tool cannot produce good results. But, I do think and will argue in the blog series that a number of the tools need a face lift. Hope you stay engaged and post your contributions as we go along.

Caroline and Alex thank you…

Caroline and Alex thank you so much for your comments. Not only do we need to rethink evaluation but we need to be think the aims of development. Too often it is we who design development projects with what we think they should have rather than consulting local partners and participants in-country from the onset on what they think they should have. Far too often we evaluate how well they fulfilled our plans rather than them evaluating us on how well we have helped them have sustainable health, livelihoods, environmental conditions (as your article alludes to). Do we design for sustained impact? Not yet ;). Thank you.
More at www.ValuingVoices.com

Point taken. And, by…

Point taken. And, by rethinking evaluation -- including our purpose, approaches, tools -- we can stimulate a conversation and generate evidence to contribution to the dialogue that the development community is undergoing as well.

Readers will probably have…

Readers will probably have many candidates for additional or replacement criteria, but one which I think quite a few people would like to see included is: equity

Many thanks, Rick. Very…

Many thanks, Rick. Very useful. We are right now in the process of evaluating what the World Bank Group calls "shared prosperity", which is all about distributional effects and equity. There will be a lot to learn in terms of evaluation methods. Looking forward to your suggestions.

There is a trend towards…

There is a trend towards technicist instrumentalism, a kind of elaborated tick box approach to evaluation, that answers the questions of the evaluation criteria but in the end often leaves me feeling flat, dulled by the blandness of reports that lack sharply observed engagement with the complex realities of the programme. Evaluation criteria and the associated norms, standards and quality assurance measures were needed to support the many new evaluators who came into the field as it expanded in 2000s. That helped make sense of evaluation, but now we are in danger of losing its real meaning. The criteria have done their time, bravo, but lets now loosen the straightjacket.

Thanks Caroline for a…

Thanks Caroline for a stimulating blog post. What we need are evaluative omnivores, more flexibility - willing to try new approaches and methods (developing the menu - not sticking to it) that are driven by context and questions and the demands of those wanting evaluation (who, at least outside of the bilaterals and multilaterals not always aware of R/E/E/I/S or want it when they are made aware). We use it for some evaluations (when appropriate) and others we are trying out feedback (constituent voice - with social entrepreneurs) and exploring the use of developmental evaluation for impact investing - where flexibility, embedding evaluation / and evaluator into operations, quick feedback and learning is needed in interventions which are not neatly conceptualised and subject to change. R/E/E/I/S in the way it is presently applied does not offer much flexibility. It probably has contributed to the mass of impentrable evaluation reports being left of the shelf - the same old reports spat out to the same old formular. We are in the business of evidence, learning, influence and change - can we always get it OECD DAC criteria? No. So I agree, time for a rethink and a fun and interesting debate to make relevant for the challenges of the SDGs, impact investing and corporations who want to be 'a force for good'.

Many thanks, Lee, for your…

Many thanks, Lee, for your comment and contribution. It is exactly this -- opening our tool box and asking ourselves what else do we need to be most useful to policy-makers and practitioners so that they can make better informed decisions -- that we aim to stimulate with this discussion. Thanks for sharing your experience.

Thanks Caroline for a…

Thanks Caroline for a succinct, but very powerful message on the need to rethink the current practices in evaluation. I am relatively new to evaluation. Couple of observations: (a) Evaluation criteria are set to measure the "project mode" of work. There is a need to adapt evaluations/use flexible approaches in several areas where global organizations work, which involve supporting political processes and aims at policy changes. As you rightly pointed out, evaluation in such cases are complex in nature, and attribution is often difficult; (b) Impact of normative work of global agencies are not easy to measure, as decision to use and the methods of use of the normative tools/products/instruments involve political decisions.

Look forward to more thought provoking posts.

Thanks for sharing these…

Thanks for sharing these thoughts. The evaluation tools have often been used well beyond the project for larger evaluations, where they make some sense, but might have over-powered other important questions that need to be asked. For the normative work, the UN Evaluation Group discussed some time back the evaluation of normative work. Hopefully some of them will contribute to the discussion here.

Thank you, I enjoyed reading…

Thank you, I enjoyed reading the ideas here. Doing “complexity aware evaluation” or one that has “equity” at its core is a lens through which evaluation projects could take place, regardless of what criteria is used. But I do agree that developing these concepts into explicit evaluation criteria, or adapting the current ones to better represent these ideas, help keep them at the forefront of evaluation projects. It does matter to non evaluators, absolutely, particularly to program designers, but also to anyone who genuinely wants to help put data into action.

Good points. Thanks for…

Good points. Thanks for pointing out that users of evaluation also feel the need for this information and clarity about what gets evaluated and how.

It is time to improve the…

It is time to improve the evaluation practice by adopting better and more robust approaches to the assessment of program impacts and of their relative contributions to observed results in the normal context in which these programs operate. Politicians need more reliable assessments of the cost-effectiveness of program interventions to support (re)allocation decisions. Unfortunately, because of capacity issues, the evaluation community has been focusing too much on efficiency and has generally failed to deliver critical information on effectiveness and cost-effectiveness. Economists have been able to develop complex monitoring systems to measure and manage the impact of fiscal and monetary programs/policies. The technology and methodology has been there for some time. The evaluation community should start pushing for equivalent system and capabilities to monitor and manage the impact of public programs that are more oriented towards social development, health and environmental issues.

Michel, interesting point…

Michel, interesting point. First of all: thank you for recognizing that decision-makers need evaluation inputs, and for pointing to some areas where more work would be helpful. The only question I have: where have evaluators focused more on efficiency? i am not sure that this is the case where I have worked and would like to learn more about examples and practices.

Thank you for challenging…

Thank you for challenging the status quo. The five DAC criteria indeed have been very helpful in bringing discipline, comparability and reliability into the evaluation world , but they often need to be adapted, stretched or accompanied by other criteria to better analyse and unpack the complexity of the world we try to evaluate. For instance by including other criteria such as equity, accessibility, sustainability, policy coherence, etc. The new SDG indeed offer us a great opportunity to stop and rethink how best we can meet the increased demand and desire to unpack and analyse out reality to guide future policy interventions.

Interesting discussion. I…

Interesting discussion. I agree that diversity, social distribution, environment, unintended effects, are increasingly important to consider in evaluations but in my opinion the five criteria will still remain valid and can cover these issues, with enlarged and rethought methods though. In the Council of Europe we address often the political framework conditions in legal and political terms, providing advice for changes of constitutions and laws in various fields, but also through projects ranging from policy advice to training and technical assistance in many areas. They are not growth oriented nor affecting the environment nor the income distribution but nevertheless can be and are evaluated by us using the five criteria and mostly qualitative methods of data collection. Our main problem is documenting impact, as this is long term and the causal chain is difficult to isolate.

Ansgar, thank you. I agree,…

Ansgar, thank you. I agree, there are a lot of things that can be addressed with the criteria, but that will depend on individual evaluators and lead to rather uneven practices. While there is value in interpretation and flexibility, if there are areas where consistently we need to interpret and adapt, maybe it is time to rethink a little, without throwing the baby out with the bathwater.

Good to discuss, but I think…

Good to discuss, but I think that you can build most of the social, equity. environment related issues into the initial set of objectives and then they will show up in the evaluation, even if using the five DAC criteria. Afterwards it is rather a question how to get the data, including on unintended effects.

Very interesting discussion…

Very interesting discussion. From my perspective, the big Five remain a valid to discuss about evaluation objectives. What I’ve found frustrating is when these criteria are put in ToR as a sort of shopping list where implementers or donors (more often implementers) ask for everything without really focus on specific, realistic and useful key research questions.

Agreed

This is an interesting…

This is an interesting discussion. I would like to add some points related to the origins of the DAC Evaluation Criteria, provide a suggested explanation for the wide-spread use of the criteria, and comment on the perception that they have been conceived as a ”straightjacket”.

First on the origins. In 1989, and quite freshly recruited to the OECD, I took on responsibility for the DAC Expert Group on Aid Evaluation (now the DAC Network on Development Evaluation). One of my first major tasks was to lead and co-ordinate the drafting of the DAC Principles for Aid Evaluation which were developed over a two year period with member countries and partner organisations. The principles were adopted formally by the Expert Group and then by the DAC in 1991. The principles spelled out the role and purpose of evaluation in development co-operation and clarified important principles for the evaluation function in agencies – such as impartiality , independence and credibility, as well as on evaluation programming and management. The principles also state that it is essential to define the questions which will be addressed in an evaluation – these were referred to as the ‘issues’ of evaluation, and laid out a manageable framework with basic groups of evaluation issues. These issues are the origin of what then became the DAC Evaluation Criteria.

Why have these criteria become so successful in terms of their wide-spread use? Even I have even been somewhat surprised by that, to tell the truth. While we expected them to be used by our members, it is clear that they are used by many more actors in development including partners, civil society, etc. I believe that a key reason for their success is that the DAC criteria are a manageable and relatively easy framework to understand and to use for grouping your key evaluation questions. Moreover, if each funder or organisation used their own specific criteria, it would be difficult to compare and collaborate on evaluations. We have worked to unify basic concepts, terminology, management of evaluations, and quality standards so that we can work together and communicate with a shared understanding. It should be noted that much of the normative work is available in a number of languages besides English – which has not only involved translation but adaptation to the linguistic and cultural setting in which these tools are to be used.

Finally on the “straightjacket” argument. The criteria were never meant to be a straightjacket, but conceived rather as a tool to help more or less experienced evaluation managers to structure the key questions of the evaluation. We have recommended an application of the criteria that should be tailored to the purpose and use of the evaluation – this may mean that some criteria will be in focus while others may not be relevant for the evaluation. Moreover, we have developed additional criteria for use in evaluations of humanitarian assistance - such as coverage and coherence - and also in our guidance on evaluating in settings of conflict and fragility. The DAC Evaluation Standards, adopted in 2010, were developed through a three-year test phase followed by a consensus building processes led by the Network secretariat with many development agencies, partners and ministries involved. The quality standards recommend the application of the agreed criteria, but it is also explicitly stated that the application of the five criteria and any additional criteria depends on the evaluation questions and the objectives of the evaluation. We are not in favour of a straightjacket or a mechanistic application. Rather, they should be used as a framework to help frame the key evaluation questions.

This said, a discussion on the criteria, their application, and potential new dimensions is certainly welcome. Should a change be needed we should be open to it - but not a change for the sake of change but only if we are convinced that the current approach does not help us in framing the right questions to assess an activity, programme or policy.

My apologies for reacting…

My apologies for reacting late to the last contribution, certainly a most valuable one to those who view evaluation practices in an historical perspective. Indeed, I concur with the view that the establishing a set of uniform criteria has proved to be effective in structuring development evaluations, and thereby their overall technical quality.
However, not all development agencies enjoy the ‘luxury’ of having independent evaluation units or departments. Even if they have, I wonder whether the REEIS criteria have really served as a kind of safety velvet in ensuring the “impartiality, independence and credibility” of the evaluation function, as mentioned by Hans (Lundgren). If this were the case, individual professionals – all learning on the job – would have become more autonomous in exercising their evaluation role than some decades ago. This hypothesis deserves to be tested, so let me illustrate this below.
Last week I realized that based in a private consultancy and research foundation (CDR in San José, Costa Rica) I had become involved in external evaluations for a period of exactly 20 years. The assignments mostly focused on rural development ‘interventions’ relating to financial services, value chains and (fair) trade networks, environmental governance, sector programmes and institutional development. Most of them were ex–post evaluations, carried out in the western hemisphere. Partly I undertook the evaluations alone, but to a significant extent it was teamwork.
I decided to assess the extent by which I had experienced the three above mentioned elements in each of the in all 64 assignments, during the period 1997 – 2016. Such exercise is obviously due to subjective judgement and prone to some retrospective mental bias, so I should at least try to reduce that risk.
‘Professional autonomy’ may for this purpose viewed in three dimensions. First, the quality assessment of the technical feedback with the client, concerning evaluation methods and tools for data collection. Second, the support received and latitude offered during the field phase in meeting with relevant actors. Third, the receptivity of the client in dealing with – i.e. ultimately accepting – the contents, conclusions and recommendations of the evaluation. On each of these indicators, I gave a score to the assignments between 1 (lowest) and 4 (highest). In determining the overall score of an assignment, I decided to give the first two factors single weight, and the last one a double weight. As a final step, I wanted to average the yearly averages in a quinquennial (5 yr.) average. So apart from an average 20 year score, for each half a decade I got a separate score.
The resulting average of ‘professional autonomy’ for the entire 1997-2016 period is 3.45, which may be interpreted as between sufficient and good. However, the five year averages turned out to be 3.58, 3.59, 3.40 and 3.21, respectively. This in my view reflects the trend as perceived by myself that, in spite of methodological upgrading, there has been a steady loss of impartiality, independence and credibility. This has less to do with the REEIS criteria (including the relevance of the R) and more with the institutional and policy context in which evaluation work has been carried out over the last ten years.
At the time of writing, at many levels there is a debate going on about journalists, judges and other professionals, whose work – with unpleasant results to those at the top – has come under immense pressure. Not in the public domain, but to some extent certainly comparable, has remained the work of development evaluators when facing their clients. I fully agree with overarching goals of respecting underprivileged groups, limiting overconsumption and dealing with complexity. However, the conditions for that include a minimum degree of autonomy of the profession, which in my impression has not enjoyed the same degree of global recognition as REEIS. Time for a rethink?

Hans, thanks for your very…

Hans, thanks for your very personal reflection and sharing your long-standing practice. I agree that the criteria -- REEIS -- do not automatically lead to independence, credibility, and utility. For that some other dimensions are needed. But, transparency and clarity about criteria, methods, and processes is an important factor in ensuring credibility and can lead to utility. In my view it even helps with independence, because it makes it more obvious whether and if so where interference happens in assembling and interpreting evidence.

Thanks, Hans, you are really…

Thanks, Hans, you are really the institutional memory on this topic. And, you are right: I shaved some 10 years off their existence and struggled to reconcile my knowledge and use of them well ahead of the compendium. Thanks for that clarification. I agree with you about the value of the criteria in shaping the profession. Equally, I hope this renewed debate about rethinking evaluation will help renew the profession in ways that the criteria did way back in the early 1990s.

You do not need contemporary…

You do not need contemporary reasons for ditching the REEIS - they have always been terrible: a bureaucratic solution that imbedded the input-focused desires of those who came up with them. Given the high degree of failure in ODA projects (I am not being cynical here - failure should be expected as normal - in complex dynamic environments in developing countries, it may be higher than private business start-ups in our own countries [where over 50% of new businesses close within two years]), what is crucial is understanding impact. Yet under REEIS that is just one of five criteria, and indeed - god knows how or why - is separated from sustainability. I suppose if we put them together we might have to take post-evaluations seriously. "Relevance" is the most irrelevant, certainly in practice as it is a box ticked by being a project supporting some vague sentences of a recipient government 5-year plan (why do developing countries need these and ours do not?). "Efficiency" and "Effectiveness" are just arse-covering for ongoing funds by donor organisations: private firms would laugh at the whole value-for-money black hole that DFID has gone down. Who cares about these when projects have no impact - oh, sorry, there are no failures!

This is a beautiful and…

This is a beautiful and inspiring article (and comments!!!). REEIS are not terrible. The challenge for evaluators remains the operationalization and tailoring of the big-five criteria to the specific context and operation being evaluated at the time of preparation of the evaluation terms of reference. If we tailor, the straight jacket gets more comfortable…even if excessive tailoring comes at the price of reduced comparability. In the future, in a complex world where resources are scarce and the risk of social resentment is high, concepts such as social equity, social well-being, policy synergies, coherence and connectedness can become more and more dominant in our evaluations for the simple reason that they are key elements in determining the merit and worth of development operations. Not sure though whether these can be elevated to criteria. Similarly, as said in a more recent article, the challenge will be having the right instruments to draw conclusions on the sustainability….

Inspirational blog, thinking…

Inspirational blog, thinking beyond the boxes! Conventionally, evaluation sounds like giving subject grades at high school and REEIS echo the subjects (Maths, Science etc.). The evaluation focus is also on the linear relationship of inputs to impacts; and resonates reactive. The question for me is on how I can make my evaluation proactive in getting expected and unexpected OUTCOMES on poverty, gender, health, environment, climate change (MDGs/SDGs) and learning. As an aspirant to evaluation, I was wondering if improving/changing criteria would make the evaluation a keyed up process.

Indeed, it is reaffirming to…

Indeed, it is reaffirming to read this blog and the resonance in the comments. It is a topic that we have also been discussing and will have a related panel presentation at the American Evaluation Association conference in DC this November called, “Beyond the DAC-gnificent evaluation criteria: From learning to action through methodological innovation.” (Panel Members: Michele Tarsilla, Steven Hansch, Sara Vaca, Riccardo Polastro, and myself.) “The panel will consist of an in-depth reflection on the use and misuse of criteria in contemporary evaluation practice. Based on the realization that evaluation commissioners often include OECD-DAC criteria in Terms of References (ToR) by default, this panel will encourage the audience to rethink the way evaluation criteria are selected during the development of either ToR or evaluation proposals. The first presentation will showcase instances where the integration of OECD criteria with others outside of the “conventional” paradigm, has proved particularly beneficial. The second presentation will focus on how the use and periodic revision of internationally agreed standards in humanitarian evaluation (e.g., Spheres) has generated operationally relevant learning. The third presentation will specifically engage the audience in a visualization exercise that will illustrate the linkages between the OECD-DAC criteria and a set of newer ones increasingly used in development and humanitarian evaluations.”

Good story Caroline. The…

Good story Caroline. The five criteria could remain relevant to achieving SDG with a new approach to risk tolerance and management. LDC's will struggle to make progress towards middle income security and growth without 'game-changing' or transformative strategies, policies and programmes, coupled with improved implementation capacity and resources to match. The five criteria, 'R/E/E/I/S', dont really deal with transformative impact, just impact. Ethiopia is one country that's explored the transformative strategy and achieved some success but implementation has remained challenging. For development partners to remain 'relevant' in a 'Copernican moment' or transformation phase they need to increase their tolerance for risk in taking on solutions to the serious binding constraints facing LDCs and not just run away when an investment opportunity looks a little risky. Can expect some projects to fail with higher risk tolerance but others to produce the high level outcomes that are need to have a transformative impact.