Rethinking Evaluation - Where to Next?
A summary of the ongoing conversations in the evaluation community about updating the DAC Evaluation Criteria.
I was pleased to hear that the DAC Secretariat was committed to revisiting the criteria. Not to start from scratch but build on the successful foundation that the criteria had established. But, also in recognition of the need for change in changing times.
TWEET THIS
- The DAC evaluation criteria have played an important role in shaping the global evaluation practice, but there are challenges in interpretation and application.
- While a major revamp and addition of international evaluation criteria is not necessary, the criteria will be revisited by DAC following a broad consultative process.
Our Rethinking Evaluation series triggered a much larger discussion than just on our blog. As the question of updating the DAC Evaluation Criteria was discussed at the recent National Evaluation Capacity Conference in Istanbul and the regular DAC EvalNet meeting, it was rewarding to hear from many participants that this series of think pieces had triggered lots of inspiring discussions around the world.
Many people suggested continuing the series on our blog. Others asked what's next?
And that is where the DAC EvalNet comes in. As a norm-setter, it played a central role in developing the first round of standards back in the 1990s. This is why I had invited Hans Lundgren from the EvalNet Secretariat into a conversation in response to the many questions we had received. And, it is also the reason why it was on the agenda at the recent EvalNet meeting in Paris.
Here are my reflections on the discussions at that meeting.
Everyone appreciated the important role that the evaluation criteria had played in shaping the evaluation practice around the world. They had provided a framework, shared definitions and language that helped establish common understanding and comparability of evaluations across institutions. No doubt, they were needed and had done extremely well in fulfilling a unifying role. Even more than expected.
It was recognized that this universal acceptance of the criteria had implications. First of all, it was suggested to make them the "International Evaluation Criteria" rather than the DAC criteria. But, maybe more importantly: any revision will require a broad-based consultative process to engage practitioners from the different evaluation communities around the world. As the review process is designed, avenues will be considered to bring different perspectives together and generate a common vision for the future of the criteria.
However, many institutional representatives also raised challenges they faced, both in the interpretation and application of evaluation criteria. Some echoed concerns that relevance is a criterion too easy to meet. Others spoke about difficulties in measuring efficiency. There was a clear need to review and reflect on the multidimensionality of sustainability. A number of agencies mentioned they had given criteria different interpretations to serve their requirements, and had added criteria as their needs evolved in light of the Sustainable Development Goals or other circumstances.
At the same time, the appetite for a major revamp and addition of criteria was muted. The simplicity and limited number of criteria had worked well for everyone.
I was pleased to hear that the DAC Secretariat was committed to revisiting the criteria. Not to start from scratch but build on the successful foundation that the criteria had established. But, also in recognition of the need for change in changing times.
An additional, often overlooked reason for doing so is that evaluation -- the criteria we use to assess what is being done and how -- incentivizes policy-makers, program designers, and project implementers to focus on the issues that will be evaluated. Hence, if it is important that development projects and institutions produce more together than on their own, evaluators can give impetus to such complementarity with the evaluation of synergy effects.
See also:
Conversations: the Future of Development Evaluation
A candid conversation as to whether it is time to re-think DAC evaluation criteria – relevance, effectiveness, efficiency, impact, and sustainability; in short R/E/E/I/S – that underpin most evaluations in international development.
About the Author
Director General, Evaluation, 2011-18
"Independent evaluation helps institutions look "outside the system" to identify issues"
Comments
Dear iIEG- thank you so much…
Dear iIEG- thank you so much for this discussion. It is wonderful that an update of the DAC criteria is being contemplated and possibly even undertaken. I would encourage there to be a deeper look at sustainability definitions, expectations, and methods, especially in light of evaluating the SDGs and post project impacts. I look forward to fora that are open to developing country input especially valuing voices of our participants and partners in shaping such evaluation criteria.
Dear Caroline, thank you…
Dear Caroline, thank you very much for this blog.
This summer I was facilitating consultations on the Sri Lanka Equity-Focused and Gender-Responsive Evaluation Guidelines. And we had a very interesting discussion about DAC criteria. The consensus that emerged from this discussion was that DAC criteria are still relevant, but they should be adapted by infusing into them the file pillars of the SDGs Agenda: people, planet, peace, prosperity and partinership.
So we came up with revised definitions (applicabe in Sri Lanka) which are as follows:
Relevance:
Extent to which the intended and actual outcomes of an intervention are consistent with:
• the needs of all sub-groups of beneficiaries, including those most vulnerable and prone to marginalization;
• Sri Lanka national SDGs related to all five pillars of the 2030 Agenda (people, planet, prosperity, peace and partnership);
• Sri Lanka national policies;
• International human rights and gender equality treaties and conventions ratified by Sri Lanka.
Effectiveness:
Extent to which an intervention:
• has achieved similar levels of intended outcomes for all sub-groups of beneficiaries, including those most vulnerable and prone to marginalization;
• has ensured that none of the target beneficiaries was left behind.
Efficiency:
Extent to which intervention has ensured that all resources/inputs (funds, expertise, time, etc.) were allocated and used economically and in timely manner to achieve similar levels of intended outcomes for all sub-groups of beneficiaries, including those most vulnerable and prone to marginalization.
Sustainability:
Extent to which the benefits from an intervention for all sub-groups of beneficiaries, and especially for those most vulnerable and prone to marginalization, are likely to continue after all intervention activities have been completed.
Impact:
Extent to which an intervention has contributed to or constrained the progress towards Sri Lanka national SDGs related to all five pillars of the 2030 Agenda (people, planet, prosperity, peace and partnership).
Evaluation of synergies,…
Evaluation of synergies, based on Leopold matrix, translated into an input-output matrix, then reworked into a correlation matrix. The procedure developed by Slovenian Evaluation Society (2006-2017). Our paper: "Apples and Oranges: Synthesis
without a common denominator", 1/7(2014), https://www.dlib.si/details/URN:NBN:SI:DOC-LEB7LI9O/?query=%27contribut…. New version of the text available only on demand (bradej at gmaildotcom).
Thank you for sharing your…
Thank you for sharing your key considerations.
One aspect that could be considered for the next refined version of evaluation criteria is the "outside world". I mean that this material can be (and should be more) used by the community that is not in international development and not directly in evaluation. Keeping criteria as simple and clear as possible will allow you to share them with non-international project managers (who are following PMI or PRINCE2 guidance, not DAC) to encourage them to have more assessment as part of a normal process (in the planning, executing or closing phases).
All this simplification/clarification of evaluation criteria should be communicated to encourage more communities/industries to use them (at the very least in planning their work, and ideally in assessing it). In the end, this could at least add some integrity in their approach.
Please use the opportunity of this renewed/refined version of the criteria to communicate them to non-specialists.
Wishing you success with the next steps.
Dear IEG and Ms. Heider -…
Dear IEG and Ms. Heider - Regarding the DAC criteria revisions, I would be delighted if as an industry we re-examined and re-prioritized 'sustainability'.
How I long to have substantive data on #WhatWorks And #ForHowLong :)!
If you look at the results here on post project sustained impact evaluation: http://valuingvoices.com/cant-wait-to-learn-from-post-project-sustainab… and here on how hard it is to find methodologically clear results post project (Note: finding only 1 World Bank substantive evaluation in your PPARs that asked participants what still stood after closeout, much less lessons learned): http://valuingvoices.com/building-the-evidence-base-for-post-project-ev…. Otherwise it all seems like beautiful #WhatWorks words without substance to our participants and partners who dream of sustained impacts long after we all leave. Thoughts? Jindra
Add new comment