evaluation criteria https://ieg.worldbankgroup.org/ en Consulting on the “Big 5” Evaluation Criteria - What got us here? https://ieg.worldbankgroup.org/blog/consulting-big-5-evaluation-criteria-what-got-us-here <span>Consulting on the “Big 5” Evaluation Criteria - What got us here?</span> <span><span>mpadrino</span></span> <span><time datetime="2018-07-10T12:06:32-04:00" title="Tuesday, July 10, 2018 - 12:06">Tue, 07/10/2018 - 12:06</time></span> <section class="comment-section"><h2>Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&1=21311&2=field_comments&3=comment" token="t3z6Vs803qokH9iVEP-ZmdX9NbmO6pdljXCp6SHEOm0"></drupal-render-placeholder></section> Tue, 10 Jul 2018 16:06:32 +0000 mpadrino 21311 at https://ieg.worldbankgroup.org https://ieg.worldbankgroup.org/blog/consulting-big-5-evaluation-criteria-what-got-us-here#comments Rethinking Evaluation- Tracing the Origins of the DAC Evaluation Criteria https://ieg.worldbankgroup.org/blog/rethinking-evaluation-tracing-origins-dac-evaluation-criteria <span>Rethinking Evaluation- Tracing the Origins of the DAC Evaluation Criteria</span> <span><span>kmilhollin</span></span> <span><time datetime="2018-03-27T11:15:10-04:00" title="Tuesday, March 27, 2018 - 11:15">Tue, 03/27/2018 - 11:15</time></span> <section class="comment-section"><h2>Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&1=20111&2=field_comments&3=comment" token="v2VX9hVxfTh-_RBOuvM4JxYKaMzGM_4kex8GqYCuoOo"></drupal-render-placeholder></section> Tue, 27 Mar 2018 15:15:10 +0000 kmilhollin 20111 at https://ieg.worldbankgroup.org https://ieg.worldbankgroup.org/blog/rethinking-evaluation-tracing-origins-dac-evaluation-criteria#comments Rethinking Evaluation - Where to Next? https://ieg.worldbankgroup.org/blog/rethinking-evaluation-where-next <span>Rethinking Evaluation - Where to Next?</span> <span><span>kmilhollin</span></span> <span><time datetime="2017-12-05T11:58:51-05:00" title="Tuesday, December 5, 2017 - 11:58">Tue, 12/05/2017 - 11:58</time></span> <section class="comment-section"><h2>Comments</h2> <article data-comment-user-id="0" id="comment-4186" class="js-comment"><mark class="hidden" data-comment-timestamp="1512575209"></mark><footer><article></article><p class="author">Submitted by <span>Jindra Cekan PhD</span> on Wed, 12/06/2017 - 07:15</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/4186#comment-4186" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/4186#comment-4186" class="permalink" rel="bookmark" hreflang="en">Dear iIEG- thank you so much…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Dear iIEG- thank you so much for this discussion. It is wonderful that an update of the DAC criteria is being contemplated and possibly even undertaken. I would encourage there to be a deeper look at sustainability definitions, expectations, and methods, especially in light of evaluating the SDGs and post project impacts. I look forward to fora that are open to developing country input especially valuing voices of our participants and partners in shaping such evaluation criteria.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=4186&1=default&2=en&3=" token="6-6beB7QN5EJZQwuaEtjsRsNP71uQ3ei7id8t-w9hTk"></drupal-render-placeholder></div> </article><article data-comment-user-id="0" id="comment-4196" class="js-comment"><mark class="hidden" data-comment-timestamp="1512746007"></mark><footer><article></article><p class="author">Submitted by <span>Natalia Kosheleva</span> on Thu, 12/07/2017 - 01:47</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/4196#comment-4196" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/4196#comment-4196" class="permalink" rel="bookmark" hreflang="en">Dear Caroline, thank you…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Dear Caroline, thank you very much for this blog.<br /> This summer I was facilitating consultations on the Sri Lanka Equity-Focused and Gender-Responsive Evaluation Guidelines. And we had a very interesting discussion about DAC criteria. The consensus that emerged from this discussion was that DAC criteria are still relevant, but they should be adapted by infusing into them the file pillars of the SDGs Agenda: people, planet, peace, prosperity and partinership.<br /> So we came up with revised definitions (applicabe in Sri Lanka) which are as follows:</p> <p>Relevance:<br /> Extent to which the intended and actual outcomes of an intervention are consistent with:<br /> • the needs of all sub-groups of beneficiaries, including those most vulnerable and prone to marginalization;<br /> • Sri Lanka national SDGs related to all five pillars of the 2030 Agenda (people, planet, prosperity, peace and partnership);<br /> • Sri Lanka national policies;<br /> • International human rights and gender equality treaties and conventions ratified by Sri Lanka.</p> <p>Effectiveness:<br /> Extent to which an intervention:<br /> • has achieved similar levels of intended outcomes for all sub-groups of beneficiaries, including those most vulnerable and prone to marginalization;<br /> • has ensured that none of the target beneficiaries was left behind.</p> <p>Efficiency:<br /> Extent to which intervention has ensured that all resources/inputs (funds, expertise, time, etc.) were allocated and used economically and in timely manner to achieve similar levels of intended outcomes for all sub-groups of beneficiaries, including those most vulnerable and prone to marginalization.</p> <p>Sustainability:<br /> Extent to which the benefits from an intervention for all sub-groups of beneficiaries, and especially for those most vulnerable and prone to marginalization, are likely to continue after all intervention activities have been completed.</p> <p>Impact:<br /> Extent to which an intervention has contributed to or constrained the progress towards Sri Lanka national SDGs related to all five pillars of the 2030 Agenda (people, planet, prosperity, peace and partnership).</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=4196&1=default&2=en&3=" token="AJpzBNw_w5U4ERIhxPLXqoiYyHYNDMIQM0SFDKpUJXg"></drupal-render-placeholder></div> </article><article data-comment-user-id="0" id="comment-4201" class="js-comment"><mark class="hidden" data-comment-timestamp="1512746007"></mark><footer><article></article><p class="author">Submitted by <span>bojan radej</span> on Thu, 12/07/2017 - 05:39</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/4201#comment-4201" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/4201#comment-4201" class="permalink" rel="bookmark" hreflang="en">Evaluation of synergies,…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Evaluation of synergies, based on Leopold matrix, translated into an input-output matrix, then reworked into a correlation matrix. The procedure developed by Slovenian Evaluation Society (2006-2017). Our paper: "Apples and Oranges: Synthesis<br /> without a common denominator", 1/7(2014), <a href="https://www.dlib.si/details/URN:NBN:SI:DOC-LEB7LI9O/?query=%27contributor%3DRadej%2C+Bojan%27&pageSize=25">https://www.dlib.si/details/URN:NBN:SI:DOC-LEB7LI9O/?query=%27contribut…</a>. New version of the text available only on demand (bradej at gmaildotcom).</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=4201&1=default&2=en&3=" token="pdLmPKfIZpMFcYSmA3l-F8XhywELZbhVwCKNMWZyCSw"></drupal-render-placeholder></div> </article><article data-comment-user-id="0" id="comment-4516" class="js-comment"><mark class="hidden" data-comment-timestamp="1515420549"></mark><footer><article></article><p class="author">Submitted by <span>Yves Bernard, …</span> on Sat, 01/06/2018 - 14:45</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/4516#comment-4516" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/4516#comment-4516" class="permalink" rel="bookmark" hreflang="en">Thank you for sharing your…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Thank you for sharing your key considerations.<br /> One aspect that could be considered for the next refined version of evaluation criteria is the "outside world". I mean that this material can be (and should be more) used by the community that is not in international development and not directly in evaluation. Keeping criteria as simple and clear as possible will allow you to share them with non-international project managers (who are following PMI or PRINCE2 guidance, not DAC) to encourage them to have more assessment as part of a normal process (in the planning, executing or closing phases).<br /> All this simplification/clarification of evaluation criteria should be communicated to encourage more communities/industries to use them (at the very least in planning their work, and ideally in assessing it). In the end, this could at least add some integrity in their approach.<br /> Please use the opportunity of this renewed/refined version of the criteria to communicate them to non-specialists.<br /> Wishing you success with the next steps.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=4516&1=default&2=en&3=" token="n0Rjam_yQy1-b7P8KbRVuecjGSM2ql4OBwQVtGAOHgg"></drupal-render-placeholder></div> </article><article data-comment-user-id="0" id="comment-4526" class="js-comment"><mark class="hidden" data-comment-timestamp="1515594985"></mark><footer><article></article><p class="author">Submitted by <span>Jindra cekan</span> on Wed, 01/10/2018 - 05:28</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/4526#comment-4526" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/4526#comment-4526" class="permalink" rel="bookmark" hreflang="en">Dear IEG and Ms. Heider -…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Dear IEG and Ms. Heider - Regarding the DAC criteria revisions, I would be delighted if as an industry we re-examined and re-prioritized 'sustainability'.<br /> How I long to have substantive data on #WhatWorks And #ForHowLong :)!<br /> If you look at the results here on post project sustained impact evaluation: <a href="http://valuingvoices.com/cant-wait-to-learn-from-post-project-sustainability-evaluation-if-not-why/">http://valuingvoices.com/cant-wait-to-learn-from-post-project-sustainab…</a> and here on how hard it is to find methodologically clear results post project (Note: finding only 1 World Bank substantive evaluation in your PPARs that asked participants what still stood after closeout, much less lessons learned): <a href="http://valuingvoices.com/building-the-evidence-base-for-post-project-evaluation-a-report-to-the-faster-forward-fund/">http://valuingvoices.com/building-the-evidence-base-for-post-project-ev…</a>. Otherwise it all seems like beautiful #WhatWorks words without substance to our participants and partners who dream of sustained impacts long after we all leave. Thoughts? Jindra</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=4526&1=default&2=en&3=" token="_NocV4Bn4VFfZ3JF3RSN3RYcqLFU-80OFG6BtXEP_pM"></drupal-render-placeholder></div> </article><h2>Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&1=19376&2=field_comments&3=comment" token="_0MtcTMI9mawhlU04dfIJepmSbdT1GEDg2MxXIfElR4"></drupal-render-placeholder></section> Tue, 05 Dec 2017 16:58:51 +0000 kmilhollin 19376 at https://ieg.worldbankgroup.org https://ieg.worldbankgroup.org/blog/rethinking-evaluation-where-next#comments Conversations: the Future of Development Evaluation https://ieg.worldbankgroup.org/news/conversations-future-development-evaluation <div data-history-node-id="18566" class="view-mode-rss ds-1col clearfix"> <section class="comment-section"><h2>Comments</h2> <article data-comment-user-id="0" id="comment-3440" class="js-comment"><mark class="hidden" data-comment-timestamp="1498504892"></mark><footer><article></article><p class="author">Submitted by <span>Naeem Shinwari</span> on Sat, 06/24/2017 - 10:36</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3440#comment-3440" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3440#comment-3440" class="permalink" rel="bookmark" hreflang="en">Thanks for sharing such a…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Thanks for sharing such a great conversation! I am not a very pessimistic person, but I have got a huge doubt on the effectiveness of development cooperation. I think before we think about the effectiness of DAC evaluation framework, I am convinced that we need to rethink the whole development cooperation delivery mechanism. This is because at least during my age, I have not seen any country that has been transformed as result of development cooperation. Development cooperation however has become part of the problem particularly in creating vicious circle of dependency, endemic corruption and a sophisticated beaurocracy that sucks the money of hard work tax payers. Many of our problems are entirely political and I think we need to judge political and social commitments before we think of engaging in development cooperation with any country!</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3440&1=default&2=en&3=" token="_voOQfiPM6JcE5loGLTqgnWeoGediz6QltOTVKf4WCw"></drupal-render-placeholder></div> </article><article data-comment-user-id="0" id="comment-3462" class="js-comment"><mark class="hidden" data-comment-timestamp="1499958402"></mark><footer><article></article><p class="author">Submitted by <span>Ridley Nelson</span> on Thu, 07/13/2017 - 10:50</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3462#comment-3462" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3462#comment-3462" class="permalink" rel="bookmark" hreflang="en">Over some 45 years with the…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Over some 45 years with the Bank, and I forget how many with IEG, the main change I have seen in projects is a shift from top-down simple planned projects, setting objectives and ensuring that activities are designed to meet them efficiently, to messier projects which attempted to follow a more organic approach. Community development projects are the obvious case. The need for such a shift was best summed up in a book called Redesigning Rural Development written long ago in 1982 by Johnston and Clark. They believed that rural development projects should be handled more like an Eskimo/Inuit whittling a piece of bone. "He carves a little not quite knowing what he is making, exploring the bone for its texture, faults and potential. He pauses, examines, carves a bit more ... Finally, a smile of recognition: 'Hello seal, I wondered if it might be you'. Problem posed and resolved all in the same process ..." This was a fine concept but how to make it work? And how would one evaluate?<br /> I see no merit in abandoning the three main criteria of relevance, efficacy and efficiency. But the way Bank operations plan and implement against those criteria, and the way we evaluate them, needs to accommodate such iterative, interactive, approaches and be more generally flexible. But this is easier set out in such platitudes than turned into a manageable practice. One cannot have criteria that are so malleable through alternative interpretations that nobody can be held accountable.<br /> We also need to focus on the highest priorities in evaluation. I my view, the biggest weakness in evaluation in all agencies is, and always has been, the tricky process of turning the evidence into useful written lessons. And a part of that problem is that we have never developed any sets of evolving lessons for different sectors that might be built upon with the findings of each new evaluated project and used as a measuring stick. We rarely, if ever, say, "This is what was found in this project case and it is similar to projects x, y and z and in countries of types a and b under circumstances 1, 2 and 3." In other words, we have no constructed learning pyramids, just a flat expanse of plain onto which we randomly throw the odd interesting rock or two.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3462&1=default&2=en&3=" token="fOkaPI1A2bUSZOJym00blENrAzv0fpZaJoEDnjbPm90"></drupal-render-placeholder></div> </article><article data-comment-user-id="0" id="comment-3470" class="js-comment"><mark class="hidden" data-comment-timestamp="1500385687"></mark><footer><article></article><p class="author">Submitted by <span>Dale Hill</span> on Tue, 07/18/2017 - 01:19</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3470#comment-3470" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3470#comment-3470" class="permalink" rel="bookmark" hreflang="en">I was involved in several…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>I was involved in several recent exercises to review the OECD-DAC criteria and their applicability to economic and social development endeavors. These adaptations were timely and needed. Note that I use the word "endeavors" to encompass not only the donor-financed "traditional projects" for which the OECD-DAC criteria were originally designed, but also longer term humanitarian efforts, multidonor partnership programs, social entrepreneurship ventures, advocacy campaigns, networking efforts and communities of practice, etc. The adapting of the original OECD-DAC criteria to humanitarian efforts was needed, but some of them are still hard to apply (those related to human rights). More work can be done there. The adaptation of the OECD-DAC criteria to global and regional partnership programs (GRPPs), in which I was involved with a team at World Bank Independent Evaluation Group (IEG) in 2007, was an important addition, adding the importance of evaluating governance, and refining the "sustainability" concept which is too limiting, addressing as it does mainly traditional donor-funded projects, and leaving out other legitimate long-term objectives such as "sustainability plus, or scaleup" , replicability in different contexts, "building back better to incorporate environmental improvements or resilience, scale-down (if program not working) or devolution of implementation responsibility to local implementers, etc. The other exercise in which I was involved was funded by the Faster Forward Fund-- to look at both general critiques of the OECD-DAC criteria with a fresh eye-- and -- how they might need to be adapted to apply to social entrepreneurship ventures. My findings and suggestions are available on request. The critique captures some of Caroline Heider's points about complexity-- social entrepreneurship ventures are dynamic enterprises, with often conflicting dual objectives-- of profit or financial sustainability and social purpose-- which means the optimal economic equilibrium may be shifting over time. How can this dynamic be captured? To sum, I believe the OECD-DAC criteria served evaluations well for awhile, serving as they did mainly as a guide for more in-depth and flexible evaluation questions-- but there are flaws that, when addressed, would strengthen them further. One of the most obvious-- and that most evaluations just naturally overcame by adding the appropriate evaluation question-- was "Reach" and "Results" -- there was no requirement for a description of actual results -- whether outputs or outcomes--or the factors that improve results and subtract from them. (You could say that impact covered this, but in fact, there were many cases where intermediate outcomes were the most visible and measurable). The other flaw often mentioned is that one doesn't have to base an evaluation on pre-determined objectives (as efficacy calls for). One can have a "goal-less evaluation". A similar flaw is that many times different stakeholders have different objectives, or "stakes" in the program. With the increased emphasis on stakeholder participation, the assessment of "efficacy" (and the other criteria which implicitly also rely on objectives) is limiting. Finally, the concept of sustainability was often misunderstsood, even when applied correctly in the context for which it was meant-- donor-financed projects. The correct interpretation was "sustainability of outcomes beyond the project period after donor financing ceased". (Many mistakenly thought it referred to sustainability of the project activities). But even when applied correctly- the time frame was not well-defined, and the most important question-- how is exit to be managed?-- was not always asked.. Nowadays, with so much emphasis on environmental sustainability, the term simply creates more confusion than is worthwhile. And for the many other "endeavors" we find in the world of economic development today-- sustainability is indeed, only one of many long term objectives.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3470&1=default&2=en&3=" token="-6F1WVmoOQy85vIienrrRPumjo56OESZ_s1VrFU-q-w"></drupal-render-placeholder></div> </article><article data-comment-user-id="0" id="comment-3525" class="js-comment"><mark class="hidden" data-comment-timestamp="1503929085"></mark><footer><article></article><p class="author">Submitted by <span>Alejandro Uriz…</span> on Tue, 08/15/2017 - 18:30</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3525#comment-3525" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3525#comment-3525" class="permalink" rel="bookmark" hreflang="en">It is a very good discussion…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>It is a very good discussion .. with many elements of reflection ... the challenge is how people take advantage of the tools, as a means and not an end .. change the view of the use of criteria CAD and reflect on their improvement, always Is necessary .. since in some cases the same projects use the same criteria and evaluations as a school qualification list and the evaluation itself does not see it as an opportunity for improvement .... but as a requirement within the project. ... and are not giving adequate use to the results of an evaluation and do not make the adjustments that must do or how to enhance the changes generated positively by the intervention and scale in the model ... Another important point is how involving to different authors .. example independent consultants who are applying the evaluations. And how to move these discussions to different regions and in different languages.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3525&1=default&2=en&3=" token="ycho-3o4LjI1hPDHtCdTqmYYNfiY_WWucz00cOBoV2I"></drupal-render-placeholder></div> </article><h2>Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&1=18566&2=field_comments&3=comment" token="omP4GNOaSFHBvQ81IR-A57a24-QX4lCSuDdhWXYDg2E"></drupal-render-placeholder></section></div> Thu, 22 Jun 2017 19:28:01 +0000 kmilhollin 18566 at https://ieg.worldbankgroup.org https://ieg.worldbankgroup.org/news/conversations-future-development-evaluation#comments Evaluability and why it is Important for Evaluators and Non-Evaluators https://ieg.worldbankgroup.org/blog/evaluability <span>Evaluability and why it is Important for Evaluators and Non-Evaluators</span> <span><span>kmilhollin</span></span> <span><time datetime="2017-06-13T10:40:17-04:00" title="Tuesday, June 13, 2017 - 10:40">Tue, 06/13/2017 - 10:40</time></span> <section class="comment-section"><h2>Comments</h2> <article data-comment-user-id="0" id="comment-3431" class="js-comment"><mark class="hidden" data-comment-timestamp="1497636460"></mark><footer><article></article><p class="author">Submitted by <span>Susan Stout</span> on Thu, 06/15/2017 - 15:12</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3431#comment-3431" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3431#comment-3431" class="permalink" rel="bookmark" hreflang="en">Hi Joz,…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Hi Joz,<br /> Great post -- glad to see this topic getting more attention. I've always believed that thinking about 'evaluability' (and how to improve it during the design period) is key to ensuring that evaluations actually contribute to learning -- and also an antidote to my two favorite diseases in our field: 'resultsophobia' -- fear of finding failure or 'less than perfect' results, even though disappointing results are likely -- at least in part -- inevitable; and 'indicatoritis' -- the tendency to focus "M and E" plans on debates about long lists of indicators rather than focusing on 'who is going to use what information to make what decision'....<br /> Thanks,</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3431&1=default&2=en&3=" token="mldGCISFHndHkymuQajKgKlGFVhJyNS6uXBgaDZcVrM"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3432" class="js-comment"><mark class="hidden" data-comment-timestamp="1497638631"></mark><footer><article></article><p class="author">Submitted by <span>Jos Vaessen</span> on Fri, 06/16/2017 - 14:42</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3431#comment-3431" class="permalink" rel="bookmark" hreflang="en">Hi Joz,…</a> by <span>Susan Stout</span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3432#comment-3432" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3432#comment-3432" class="permalink" rel="bookmark" hreflang="en">Thanks Susan for your…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Thanks Susan for your interest and comment. Quite right. And the different ‘itises’ and ‘phobias’ are probably strongly correlated in practice. A culture of making informed choices about evaluation (less ‘evaluitis’) goes hand in hand with a willingness to learn from failure and success (less ‘resultsophobia’) as well as the efficient and effective use of different types of evaluative evidence for learning, accountability and program improvement purposes (less ‘indicatoritis’).</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3432&1=default&2=en&3=" token="kcO17t3mBPiLwTkJi3yBmz2gRZHyGKgLAnd3Xhjbtb0"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3433" class="js-comment"><mark class="hidden" data-comment-timestamp="1497706509"></mark><footer><article></article><p class="author">Submitted by <span>Susan Stout</span> on Fri, 06/16/2017 - 15:20</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3432#comment-3432" class="permalink" rel="bookmark" hreflang="en">Thanks Susan for your…</a> by <span>Jos Vaessen</span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3433#comment-3433" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3433#comment-3433" class="permalink" rel="bookmark" hreflang="en">Yes indeed, as I've long…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Yes indeed, as I've long said -- we should interpret M and E as 'motivation and empowerment'. Most everyone working to deliver development results, in countries as well as among agencies -- wants to be making a difference. They can be motivated by evidence of success and empowered to change things when evidence is suggesting failure -- only if we recognize and value their own perceptions and ensure that they have the authority and flexibility to respond to feedback.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3433&1=default&2=en&3=" token="gmZhO9iVboF-FRDlMaPq9Oqk-D2EbfuI_mtbiZ6WJFc"></drupal-render-placeholder></div> </article></div></div> <article data-comment-user-id="0" id="comment-3435" class="js-comment"><mark class="hidden" data-comment-timestamp="1497883691"></mark><footer><article></article><p class="author">Submitted by <span>rick davies</span> on Sat, 06/17/2017 - 14:03</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3435#comment-3435" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3435#comment-3435" class="permalink" rel="bookmark" hreflang="en">Hi Jos…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Hi Jos</p> <p>Re the third of the three dimensions of evaluability (theory, data and stakeholders), I think there are two facets of the stakeholders that need to be considered. They could be thought of as “demand” and “supply”. Demand is the nature of stakeholders’ interests in the evaluation. Not only what evaluation questions are of interest to whom, but what risks and opportunities are of concern to whom. On the supply side, there is the question of who will be available or accessible by an evaluation team, when and where and under what circumstances. [The latter was the "practicality" dimension in the 2013 DFID Working Paper]</p> <p>The other point that may be of interest is how evaluability judgements relate to choices about what evaluation methods to use. Choices about methods can be seen as sitting in the middle of a triangle formed by the above three issues. One way or another, the choices made about methods to use need to be consistent with what is known about the program theory, with what is known about the data that is or could become available, and what is known about the stakeholders (both demand and supply aspects).</p> <p>regards, rick</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3435&1=default&2=en&3=" token="AqcYBt_um_MSP1OK0Ap6N-UVCUBZUFsDM5DTOPlyz60"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3436" class="js-comment"><mark class="hidden" data-comment-timestamp="1497903965"></mark><footer><article></article><p class="author">Submitted by <span>Jos Vaessen</span> on Mon, 06/19/2017 - 16:25</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3435#comment-3435" class="permalink" rel="bookmark" hreflang="en">Hi Jos…</a> by <span>rick davies</span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3436#comment-3436" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3436#comment-3436" class="permalink" rel="bookmark" hreflang="en">Dear Rick, thanks for your…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Dear Rick, thanks for your thoughtful comments. Regarding your second point, I fully agree that choices regarding methodological design of an evaluation should flow from these (and other) considerations. The problem is that at higher levels of intervention (e.g. global thematic areas, strategies, programs) the evaluability assessment (as a stand-alone exercise or an approach paper) could quickly become rather complicated and costly. This raises questions about the optimal level of (financial and analytical) resources to be devoted to it.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3436&1=default&2=en&3=" token="GxiyR2sp4eIMbfnlrlFFYevWsNprA1nboJn8bGEPX7A"></drupal-render-placeholder></div> </article></div> <h2>Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&1=18589&2=field_comments&3=comment" token="5Yeosi0XIgHKcf36y6LmgH_buQab8SNj5iApJCztjYU"></drupal-render-placeholder></section> Tue, 13 Jun 2017 14:40:17 +0000 kmilhollin 18589 at https://ieg.worldbankgroup.org https://ieg.worldbankgroup.org/blog/evaluability#comments Rethinking Evaluation - Impact: The Reason to Exist https://ieg.worldbankgroup.org/blog/rethinking-evaluation-impact-reason-exist <span> Rethinking Evaluation - Impact: The Reason to Exist</span> <span><span>kmilhollin</span></span> <span><time datetime="2017-05-02T12:58:05-04:00" title="Tuesday, May 2, 2017 - 12:58">Tue, 05/02/2017 - 12:58</time></span> <section class="comment-section"><h2>Comments</h2> <article data-comment-user-id="0" id="comment-3394" class="js-comment"><mark class="hidden" data-comment-timestamp="1494857494"></mark><footer><article></article><p class="author">Submitted by <span>Adam McCarty</span> on Sun, 05/14/2017 - 23:16</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3394#comment-3394" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3394#comment-3394" class="permalink" rel="bookmark" hreflang="en">I would add two other themes…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>I would add two other themes: 1) Use of big data through new technologies. This is connected but different to "enhanced modeling capacities". Vast databases can be built using video or mobile phones. Consider the analysis by marketing people about customers in supermarkets using videos. Could we not do similar to understand schools and health clinics? 2) promote post-evaluations: end-project evaluations are only impact hypotheses as they can only guess at sustainability ("for how long will this water pump last?").<br /> ALSO: Focus on impact evaluations is distracting ("crowding out") more pure research into causation analysis (e.g. In Lao PDR dozens of projects over 20+ years have failed to reduce child dropout rate from primary schools. Why not stop and have a team study the obviously complex problem for six months, and in doing so design experiments to rigorously test? Rather than push through a large loan, which tests a few variant "seem good idea" models via household surveys).</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3394&1=default&2=en&3=" token="Nh4lLbhB3g6GqGKfZtfFbQsONM-YsMTj7fnHqC0m9Qs"></drupal-render-placeholder></div> </article><article data-comment-user-id="0" id="comment-3399" class="js-comment"><mark class="hidden" data-comment-timestamp="1495472078"></mark><footer><article></article><p class="author">Submitted by <span>Peter Eerens</span> on Fri, 05/19/2017 - 19:12</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3399#comment-3399" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3399#comment-3399" class="permalink" rel="bookmark" hreflang="en">Indeed. Dealing with…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Indeed. Dealing with Complexity in Development Evaluation is an excellent resource to reflect on this title, “rethinking evaluation”. A good introduction for the beginner, and an excellent companion work for the experienced evaluator.<br /> But we still need to take more risks to adjust our methods to the reality of a non-linear world. The more we perfect the evaluation, the more we separate the permanent interrogation that should accompany the actor from what she/he is actually undertaking. It is not necessarily true that impact is only at the end of a long temporal sequence. In my experience, as a public health practitioner, impact can be sudden, simultaneous, dramatic, at nanoscale, and too fluid to capture. Pure aesthetics. An experience rarely tackled in evaluation and better captured by poets, artists, or silent communion.<br /> Nature has no purpose and is full of impact. Man is full of purpose and often fails to impact. We still have a lot to learn.<br /> Peter Eerens<br /> Living Health Systems</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3399&1=default&2=en&3=" token="g9mpxytFA_6kYX2iDVUSpqhm6ywEadUVunGpDUijVZY"></drupal-render-placeholder></div> </article><h2>Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&1=18478&2=field_comments&3=comment" token="W9fknzHcDViSB4wCVtduXiNVqXDCa-D3QW7vRETT71M"></drupal-render-placeholder></section> Tue, 02 May 2017 16:58:05 +0000 kmilhollin 18478 at https://ieg.worldbankgroup.org https://ieg.worldbankgroup.org/blog/rethinking-evaluation-impact-reason-exist#comments Rethinking Evaluation - Assessing Design Quality https://ieg.worldbankgroup.org/blog/rethinking-evaluation-assessing-design-quality <span>Rethinking Evaluation - Assessing Design Quality</span> <span><span>kmilhollin</span></span> <span><time datetime="2017-04-11T12:15:49-04:00" title="Tuesday, April 11, 2017 - 12:15">Tue, 04/11/2017 - 12:15</time></span> <section class="comment-section"><h2>Comments</h2> <article data-comment-user-id="0" id="comment-3351" class="js-comment"><mark class="hidden" data-comment-timestamp="1492391589"></mark><footer><article></article><p class="author">Submitted by <span>Anna Gueye</span> on Sat, 04/15/2017 - 23:56</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3351#comment-3351" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3351#comment-3351" class="permalink" rel="bookmark" hreflang="en">Great!!!!</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Great!!!!</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3351&1=default&2=en&3=" token="6Js7UHsSIciYmxxMLyUX1HZuLT3inVizyv2hguATDB0"></drupal-render-placeholder></div> </article><article data-comment-user-id="0" id="comment-3358" class="js-comment"><mark class="hidden" data-comment-timestamp="1492610927"></mark><footer><article></article><p class="author">Submitted by <span>David Thomson Tembo</span> on Wed, 04/19/2017 - 01:20</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3358#comment-3358" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3358#comment-3358" class="permalink" rel="bookmark" hreflang="en">Wonderful article</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Wonderful article</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3358&1=default&2=en&3=" token="I1TJ4dlIvWTemLdOmCRp25O-kE9_gh7w0WTHG19K1O8"></drupal-render-placeholder></div> </article><article data-comment-user-id="0" id="comment-3372" class="js-comment"><mark class="hidden" data-comment-timestamp="1492817403"></mark><footer><article></article><p class="author">Submitted by <span>Charles Y. Ahe…</span> on Fri, 04/21/2017 - 19:09</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3372#comment-3372" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3372#comment-3372" class="permalink" rel="bookmark" hreflang="en">This is an interesting new…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>This is an interesting new realm in conducting evaluation in evaluation. This approach is an in-depth assessment of the basic principles underlying the Evaluation Practice. Clearly its a serious rethinking of whether logic models and results chains are achieved without compromising the real values of what is intended to be delivered.<br /> A very good opportunity to be more introspective in our inquiry during Evaluations.<br /> Great thoughts.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3372&1=default&2=en&3=" token="7TuXFEwq7-tsBRcrEQpdWjmK_DqDIXskg1lYrBN0NsM"></drupal-render-placeholder></div> </article><h2>Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&1=18430&2=field_comments&3=comment" token="Gxv0-xfiGP2CRD2RsEM7p2gfb_co-ePoKiNbYsY3bMs"></drupal-render-placeholder></section> Tue, 11 Apr 2017 16:15:49 +0000 kmilhollin 18430 at https://ieg.worldbankgroup.org https://ieg.worldbankgroup.org/blog/rethinking-evaluation-assessing-design-quality#comments Rethinking Evaluation - What is Wrong with Development Effectiveness? https://ieg.worldbankgroup.org/blog/rethinking-evaluation-development-effectiveness <span>Rethinking Evaluation - What is Wrong with Development Effectiveness?</span> <span><span>kmilhollin</span></span> <span><time datetime="2017-03-21T12:36:23-04:00" title="Tuesday, March 21, 2017 - 12:36">Tue, 03/21/2017 - 12:36</time></span> <section class="comment-section"><h2>Comments</h2> <article data-comment-user-id="0" id="comment-3301" class="js-comment"><mark class="hidden" data-comment-timestamp="1490191539"></mark><footer><article></article><p class="author">Submitted by <span>Michele Tarsil…</span> on Wed, 03/22/2017 - 03:53</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3301#comment-3301" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3301#comment-3301" class="permalink" rel="bookmark" hreflang="en">Dear Caroline,…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Dear Caroline,</p> <p>Thank you very much for your critical reflection on what I believe is the OECD criterion that receives the most attention in evaluation reports. In particular, I appreciated that you brought up the issue of unintended outcomes in evaluation. This is a very dear topic to me: I researched on it with Michael Bamberger and I have been discussing with government officials and NGO staff as well as workshop trainees for the past 10 years. </p> <p>In an effort to better understand why unintended or unexpected outcomes are often invisible in evaluation, I identified two possible causes that I hope could contribute some food for thought for further discussion. Needless to say, my contribution intends to go beyond the common belief that "organizations never want to look bad and therefore have no incentive to ask about occurrences within the scope of their programs that were not adequately envisaged at planning".</p> <p>The first cause is what I called the "RBM-isation of the evaluation culture" in a brief article that I wrote for OECD back in 2014. Otherwise said, too many people still associate evaluation with logical frameworks and believe that its focus should be only on logframe boxes. When I engage in a discussion with planners and manager that embrace this perspective, I always try to explain to them that evaluation is certainly an integrated component of the Results-Based Management Cycle but that it has also some specific features that set it apart from monitoring (e.g., assessing program or policy effects - good or bad- that were contemplated at the planning stage). In such cases, one possible solution would be to clarify to stakeholders (e.g., decision-makers and planners in ministries as well as NGOs) that evaluation function does not simply consist in verifying the compliance between what's found in a project log frame and what happened in reality during implementation. In doing so, using RBM terminology and referring to the RBM cycle as a starting point for discussion might be particularly effective in capacity development programs aimed at professionals who have been exposed to RBM training for over a decade and that are only now starting to familiarize themselves with evaluation. </p> <p>The second cause that I was able to identify in relation to the limited visibility of unintended outcomes in evaluation is the neglect of critical assumptions at planning or the lack of an adequate monitoring of them during implementation. Put simply, critical assumptions, which, by definition, are taken for granted and never measured or assessed, often do not hold and that's exactly why unintended outcomes often come about. In this vein, one possible solution would be to make a greater effort in monitoring program and policy assumptions as much as possible, making sure to allocate adequate budget resources for it during the planning stage.</p> <p>On a more general and final note, I believe that your blog series reflects a new trend in the current evaluation discourse where practitioners are finally taking a stance vis-a-vis some of the dogmas of our profession (e.g. the OECD criteria). Therefore, blogs like yours are extremely useful for enriching the ongoing discussion. Once again, thank you very much for bringing up this very important issue within our community!</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3301&1=default&2=en&3=" token="5-ZvPv39f326om_mO1OkjNFUR7bQ0TcoHoWWk69yV1w"></drupal-render-placeholder></div> </article><article data-comment-user-id="0" id="comment-3302" class="js-comment"><mark class="hidden" data-comment-timestamp="1490365066"></mark><footer><article></article><p class="author">Submitted by <span>Ting Yang</span> on Thu, 03/23/2017 - 12:36</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3302#comment-3302" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3302#comment-3302" class="permalink" rel="bookmark" hreflang="en">Thanks Caroline for another…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Thanks Caroline for another thought-provoking blog. It's indeed key to differentiate terminologies at first instance. As it happens that same term might be used to refer to very different notions or different terms used for same meaning, comparison and conclusion possibly made on such basis becomes problematic. Might it help by differentiating and clarifying the underlying consideration and hypothesis involved in 'effectiveness' or 'impacts'? For example, which specific layer/level of result this ’effectiveness’ might correspond to or what might be its relevant domain along a results chain...</p> <p>Besides, in terms of measuring against preset yardstick as you mentioned and Michele’s comment on the function of logframe in evaluation, I think it also depends on how it gets used. Perhaps we might use logframe mainly as an evaluation framework providing theory/theories of change and facilitating evaluative thinking, rather than a major evaluation tool itself; or if logframe has to be used to fulfil some bureaucratic imperatives as preferred by many, it perhaps can be used together with other tools rather than as the sole or central tool, so that some unintended results might be captured.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3302&1=default&2=en&3=" token="XRp1aNsPa3Ltr-n0ce7zgQegNwNEi2ONrs-ELMDmks0"></drupal-render-placeholder></div> </article><article data-comment-user-id="0" id="comment-3360" class="js-comment"><mark class="hidden" data-comment-timestamp="1492610896"></mark><footer><article></article><p class="author">Submitted by <span>Magda Stepanyan</span> on Wed, 04/19/2017 - 09:14</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3360#comment-3360" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3360#comment-3360" class="permalink" rel="bookmark" hreflang="en">Very intense blog, thank you…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Very intense blog, thank you! Our improved understanding and ability to work with complexity indeed require serious revision of how we shape interventions, implement and evaluate them. I’d think that our focus should be on understanding the ‘risk landscape’ of individuals, households, communities and how we alter them with different interventions. Measuring outcomes therefore should be focused on how we change the evolving dynamic of each particular risk landscape rather than vis-à-vis intended results here and now.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3360&1=default&2=en&3=" token="uHd35uhPG6qhbVz46YRrSDr3vKS30EksPZecKfjwwrw"></drupal-render-placeholder></div> </article><h2>Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&1=18400&2=field_comments&3=comment" token="JNKYrKQDu01C4kGwi-ToZQfiaI1UK-VjSfE2eAS3xzk"></drupal-render-placeholder></section> Tue, 21 Mar 2017 16:36:23 +0000 kmilhollin 18400 at https://ieg.worldbankgroup.org https://ieg.worldbankgroup.org/blog/rethinking-evaluation-development-effectiveness#comments Rethinking Evaluation: Agility and Responsiveness are Key to Success https://ieg.worldbankgroup.org/blog/rethinking-evaluation-agility <span>Rethinking Evaluation: Agility and Responsiveness are Key to Success</span> <span><span>kmilhollin</span></span> <span><time datetime="2017-02-07T14:10:10-05:00" title="Tuesday, February 7, 2017 - 14:10">Tue, 02/07/2017 - 14:10</time></span> <section class="comment-section"><h2>Comments</h2> <article data-comment-user-id="0" id="comment-3235" class="js-comment"><mark class="hidden" data-comment-timestamp="1486562421"></mark><footer><article></article><p class="author">Submitted by <span>Max Merit</span> on Tue, 02/07/2017 - 19:11</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3235#comment-3235" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3235#comment-3235" class="permalink" rel="bookmark" hreflang="en">Incorporating the new…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Incorporating the new criterion into evaluations will add complexity to the art and science of evaluation. For example, while responsiveness and agility may be seen by a subset of the team as more than less of an imperative to achieving success, it may be that key stakeholders disagree on the what, how and when. Also, it would seem natural that a leader be empowered to change course on policies and programs in a timely manner, though there are different incentives and numerous approaches to leadership that will need to also be considered. This implies a need for additional training for evaluators, as well as new requirements for self-evaluation to capture some of the context/thinking while still in the moment.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3235&1=default&2=en&3=" token="XcZhfEMhjTynpwlbncTh7ptkMGop_uKd0gUemQdI9-s"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3239" class="js-comment"><mark class="hidden" data-comment-timestamp="1486644149"></mark><footer><article></article><p class="author">Submitted by <span>Caroline Heider</span> on Wed, 02/08/2017 - 16:35</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3235#comment-3235" class="permalink" rel="bookmark" hreflang="en">Incorporating the new…</a> by <span>Max Merit</span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3239#comment-3239" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3239#comment-3239" class="permalink" rel="bookmark" hreflang="en">Entirely right, it would…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Entirely right, it would take new skills and tools to do this, and to do it in ways to identify where and when systems limit the capacity of leaders to make necessary course corrections, as well as understand when wrong choices were exercised to avoid making the same mistakes over.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3239&1=default&2=en&3=" token="E5p1NzBe0aausi5H802NjdOYzqS2aoTQK5VrRESHoeQ"></drupal-render-placeholder></div> </article></div> <article data-comment-user-id="0" id="comment-3236" class="js-comment"><mark class="hidden" data-comment-timestamp="1486574451"></mark><footer><article></article><p class="author">Submitted by <span>Anna Guerraggio</span> on Wed, 02/08/2017 - 09:25</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3236#comment-3236" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3236#comment-3236" class="permalink" rel="bookmark" hreflang="en">Dear Caroline, agility and…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Dear Caroline, agility and responsiveness are indeed very important, especially in humanitarian and peacekeeping contexts. I find that the argument that development practitioners often advance about ‘objectives no longer valid’ is a bit prosaic, as it very much depends at what level objectives are defined. Expected accomplishments and longer-term goals might remain the same, even in case of a change in environment, while immediate outcomes could significantly vary. That being said, I think it is very important, as you very well put it, that evaluators maintain an equal level of flexibility in assessing the relevance, efficiency, and effectiveness of programs, and acknowledge managers' capacity to adapt to change when it occurred. Equally important, I think that there is room for evaluators to further assess any resistance or 'immunity to change', not only at systemic and institutional level but also at team and individual level. This is where evaluators could join efforts with organizational psychologists to understand better the internal dynamics and inform more sustainable patterns of change.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3236&1=default&2=en&3=" token="nAWCVMd9yf8m3QiRHekUVEOlUlT9bP3Pn-rqrXn7jHs"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3240" class="js-comment"><mark class="hidden" data-comment-timestamp="1486644149"></mark><footer><article></article><p class="author">Submitted by <span>Caroline Heider</span> on Wed, 02/08/2017 - 16:37</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3236#comment-3236" class="permalink" rel="bookmark" hreflang="en">Dear Caroline, agility and…</a> by <span>Anna Guerraggio</span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3240#comment-3240" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3240#comment-3240" class="permalink" rel="bookmark" hreflang="en">Many thanks, Anna, great…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Many thanks, Anna, great suggestions.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3240&1=default&2=en&3=" token="iFq1mSjdPliCTZtWTgeWOqNn4x4P31y6IrhsnCyBA0g"></drupal-render-placeholder></div> </article></div> <article data-comment-user-id="0" id="comment-3237" class="js-comment"><mark class="hidden" data-comment-timestamp="1486574457"></mark><footer><article></article><p class="author">Submitted by <span>Ting Yang</span> on Wed, 02/08/2017 - 10:56</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3237#comment-3237" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3237#comment-3237" class="permalink" rel="bookmark" hreflang="en">Thanks Caroline for raising…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Thanks Caroline for raising this very important but challenging aspect. This resonates with the previous topic about relevance (in the evaluation criteria chain): being responsive and agile in order to remain relevant? This needs an enabling evaluative culture and environment, both at institutional and individual level, as well as methodological diversity and flexibility. However, current donor environment shows apparent preference and requirement for rigid tools such as logframes. A balance needed on manageable degree of responsiveness and agility, which perhaps is most challenging and needs some underlying transformation. Even though challenging as it is, it still needs us to face, discuss and better our responses.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3237&1=default&2=en&3=" token="ISdZVhQoiaON7TRXKO2fp-U4_PRatKhxu7dEOUjdo-o"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3241" class="js-comment"><mark class="hidden" data-comment-timestamp="1486644149"></mark><footer><article></article><p class="author">Submitted by <span>Caroline Heider</span> on Wed, 02/08/2017 - 16:39</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3237#comment-3237" class="permalink" rel="bookmark" hreflang="en">Thanks Caroline for raising…</a> by <span>Ting Yang</span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3241#comment-3241" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3241#comment-3241" class="permalink" rel="bookmark" hreflang="en">Thank you Ting for your…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Thank you Ting for your comments. I agree that the discussion is needed beyond the evaluation community, but am focused on the professional group where I have the greatest responsibility and maybe some influence. Wouldn't it be great if the discussion among evaluators would influence practitioners, including donors, to adopt new ways of working?</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3241&1=default&2=en&3=" token="l-T6wecbWl3KZ8e_PkBcrJCgOmze8TKZvRcrB7USwis"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3258" class="js-comment"><mark class="hidden" data-comment-timestamp="1486660135"></mark><footer><article></article><p class="author">Submitted by <span>Ting Yang</span> on Thu, 02/09/2017 - 09:47</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3241#comment-3241" class="permalink" rel="bookmark" hreflang="en">Thank you Ting for your…</a> by <span>Caroline Heider</span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3258#comment-3258" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3258#comment-3258" class="permalink" rel="bookmark" hreflang="en">Thanks Caroline for your…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Thanks Caroline for your prompt reply. Yes, true. People tend to stay in the familiar and comfort zone, it's great to have such thought-provoking discussions initiated and led by IEG, which certainly benefits not only the evaluation community but also wider range of groups of development practitioners.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3258&1=default&2=en&3=" token="2j0IVyDo1BBs1MbqSlhOgtwWw83HjG09w7lgEOd9fL8"></drupal-render-placeholder></div> </article></div></div> <article data-comment-user-id="0" id="comment-3238" class="js-comment"><mark class="hidden" data-comment-timestamp="1486578930"></mark><footer><article></article><p class="author">Submitted by <span>Zubair Faisal Abbasi</span> on Wed, 02/08/2017 - 13:22</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3238#comment-3238" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3238#comment-3238" class="permalink" rel="bookmark" hreflang="en">I have recently worked on a…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>I have recently worked on a number of assignments on evaluation of humanitarian programmes. After reading this blog post, I feel nowhere the indicators of agility and responsiveness are relevant that humanitarian programmes. Although we try to cover things under the rubric of relevance but the programmes are actually either responsive to the needs or they waste resources. However, in development interventions too. Responsiveness should be measured possibly while understanding "the most significant changes" and while unpacking the logic model. Thanks for the blog and the ideas. I shall try to experiment with it in the next assignment.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3238&1=default&2=en&3=" token="M4f0ZnS9kHoFmtJx_6VyCK1aV-B-3ZSbw6LkR4T8zCU"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3242" class="js-comment"><mark class="hidden" data-comment-timestamp="1486644149"></mark><footer><article></article><p class="author">Submitted by <span>Caroline Heider</span> on Wed, 02/08/2017 - 16:40</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3238#comment-3238" class="permalink" rel="bookmark" hreflang="en">I have recently worked on a…</a> by <span>Zubair Faisal Abbasi</span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3242#comment-3242" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3242#comment-3242" class="permalink" rel="bookmark" hreflang="en">Thank you, Zubair. And…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Thank you, Zubair. And remember to share your experience so that we can all learn from it.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3242&1=default&2=en&3=" token="r2ZCxx5D-tAIMANbGppapTLgtH5kh4JkT0Fo7TOkolM"></drupal-render-placeholder></div> </article></div> <article data-comment-user-id="0" id="comment-3259" class="js-comment"><mark class="hidden" data-comment-timestamp="1487104604"></mark><footer><article></article><p class="author">Submitted by <span>Ehtisham ul Ha…</span> on Tue, 02/14/2017 - 10:41</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3259#comment-3259" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3259#comment-3259" class="permalink" rel="bookmark" hreflang="en">Dear Caroline, Many thanks…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Dear Caroline, Many thanks for sharing another perspective of looking at evidence building agenda with another perspective. I will test these additional questions to include in the upcoming evaluation studies. These are very useful points to learn/understand about the programme management and how often programme teams utilised the evidence to inform decision making. on similar lines , we have developed process indicators to ensure the utilisation of evaluation studies. we will gather more evidence on the utility of this system. lets see how these things will work in future.<br /> I must say that you have great contribution to bring in a fresh and interesting perspective in the development sector. I really find your blogs excellent.</p> <p>Thanks for your support and contribution.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3259&1=default&2=en&3=" token="JsykHuRV9hOUvG-w1Ilh1H4SmwkU_OhDDD3eXkvvoTI"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3260" class="js-comment"><mark class="hidden" data-comment-timestamp="1487181947"></mark><footer><article></article><p class="author">Submitted by <span>Caroline Heider</span> on Tue, 02/14/2017 - 15:46</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3259#comment-3259" class="permalink" rel="bookmark" hreflang="en">Dear Caroline, Many thanks…</a> by <span>Ehtisham ul Ha…</span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3260#comment-3260" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3260#comment-3260" class="permalink" rel="bookmark" hreflang="en">Dear Ehtisham, many thanks…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Dear Ehtisham, many thanks for the feedback and for the details on how you are evolving your evaluation practices. Looking forward to hearing more about it.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3260&1=default&2=en&3=" token="oQFDT8POfrDu1M71jI1PhuhPLS1v-1vfv9MmT11M3UU"></drupal-render-placeholder></div> </article></div> <article data-comment-user-id="0" id="comment-3288" class="js-comment"><mark class="hidden" data-comment-timestamp="1489682727"></mark><footer><article></article><p class="author">Submitted by <span>Lanre Rotimi</span> on Thu, 03/16/2017 - 02:32</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3288#comment-3288" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3288#comment-3288" class="permalink" rel="bookmark" hreflang="en">Rethinking Evaluation should…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Rethinking Evaluation should be Dynamic Process. Caroline has in the series raised serious issues of serious business that deserve the serious attention of Policy Makers and Decision Makers within and beyond Evaluation. In "Have we had enough of REEIS" Blog article, Hans (DAC OECD) historical insight shed light on priorities and direction moving forward this dialogue in Global Interest. </p> <p>Hans contribution underline urgent need to take applied history more seriously and to better appreciate that in the first 50 years of International Development Cooperation (1960-2009) the overarching lessons learnt is that no lessons have been learnt hence failure to build bridge between lessons learning and lessons forgetting was re-occurring decimal. As the second 50 years of International Development Cooperation reach 8 years these flaws and failures persist. This is scar on the conscience of relevant authorities - World Leaders, Regional Leaders in each of the 5 Continents and National Leaders in each of 306/193 UN Member States</p> <p>The SDG which applies equally to all North and South Countries, unlike the MDG which applied to South Countries only underline need for National Leaders and World Leaders on UN Member States Governments: Executive, Legislature, Judiciary at all tiers; UN System: UNO, WBG, IMF Entities including IEG-WBG; CSOs/NGOs; Farmers and Processor Organizations; Private Sector: Micro, Small, Medium, Large, Multinational Enterprises; Academics and Researchers; Internal Consultants and External Consultants to individually and jointly address Rethinking Evaluations real and complex problems on the ground at Community, Sub-national, National, Sub-regional, Regional and Global levels. </p> <p>2017 is Year 2 of Implementation and 1st quarter end in few days, yet there is no evidence that National Leaders and World Leaders are seriously committed towards addressing Rethinking Evaluation real and complex problems on the ground in each Community, each of 306/193 UN Member States, each of 5 Continents and Worldwide.</p> <p>Logframe has been identified as part of the problem. Is this really the case? Again going back to history. Hellmut Eggers at EC Evaluation created Project Cycle Management, PCM Benefits focused Approach to Evaluation in 1987. He retired in 1993 before the idea could firmly take root. Although PCM Approach is widely used Worldwide, what is being used is far from Original PCM. Original PCM answers many of the questions that this dialogue is grappling with. </p> <p>Working independently Lanre Rotimi created Policy, Program, Project Cycle Management and Comprehensive Systems Reform, 3PCM and CSR Benefits focused Approach to Development, M & E, Performance Management (Service Delivery), Procurement and Human Rights in 1993. 3PCM and CSR is Significant Improvement on PCM. In March 2009 Hellmut and Lanre Versions of PCM were merged into the 3PCM Version of PCM.</p> <p>3PCM has 4 Principles, 4 Instruments / Tools corresponding to each Principle two of which are ToR and Logframe; 4 Practices and a Database. It is pertinent to note that a major Evaluation challenge is an Evaluation having a different ToR from the ToR of the Policy / Program/ Project being Evaluated.</p> <p>The point made by Hans underline urgent need National Leaders and World Leaders to adopt a One Worldwide Approach to Evaluation that is a Common and Systemic Approach known and practiced at Community to Global levels in each of 306/193 UN Member States and not a one-cap-fit-all Approach. </p> <p>The points made in Caroline and many contributors to this dialogue underline urgent need for National Leaders and World Leaders to adopt a One Worldwide Approach to Development, Diplomacy, Defense, Democracy, Data and Digitization - Research: Research, Planning, Statistics/Data; Implementation; Evaluation: Monitoring, Evaluation; Success: Learning, Results, Success Policy, Program, Project Interventions, 3PIs and 3PIs Training as One.</p> <p>It is our hope that this series will not be about Talking and Thinking but Action and Results as the dialogue progress in grappling with Rethinking Evaluation Challenge in our World as is and not as any Stakeholder - no matter How Powerful wish it to be.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3288&1=default&2=en&3=" token="Sb5LRgJWSCJPndvIBQg6RvoOKoTcDrait5z1ZijL9Qs"></drupal-render-placeholder></div> </article><article data-comment-user-id="0" id="comment-3293" class="js-comment"><mark class="hidden" data-comment-timestamp="1489761178"></mark><footer><article></article><p class="author">Submitted by <span>Lanre Rotimi</span> on Fri, 03/17/2017 - 04:16</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3293#comment-3293" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3293#comment-3293" class="permalink" rel="bookmark" hreflang="en">Hi Caroline, please find…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Hi Caroline, please find attached link to article moving forward points earlier made by Davis and Lanre<br /><a href="http://developmentchangechampions.blogspot.com.ng/2017/03/global-push-to-achieve-sdgs-vision-and.html">http://developmentchangechampions.blogspot.com.ng/2017/03/global-push-t…</a></p> <p>Points raised highlight Acid Test of Credibility of this IEG Initiative - How it help to deliver:-<br /> 1. Better Development, Diplomacy, Defense, Democracy, Data and Digitization<br /> 2. Better Trade, Aide, Debts, Anti Corruption, Anti Terrorism and Migration<br /> 3. Better End Hunger, Malnutrition and Poverty<br /> 4. Better Trust, Integrity, Openness and Transparency</p> <p>2nd quarter 2017 Year of Implementation is racing to an end, yet many fundamental issues that ought to have been settled by end 2nd quarter 2015 Year of Implementation are still outstanding. As long as National Leaders and World Leaders do not face new direction and adopt new priorities, these information, research and knowledge gaps will remain re-occurring decimals. Allowed to occur the ultimate consequences for over 2 billion poor the UN System: UNO, WBG, IMF serve could be catastrophic.</p> <p>The fundamental issues we consistently raise cannot be wished away. Stakeholders in the 7 blocks identified need to accept to work jointly as they discuss, negotiate and make all necessary arrangements for 2030 Agenda Implementation and Evaluation sustainable success at each specific Community - Global location context.</p> <p>We have thrown up the gauntlet. There is a need for you, other authors and Contributors to genuinely demonstrate "Walk Your Talk". Will you and others pick up the gauntlet?</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3293&1=default&2=en&3=" token="-UA46LPmEJJZQ4P9cSNua8ROIj8n7djw2rGaau5ngTI"></drupal-render-placeholder></div> </article><h2>Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&1=18341&2=field_comments&3=comment" token="2hHCtMrBotVgl44hwewRTO7mt3xwkvyYKraKkQk3LHM"></drupal-render-placeholder></section> Tue, 07 Feb 2017 19:10:10 +0000 kmilhollin 18341 at https://ieg.worldbankgroup.org https://ieg.worldbankgroup.org/blog/rethinking-evaluation-agility#comments Rethinking Evaluation— Is Relevance Still Relevant? https://ieg.worldbankgroup.org/blog/rethinking-evaluation-is-relevance-still-relevant <span>Rethinking Evaluation— Is Relevance Still Relevant?</span> <span><span>kmilhollin</span></span> <span><time datetime="2017-01-17T10:54:49-05:00" title="Tuesday, January 17, 2017 - 10:54">Tue, 01/17/2017 - 10:54</time></span> <section class="comment-section"><h2>Comments</h2> <article data-comment-user-id="0" id="comment-3211" class="js-comment"><mark class="hidden" data-comment-timestamp="1484749367"></mark><footer><article></article><p class="author">Submitted by <span>Ting</span> on Tue, 01/17/2017 - 21:20</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3211#comment-3211" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3211#comment-3211" class="permalink" rel="bookmark" hreflang="en">Thanks Caroline for the…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Thanks Caroline for the informative blog and perspective. System thinking was well embedded in those key points through taking into account of diverse perspectives and connections/links etc. In reality and in system thinking terms, how to then make those boundary choices to determine what is relevant and what is not? Perhaps an actor-centered thinking or theory of change might add some value by making explicit major actors involved, their inter-relationship and domain of interests to help identify the focus and priority? For instance, like the actor centered logic of Outcome Mapping -- differentiating various groups of project actors locating in varied project domains (sphere of control, sphere of influence and interest).</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3211&1=default&2=en&3=" token="DHVhvQp8V8DKF5YCxSDhFRUNnJ_NXR9PT5UHLj7EpOM"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3214" class="js-comment"><mark class="hidden" data-comment-timestamp="1484780597"></mark><footer><article></article><p class="author">Submitted by <span>Caroline Heider</span> on Wed, 01/18/2017 - 17:15</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3211#comment-3211" class="permalink" rel="bookmark" hreflang="en">Thanks Caroline for the…</a> by <span>Ting</span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3214#comment-3214" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3214#comment-3214" class="permalink" rel="bookmark" hreflang="en">Many thanks, Ting, for this…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Many thanks, Ting, for this interesting contribution. We will collect a number of suggestions and reflect on what they mean for our work going forward. Until then I hope you keep sharing your ideas.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3214&1=default&2=en&3=" token="VlJXG_oZCT6ekACxrpMpG3jGTGaDR2oF6cYSJmWpYQ8"></drupal-render-placeholder></div> </article></div> <article data-comment-user-id="0" id="comment-3212" class="js-comment"><mark class="hidden" data-comment-timestamp="1484749367"></mark><footer><article></article><p class="author">Submitted by <span>Ashwini Sathnur</span> on Wed, 01/18/2017 - 01:30</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3212#comment-3212" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3212#comment-3212" class="permalink" rel="bookmark" hreflang="en">As evaluators, our focus…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>As evaluators, our focus should also span across the evolution and progress of an intervention's implementation, over a specified duration of time. It signifies that, along with the application of relevance, effectiveness, impact and sustainability as evaluation criteria, there would be a requirement of an addition of "Evolution/ progress" as one more evaluation criteria. This progress would be calculated and measured at the country level or regional level, on the basis of year - on - year increase or decrease in the implementation of the intervention under focus. It would be calculated with the utilization of the mathematical formula, as mentioned below :-</p> <p>Evolution / Progress = { [ Year n's contribution - Year (n - 1)'s contribution ] / [ Year (n - 1)'s contribution ] } * 100%</p> <p>This contribution would be the contribution of the nation or the measured collective contribution of the region, for that particular year (n) and the year (n - 1).</p> <p>Then based on the above deduced values of Evolution/ Progress, countries as well as regions across the world would be ranked. Thus leading to the creation of Rankings based on the evaluation criteria "Evolution/ Progress" !</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3212&1=default&2=en&3=" token="nZ17qupTo4AI2tUARr0NC4Al1mOAIz58LdL29tj6_vQ"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3243" class="js-comment"><mark class="hidden" data-comment-timestamp="1486644149"></mark><footer><article></article><p class="author">Submitted by <span>Caroline Heider</span> on Wed, 02/08/2017 - 16:54</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3212#comment-3212" class="permalink" rel="bookmark" hreflang="en">As evaluators, our focus…</a> by <span>Ashwini Sathnur</span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3243#comment-3243" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3243#comment-3243" class="permalink" rel="bookmark" hreflang="en">Ashwini, as you can see from…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Ashwini, as you can see from my subsequent blog, I agree with you that we need tools and a focus on assessing change over time, and whether adaptation is timely and responsive. Not sure that this can always be calculated as you suggest. But, even a more qualitative approach will be useful.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3243&1=default&2=en&3=" token="WIZF8XDYX_AY42k_oTuuc9tOd_CUlaXacla4tJlfDRk"></drupal-render-placeholder></div> </article></div> <article data-comment-user-id="0" id="comment-3215" class="js-comment"><mark class="hidden" data-comment-timestamp="1485186335"></mark><footer><article></article><p class="author">Submitted by <span>Juha</span> on Sun, 01/22/2017 - 19:47</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3215#comment-3215" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3215#comment-3215" class="permalink" rel="bookmark" hreflang="en">Dear Caroline,…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Dear Caroline,<br /> I fully agree with you and would like to add that for an intervention to be relevant it must make a difference in the development problem we would like to address. To paraphrase a former colleague, all of our projects do good things but whether they have an impact is an entirely different question. Seen from that perspective, relevance must be linked to impact. In fact, your idea about the need to address the key entry points is important.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3215&1=default&2=en&3=" token="4VSJ_-_5YPxrN4-MWBePHlNFvRwrUpPgXb1wjcKGUEY"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3244" class="js-comment"><mark class="hidden" data-comment-timestamp="1486644149"></mark><footer><article></article><p class="author">Submitted by <span>Caroline Heider</span> on Wed, 02/08/2017 - 16:55</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3215#comment-3215" class="permalink" rel="bookmark" hreflang="en">Dear Caroline,…</a> by <span>Juha</span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3244#comment-3244" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3244#comment-3244" class="permalink" rel="bookmark" hreflang="en">Thanks, Juha, and I think…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Thanks, Juha, and I think this is even more relevant for the green agenda that you and your team are working on at The GEF.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3244&1=default&2=en&3=" token="8UCJaxWI4JEMzToJfRf4OSOti4tLZh7cYQ5fo0nJQWk"></drupal-render-placeholder></div> </article></div> <article data-comment-user-id="0" id="comment-3216" class="js-comment"><mark class="hidden" data-comment-timestamp="1485207249"></mark><footer><article></article><p class="author">Submitted by <span>Jindra Cekan, PhD</span> on Mon, 01/23/2017 - 13:29</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3216#comment-3216" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3216#comment-3216" class="permalink" rel="bookmark" hreflang="en">Dear IEG and Caroline- With…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Dear IEG and Caroline- With all due respect, "After nearly 15 years of applying relevance, effectiveness, efficiency, impact, and sustainability as our foundational evaluation criteria" and the Bank still cannot tell us what project outcomes and impacts were standing or newly emerged after the Bank ended its project and programme funding?! We seem to have found that you rarely consult the very participants your President extolls (see <a href="http://valuingvoices.com/ieg-blog-series-part-ii-theory-vs-practice-at-the-world-bank/">http://valuingvoices.com/ieg-blog-series-part-ii-theory-vs-practice-at-…</a>) and you want to shift the focus from evaluating sustained impact (done on less than 1% of all projects) rather than doing far more which is true accountability to those you serve? Help me to understand!</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3216&1=default&2=en&3=" token="Qi488fcDy94wB1NOPlFmL56t5oeoR0U3WuZSVnLIzuU"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3245" class="js-comment"><mark class="hidden" data-comment-timestamp="1486644149"></mark><footer><article></article><p class="author">Submitted by <span>Caroline Heider</span> on Wed, 02/08/2017 - 16:58</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3216#comment-3216" class="permalink" rel="bookmark" hreflang="en">Dear IEG and Caroline- With…</a> by <span>Jindra Cekan, PhD</span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3245#comment-3245" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3245#comment-3245" class="permalink" rel="bookmark" hreflang="en">Jindra, I am confused about…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Jindra, I am confused about your comment, as it doesn't seem to relate to the blog. I also think it is a rather sweeping statement that we do not consult stakeholders and civil society at all. Of course there is always room to do more, but as you would know there are many players and those who are not consulted can be very vocal, while those who have might not.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3245&1=default&2=en&3=" token="jGR770Fhz9dHoSYX5yUBiUgekkqCBjwk4_3eXtouCqs"></drupal-render-placeholder></div> </article></div> <article data-comment-user-id="0" id="comment-3218" class="js-comment"><mark class="hidden" data-comment-timestamp="1485270140"></mark><footer><article></article><p class="author">Submitted by <span>Kevin Billing </span> on Tue, 01/24/2017 - 08:44</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3218#comment-3218" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3218#comment-3218" class="permalink" rel="bookmark" hreflang="en">Interested to read the blog…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Interested to read the blog but can not really comment until I have - so will only comment if I can read it</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3218&1=default&2=en&3=" token="N0kYO0VB824LBUhMlpfb_v12gAq1CQvwxAjqDFuaHlw"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3220" class="js-comment"><mark class="hidden" data-comment-timestamp="1485270488"></mark><footer><article></article><p class="author">Submitted by <span>IEG website ad…</span> on Tue, 01/24/2017 - 10:07</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3218#comment-3218" class="permalink" rel="bookmark" hreflang="en">Interested to read the blog…</a> by <span>Kevin Billing </span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3220#comment-3220" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3220#comment-3220" class="permalink" rel="bookmark" hreflang="en">If you came to this place in…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>If you came to this place in the blog via a link, try scrolling up to the top of the page to be able to view the whole blog :)</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3220&1=default&2=en&3=" token="rUStbMraYLaMzQvB344ptM-qZ8h9UJJuueng1Zuo36c"></drupal-render-placeholder></div> </article></div> <article data-comment-user-id="0" id="comment-3219" class="js-comment"><mark class="hidden" data-comment-timestamp="1485270140"></mark><footer><article></article><p class="author">Submitted by <span>Kevin Billing</span> on Tue, 01/24/2017 - 09:10</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3219#comment-3219" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3219#comment-3219" class="permalink" rel="bookmark" hreflang="en">I liked the vision of adding…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>I liked the vision of adding a LOT MORE to 'relevance' - I'll be interested to follow up on some of the new lines of thinking in evaluation opened up by the article - as a market system development specialist - we have been, for a number of years, been interested in identifying the nodes of intervention that have the biggest chance of influencing whole value chains or inducing systemic changes that create the biggest impact. I agree - we all need to make what we do more relevant.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3219&1=default&2=en&3=" token="w4FMiPHd0kAJnd6TgtZhWykZJVc-WvCtKwYbr7TCcqU"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3246" class="js-comment"><mark class="hidden" data-comment-timestamp="1486644149"></mark><footer><article></article><p class="author">Submitted by <span>Caroline Heider</span> on Wed, 02/08/2017 - 16:59</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3219#comment-3219" class="permalink" rel="bookmark" hreflang="en">I liked the vision of adding…</a> by <span>Kevin Billing</span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3246#comment-3246" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3246#comment-3246" class="permalink" rel="bookmark" hreflang="en">Thanks, Kevin, would be…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Thanks, Kevin, would be interesting to hear more about the experience of the private sector in this regard.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3246&1=default&2=en&3=" token="h91dcrZzOhJsdioBYkifoSDdJjJJM1KLfQOZvJAPOUk"></drupal-render-placeholder></div> </article></div> <article data-comment-user-id="0" id="comment-3228" class="js-comment"><mark class="hidden" data-comment-timestamp="1485898995"></mark><footer><article></article><p class="author">Submitted by <span>Petra</span> on Tue, 01/31/2017 - 15:39</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3228#comment-3228" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3228#comment-3228" class="permalink" rel="bookmark" hreflang="en">I like the thoughts about …</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>I like the thoughts about "is relevance still relevant?". the paradox is: if a project had been relevant for whoever at the start and if it has been successful, it would ideally become obsolete or "irrelevant". how are we to assess that the project is still relevant after it supposedly achieved its objectives?</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3228&1=default&2=en&3=" token="9xM_xvNkwLKruDa5Ujcg4xzrPoKPnXNv1SuO7psT3JI"></drupal-render-placeholder></div> </article><div class="indented"> <article data-comment-user-id="0" id="comment-3247" class="js-comment"><mark class="hidden" data-comment-timestamp="1486644149"></mark><footer><article></article><p class="author">Submitted by <span>Caroline Heider</span> on Wed, 02/08/2017 - 17:06</p> <p class="visually-hidden">In reply to <a href="https://ieg.worldbankgroup.org/comment/3228#comment-3228" class="permalink" rel="bookmark" hreflang="en">I like the thoughts about …</a> by <span>Petra</span></p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3247#comment-3247" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3247#comment-3247" class="permalink" rel="bookmark" hreflang="en">Petra, you raise an…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Petra, you raise an important point. I think it illustrates well the difference between a project that builds a service capacity, and the continued provision of services. For instance, if a project develops the capacities to deliver health services, the project to set these up might no longer be needed/relevant, once they are up and running, but the health services themselves continue to be relevant to the needs of the population. One could imagine another example, outdated by now but useful as illustration effect, a project that build capacities for delivering IT services. Let's assume it aim to develop landlines (anyone remember those?), because that was the technology at the time of approval. By the time the evaluation comes through, the objectives -- to provide affordable IT services to people -- might still be relevant, but the technical solution is not any more.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3247&1=default&2=en&3=" token="r9frIOUsto7cpWvo06be8M7F_AhgCR8s26ze1IM5fEI"></drupal-render-placeholder></div> </article></div> <article data-comment-user-id="0" id="comment-3450" class="js-comment"><mark class="hidden" data-comment-timestamp="1499379862"></mark><footer><article></article><p class="author">Submitted by <span>Zehra</span> on Sat, 07/01/2017 - 16:56</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/3450#comment-3450" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/3450#comment-3450" class="permalink" rel="bookmark" hreflang="en">Thanks for sharing your…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Thanks for sharing your thoughts on this- I have been thinking about this criterion so much, particularly lately. I am inclined to say that, as it stands now (and with usual questions used within it)- relevance is obsolete and does not significantly contribute to the evaluation. This is due to the fact that, in developmental context (even in case government policies are not there or not elaborated), all interventions are 'relevant' as there are so many needs and so many gaps. From that point of view, your suggested points are going in the right direction, though I would for start call it differently to highlight the different angle that what we actually want to assess under relevance. In most cases, even if an evaluation findings point that an intervention has diminished relevance (due to many reasons, including also efficiency of process from programming to contracting to implementation), this cannot be well reflected, due to way the questions are set. Even when evaluators are in charge of setting evaluation questions, there is a tendency to 'go with a flow' and pose the same old questions, meaning that we will get the same set of answers. This results in the fact that, for vast majority of evaluation reports, relevance is always positive. Review of ROM monitoring reports (EC monitoring tool) also shows that if not all, majority of projects are rated with A or (soft) B. This, to me, is a good indicator that something needs to change quickly or we will just continue to do 'lip service' to our clients.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=3450&1=default&2=en&3=" token="5SKHbSNnWrUaE6fhmSRmepIJCQUN2v9TU7PULPnTMF0"></drupal-render-placeholder></div> </article><article data-comment-user-id="0" id="comment-24636" class="js-comment"><mark class="hidden" data-comment-timestamp="1528733837"></mark><footer><article></article><p class="author">Submitted by <span>Justus Kamwesigye</span> on Mon, 06/11/2018 - 04:52</p> <span class="perma-link"> <a href="https://ieg.worldbankgroup.org/comment/24636#comment-24636" hreflang="en">Permalink</a></span> </footer><div> <h5><a href="https://ieg.worldbankgroup.org/comment/24636#comment-24636" class="permalink" rel="bookmark" hreflang="en">1. Looking at relevance at…</a></h5> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>1. Looking at relevance at the evaluation stage is too late. The focus should be on if and how the intervention (project, Program or whatever) was monitoring what was happening its context that should be measured.<br /> 2. Relevance is important in more in design and less in implementation and much less important in evaluation. Analysis of relevance should be done in design and included in the monitoring systems and processes.</p> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=24636&1=default&2=en&3=" token="QLe-mTHrgcsgqspP2JH1U7zEV8EbGx_o_Sly6JudDRM"></drupal-render-placeholder></div> </article><h2>Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&1=9004&2=field_comments&3=comment" token="7W33vJ4_mBtF0AAz2Fb03vsySRGPym2RHmxgDbf7qns"></drupal-render-placeholder></section> Tue, 17 Jan 2017 15:54:49 +0000 kmilhollin 9004 at https://ieg.worldbankgroup.org https://ieg.worldbankgroup.org/blog/rethinking-evaluation-is-relevance-still-relevant#comments