2015 is setting a new agenda for development. And by implication for evaluation. 

In an earlier blog Evaluation Beyond 2015: Implications of Financing for Development, I reflected on the International Financing for Development (Fin4Dev) conference that took place in Addis Ababa and what it would take to sustainably finance economic and social progress. 

Now it is time to turn to the upcoming UN General Assembly, where world leaders will come together to adopt a new set of sustainable development goals (SDGs), which seek to build on the earlier Millennium Development Goals (MDGs).  These new goals will underpin a coordinated global effort to end extreme poverty, fight inequality and injustice, and fix climate change. 

For the development community and evaluators in particular, the SDGs present several opportunities and challenges that will have implications for our work. 

The MDGs certainly put measurement, monitoring and to some extent evaluation on the minds of decision-makers and development practitioners. I am excited to see that the SDGs are ahead of the game in this respect. Financing the MDGs were discussed in Monterrey two years after the MDGs were adopted, and the targets associated with the MDGs were shaped over time. By contrast, the Fin4Dev conference in Addis took place before the SDGs were adopted, and the SDGs include a list of targets and indicators for the global community to adopt. 

The other exciting development is the renewed commitment to statistical capacity. A new Global Partnership for Development Data was launched in Addis. As evaluators we, as much as anyone else, often lament the lack of data to underpin our analytical work and assessments. Great commitments were made to strengthening statistical capacity, including the data revolution fed by technology and Big Data. One wish I would have is that whatever data systems are resuscitated or newly developed, they will be intricately interwoven with decision-making processes. Too often, passionate data specialists come up with the "Rolls Royce" of a monitoring system, only to find that those who should use the data are not aware of the system or don't know what to do with its results, because they are not presented in ways that speak the same language. The second wish, if I have another, would be to make sure that data capacities are complemented with evaluation capacity to analyze, triangulate, and interpret data. That would go a long way to help data users make smart decisions, which is what evaluation is all about. 

But, let me turn to the SDGs themselves. They pose a number of challenges to evaluation, three of which I want to discuss here.  

One, development challenges and by implication, the new goals are dynamic. They require constant learning and adaptation, which means evaluative thinking must be embedded into day-to-day conversations and actions. What do I mean by that? Let's take the target of sustainable food production and resilient agriculture practices (target 2.4). Embedded in the target is a recognition of the effects of climate change. Response strategies have, wisely, not been set, as they will have to change depending on context and conditions. Therefore, policies and program design need to include evaluation to test response strategies and use real-time feedback to make necessary changes. Independent evaluation can assess how effective these feedback loops are working as well as take a dispassionate look at results and performance. 

Two, the SDGs are complex - goals and targets are interdependent and changing over time - evaluation (as much as development) needs to build on but grow beyond simple models of causality like logframes or results chains, and push targets to focus on systemic outcomes. The challenge is to find indicators that are measurable but also capture results at higher levels. For instance, MDG 5 on maternal health included a target that focused attention on skilled birth attendance. This was a measureable indicator, though not sufficient - as evaluations have shown that by itself, this measure will not have the desired effects on maternal mortality rates. As Will Allen put it, complex adaptive systems are composed of "components in the system [that]co-evolve through their relationships with other components." This means that typical cause-and-effect analyses need to be replaced with a much finer understanding of multidirectional effects. Once more in Allen' words "indicators of progress in a complex system are better seen as providing a focus around which different stakeholders can come together and discuss, with a view to potentially changing their practices to improve the way the wider system is trending." 

Complexity recognizes that different parts of a system react differently to interventions and might cause each other to change behavior. Establishing plausible relationships between interventions and observed changes is hard enough when assuming linear models; it will be exponentially harder in complex systems. 

Three, embedded in the new development agenda are tensions between a triangle of desired outcomes: growth, prosperity for all, and sustainability. For instance, goals to increase access, economic growth, energy, and agriculture production all compete for the same resource: water. Managing these trade-offs is not easy, as we saw in our evaluation of the World Bank Group's forestry programs that tried to achieve economic, poverty, and environmental goals. Achieving this multitude of goals will necessitate tough decisions. To start with, tools are needed to assess trade-offs; typical cost-benefit analyses could provide a starting point, but equity or natural resources have yet to be valued in ways that they feature in our typical cost-benefit calculations. Likewise, evaluation methods will need to determine whether the right choices were made to achieve possibly conflicting desirable outcomes, and how the different outcomes should be valued.  

The SDGs set the agenda for a better world by 2030 and put forward challenges to development practitioners and evaluators. By taking them on, evaluation can make significant contributions to changing the understanding of development processes and their outcomes.  


Submitted by Amitabha Sarkar on Wed, 09/16/2015 - 03:31

Thanks for the post, interesting and thought provoking indeed! The three mentioned challenges in-fact pose further challange to go for a thought experiment to discuss about the philosophical foundation, ethical premise and practical nitty-gritty of overall evaluation framework. My learning from this post is that the evaluation framework of SDG to be based on the foundation of relative paradigm where the theory of system thinking would be of use to track the process and progress of goals. You mentioned that evaluation framework should be in correspond with evolving context and modified mechanisms to identify and measure the outcomes delivered. There is probably a need for universal understanding of goals (so far seventeen) among the actors since the goals are 'dynamic' and thus might be conceptualised as per the perception and position of respective agencies. It demands the commitment of plurality among the actors involved to work in harmony. Hence, I would like to point out that the success of evaluation will be also dependent on the intention and commitment of primary actors (international community) and implementors (national outfits) instrumental in SDG. Another point which I would like to draw attention is the importance of linkages in CAS (Complex Adaptive System) like SDG initiative. Linkages are acted as regulators in the system and are capable of moderate the system's performance. It is an utmost need to efficiently and effectively identify those linkages which interconnect the various goals. Accordingly, the linkages can be appropriated in the form of diverse indicators and to fix targets. This could be a way to avoid MDG like problem with regards to indicator as pointed by you. Thanks agian for this timely and informative sharing.

Submitted by Ehtisham Ul Hassan on Wed, 09/16/2015 - 06:58

Dear Caroline Many thanks for sharing a very informative analysis. I have been implementing evaluations since last 13 years but never thought about the aspect of trade off. I really learnt that point from the above details. I also agree with your 3 points. I also would like to highlight that there has been very less attention paid to developing a clear Action plan and Monitoring Framework to track progress on the indicators. I think it would have been useful to break down the cumulative targets into the annual targets for countries so that they can consistently track progress on the targets. I think that will also enable them to closely monitor the success of the development goals. Lastly, there is another point about generating evidence to demonstrate that countries achieve sustainability even after the completion of development interventions. This also demands that all key stakeholders including Government departments and INGOs/Civil Society should follow a systematic way of implementing monitoring and evaluation framework and produce quality evidence to demonstrate success on SDGs.

Submitted by Bojan Radej on Thu, 09/17/2015 - 02:19

Ad 2 (complexity of evaluation), for all those concerned in policy impact evaluation with complex approach (Working Papers of Slovenian Evaluation Society, author B. Radej): - "Social Complexity: Operational Definition", http://www.sdeval.si/studijska-komisija/publikacije/581-social-complexity - "Apples and Oranges: Synthesis in Impact Evaluation", http://www.sdeval.si/Publikacije-za-komisijo-za-vrednotenje/Meso-Matrical-Synthesis-of-the-Incommensurable.html. - “Divided we stand”, http://www.sdeval.si/Objave/Divided-we-stand.html Comments invited.

Submitted by Lanre Rotimi on Wed, 09/23/2015 - 06:01

Inspiring article. The Big issue is that SDG How questions has not been answered just as AAAA How questions has not been answered. WBG-IEG is a Central Actor that should help Stakeholders shift focus away from talking and thinking to solutions and action. Can this shift occur without a different type of Advocacy such as that promoted here #DPINGO #post2015 #action2015? The current SDG is Vision and Words without Action. To deliver on the expectations in this article the SDG need to be transformed into Vision and Words with Action. 3 Days of UNSDS cannot achieve this transformation. This explains why our Advocacy is calling for World Leaders to postpone endorsement of SDG till November or December 2015. This way UNSDS kick start global dialogue on fair answer to SDG How questions in ways that effectively link each Community in each of the 193 UN Member States to UN headquarters. In this International Year of Evaluation, it is a puzzle that Institutions like WBG-IEG and Professional Bodies like IDEAS have made minimal contribution to the FfD and Post 2015 Processes and that explains why the AAAA and SDG, at least in the M & E / Follow Up and Review Components have not answered AAAA / SDG How questions in ways that deliver Development Impact and Development Effectiveness in each Community in each of 193 UN Member States. As long as Data Revolution Report 2014, Synthesis Report 2014, AAAA 2015 and SDG 2015 How questions are not effectively and fairly answered, our World will continue to dance around in underdevelopment. Allowed to occur the ultimate consequences could be catastrophic for Citizens in 193 UN Member States including over 2 billion poor Children, Youth, Women, Men and Elders that are in increasing numbers in US, UK and other Developed Countries. Again, UN Family Organization, WBG-IEG, 193 Member States, IDEAS etc have central role to play identifying One Worldwide Approach recommended in Synthesis Report, building Worldwide Solidarity and taking other needful actions required to achieve increasing convergence between AAAA and SDG Vision Intention and Reality in each of 193 UN Member States. These arrangements need to be negotiated and established on time and cannot be left to happen on their own. It will be recalled that had most of the recommendations in MDG Gap task Force Reports, particularly 2010 - 2014 Reports been fully implemented with effective monitoring and evaluation of this implementation the MDG would have achieved much more and MDG Gap task Force Report 2015 will not be repeating essentially the same problems. Should this mistake be reoccurring decimal the more comprehensive SDG is likely to achieve even less success in 2016 - 2030 than MDG achieved in 2000 - 2015. This is avoidable if UN Family Organization including WBG, WBG-IEG and IMF Deliver as One. It will be a great error for World Leaders to go ahead to endorse SDG this September based on forced or failed answer to SDG How questions. The bright prospects of success implementing SDG will be squandered if there is no bridge building between lessons learning and lessons forgetting.

Add new comment

This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

Enter the characters shown in the image.