The principal question for many has been, 'How do we know if we are achieving the goals?'... Evaluation’s role in relation to this question is rather limited. If, by contrast, one asks the question, 'How do we get there?', then evaluation ... becomes an essential ingredient in the evidence mix.

TWEET THIS

Ever since the international community endorsed the Sustainable Development Goals (SDGs), debates have flourished on the topic. The principal question for many has been, "How do we know if we are achieving the goals?". This largely relates to the choice and definition of indicators and targets, as well as the (potential) availability of data to measure progress.

Evaluation’s role in relation to this question is rather limited. If, by contrast, one asks the question, "How do we get there?", then evaluation (the systematic inquiry into what works, why, under what circumstances, and for whom) becomes an essential ingredient in the evidence mix.

Recently, there has been a lot of interesting work on the role of evaluation in the framework of the SDGs (see, for example, the series of briefing papers on effective evaluation for the SDGs and Evaluation for Agenda 2030). As debates tend to go into different directions, there is scope for developing a structured overview of some of the key evaluative issues and questions around this topic.

The framework we present here is organized around three different levels of analysis: global level, country level and the policy intervention level.

Global Level

The Sustainable Development Goals have been endorsed by UN member states across the globe. While this sends a strong signal for future action, much depends on the extent and nature of international advocacy, dialogue and awareness-raising work and how this influences the actual adoption of the SDGs by different types of institutional actors (e.g. state, private, civil society). There are a number of illustrative questions that can be asked around the effectiveness of the global SDG movement and the adoption of the SDGs by different institutional actors across countries: To what extent are international (intergovernmental) organizations advocating for the SDGs and supporting national actors to embed the SDGs in their work? To what extent are international initiatives on the SDGs effectively influencing awareness, attitudes and behaviors of key actors across countries? To what extent are sustainable development issues and their interlinkages perceived as priorities by different countries? What are the main explanatory factors?

Country Level 

Many of the questions listed above, i.e. those around the adoption of the SDGs by different types of institutional actors, are first and foremost to be posed at the country level (and subsequently synthesized into a global perspective). The issue of country-level adoption of the SDGs can be further unpacked into more specific questions at country level, for example: How relevant are the SDGs (and corresponding indicators and targets) at the country level? To what extent have countries integrated the three pillars of sustainable development (i.e. economic, social and environmental pillars) into national policy (e.g. including national development plans)? To what extent have countries developed coordination mechanisms for integrating SDG-related issues into program design and implementation? To what extent do these mechanisms effectively include private sector and civil society actors?

A second dimension of interest in the context of evaluating the SDGs at country level concerns the nature of the evaluation system at the country level. Relevant questions are for example:  What is the state of institutionalization of evaluation? How is evaluation organized within and across sectors? What are the capacities of evaluation stakeholders (demand and supply side)? What are the opportunities and constraints for conducting cross-sectoral evaluative exercises (e.g. reflecting the interconnected nature of the SDGs)? What are the opportunities and constraints for evaluation to influence decision-making processes?

Finally, any evaluative effort that looks into the effectiveness of policy interventions in relation to the SDGs would depend on data availability at the country level on SDG-related indicators over time. Examples of relevant questions are: What is the availability of data on the indicators of the SDGs at country level? What is the periodicity of data collection? What is the quality of the data? What are the capacities of national organizations to collect relevant data? What are the opportunities and constraints for doing so?

Policy Intervention Level (Policy, Program, Project, or Activity) 

While there are multiple dimensions of evaluative interest with regard to the SDGs at this level, we highlight two key aspects.

Dealing with causality in the context of the SDGs merits particular attention. Which interventions work, and under what circumstances, in terms of contributing to the achievement of SDG-related targets? Well-known debates on causal attribution and the application of rigorous impact evaluation methods remain as important as ever in the context of the SDGs. However, now more than ever, a causal perspective in the context of the SDGs would require paying attention to such issues as the interconnected nature of causal processes, potential trade-offs or win-win situations between two (or more) objectives of a specific intervention (e.g. poverty reduction and environmental conservation), the nature of causal processes affecting SDG-related indicators (which are more often than not non-linear, uncertain, and emergent), and the timing and sustainability of effects. Among other things, the conceptual and methodological implications of the above call for a more complexity-responsive and systemic approach to evaluation (see this blog for further discussion).

A second important evaluative dimension concerns policy coherence. Policy coherence can be conceptualized in different ways but broadly refers to the extent to which interventions complement or reinforce one another. Examples of relevant questions are: To what extent is a particular policy intervention part of a broader integrated policy framework or approach that takes into account the interconnected nature of interventions and the processes of change they may influence? To what extent is there some type of coordination between actors in relation to a particular intervention (or set of interventions)? To what extent are the efforts and objectives of a set of interventions aligned with each other or possibly working at cross purposes?

In Conclusion

To structure our thinking on the role of evaluation in the context of the Sustainable Development Goals we need a multi-level perspective. While there are obviously interlinkages between the levels, at the same time there are particular evaluative dimensions of interest at each level. The framework presented above constitutes a useful starting point to position and connect different ongoing debates on the topic of evaluation and the SDGs. Some of these debates are more nascent than others. This is a topic for a follow-up conversation...

Comments

Submitted by Alexander Dill on Tue, 04/17/2018 - 11:56

Permalink

Dear Jos, dear Stefano, thank you for featuring the subject here! We are researching on the impact and interactions between the SDGs for a while yet. I will link here to two documents we can talk on maybe. The first is the Excel of the Global Index Benchmark where you find the SDGs in column one:
http://commons.ch/wp-content/uploads/Global_Benchmark_SDG_GDP.xlsx
We found entire redundancy: the measurement of the SDGs is currently made to let Sweden confirm Norway to having met the SDGs. We have to change this by finding independent indicators.
The second document is our SDG impact matrix: http://commons.ch/wp-content/uploads/UN_Goals_SDG_Interaction_Matrix.pdf
You can see from this review that the donors are picking up a few SDGs without considering their interaction. Same with cross-country effects: How will you achieve any SDGs e.g. in Afghanistan, Iraq, Syria, Pakistan, Jemen, Palestine, Ukraine, Mali, Burkina Faso without resolving the conflicts?
And how can we connecting these results with your multi-level approach?

Dear Alexander, thanks for your interest in our blogpost. Indeed, there are important interlinkages between the different SDGs. However, these interlinkages are best understood not at the level of the Goals themselves but by looking at the complex reality underneath. Understanding how in a particular context (a community, a region, a country) different societal processes influencing the alleviation of poverty, the conservation of natural resources (and so on) work, how they are interconnected, and how particular policy interventions may influence one and/or more of these processes over a certain period of time is a daunting task that requires resources, expertise and data. Evaluation can offer approaches and methods to understand the causal influence of policy interventions in this complex reality (including the assessment of potential trade-offs or win-win situations between SDG-related outcomes).

A second issue which you raise concerns the multi-level perspective on evaluation and the SDGs. There are multiple interpretations of what ‘multi-level’ means in this context. Our multi-level framework is meant as a structure for thinking about different debates and questions around the role of evaluation in the context of the SDGs. This is different from another interpretation, which is relevant as well, which is the issue of multi-level interaction effects (i.e. what happens locally has an influence on what happens at national/regional/global levels and vice versa).

Submitted by Kim Castelin on Thu, 01/31/2019 - 04:43

Permalink

Dear Jos and Stefano,
Thank you for your cogent reflection on evaluation's role in achieving sustainable development goals.

Perhaps it is implicit in the Policy Level of your framework, but it's important to me to explicitly include participatory approaches with communities and individual participants in development interventions as a critical level in evaluation of the SDGs.
Development participants are often the first to feel the crunch of being pressed by non-coherent policies or projects and having robust evaluation systems that capture rapid feedback are key to identifying when well-meaning changes are undermining advances in other areas. Strategic use of participatory evaluation directly reinforces SDG 16-Peace, Justice, and Strong Institutions by giving power to communicate along institutional organizations internal and external to a nation through participatory evaluation.
Without including development participants in evaluation, many of the non-metric mechanisms and systems which create complexity are 'assumed' out, leaving evaluators no better off in understanding how to fix problems nor possibly replicate successes.

Add new comment