Evaluation Beyond 2015: Implications of the SDGs for Evaluation
2015 is setting a new agenda for development. And by implication for evaluation.
In an earlier blog Evaluation Beyond 2015: Implications of Financing for Development, I reflected on the International Financing for Development (Fin4Dev) conference that took place in Addis Ababa and what it would take to sustainably finance economic and social progress.
Now it is time to turn to the upcoming UN General Assembly, where world leaders will come together to adopt a new set of sustainable development goals (SDGs), which seek to build on the earlier Millennium Development Goals (MDGs). These new goals will underpin a coordinated global effort to end extreme poverty, fight inequality and injustice, and fix climate change.
For the development community and evaluators in particular, the SDGs present several opportunities and challenges that will have implications for our work.
The MDGs certainly put measurement, monitoring and to some extent evaluation on the minds of decision-makers and development practitioners. I am excited to see that the SDGs are ahead of the game in this respect. Financing the MDGs were discussed in Monterrey two years after the MDGs were adopted, and the targets associated with the MDGs were shaped over time. By contrast, the Fin4Dev conference in Addis took place before the SDGs were adopted, and the SDGs include a list of targets and indicators for the global community to adopt.
The other exciting development is the renewed commitment to statistical capacity. A new Global Partnership for Development Data was launched in Addis. As evaluators we, as much as anyone else, often lament the lack of data to underpin our analytical work and assessments. Great commitments were made to strengthening statistical capacity, including the data revolution fed by technology and Big Data. One wish I would have is that whatever data systems are resuscitated or newly developed, they will be intricately interwoven with decision-making processes. Too often, passionate data specialists come up with the "Rolls Royce" of a monitoring system, only to find that those who should use the data are not aware of the system or don't know what to do with its results, because they are not presented in ways that speak the same language. The second wish, if I have another, would be to make sure that data capacities are complemented with evaluation capacity to analyze, triangulate, and interpret data. That would go a long way to help data users make smart decisions, which is what evaluation is all about.
But, let me turn to the SDGs themselves. They pose a number of challenges to evaluation, three of which I want to discuss here.
One, development challenges and by implication, the new goals are dynamic. They require constant learning and adaptation, which means evaluative thinking must be embedded into day-to-day conversations and actions. What do I mean by that? Let's take the target of sustainable food production and resilient agriculture practices (target 2.4). Embedded in the target is a recognition of the effects of climate change. Response strategies have, wisely, not been set, as they will have to change depending on context and conditions. Therefore, policies and program design need to include evaluation to test response strategies and use real-time feedback to make necessary changes. Independent evaluation can assess how effective these feedback loops are working as well as take a dispassionate look at results and performance.
Two, the SDGs are complex - goals and targets are interdependent and changing over time - evaluation (as much as development) needs to build on but grow beyond simple models of causality like logframes or results chains, and push targets to focus on systemic outcomes. The challenge is to find indicators that are measurable but also capture results at higher levels. For instance, MDG 5 on maternal health included a target that focused attention on skilled birth attendance. This was a measureable indicator, though not sufficient - as evaluations have shown that by itself, this measure will not have the desired effects on maternal mortality rates. As Will Allen put it, complex adaptive systems are composed of "components in the system [that]co-evolve through their relationships with other components." This means that typical cause-and-effect analyses need to be replaced with a much finer understanding of multidirectional effects. Once more in Allen' words "indicators of progress in a complex system are better seen as providing a focus around which different stakeholders can come together and discuss, with a view to potentially changing their practices to improve the way the wider system is trending."
Complexity recognizes that different parts of a system react differently to interventions and might cause each other to change behavior. Establishing plausible relationships between interventions and observed changes is hard enough when assuming linear models; it will be exponentially harder in complex systems.
Three, embedded in the new development agenda are tensions between a triangle of desired outcomes: growth, prosperity for all, and sustainability. For instance, goals to increase access, economic growth, energy, and agriculture production all compete for the same resource: water. Managing these trade-offs is not easy, as we saw in our evaluation of the World Bank Group's forestry programs that tried to achieve economic, poverty, and environmental goals. Achieving this multitude of goals will necessitate tough decisions. To start with, tools are needed to assess trade-offs; typical cost-benefit analyses could provide a starting point, but equity or natural resources have yet to be valued in ways that they feature in our typical cost-benefit calculations. Likewise, evaluation methods will need to determine whether the right choices were made to achieve possibly conflicting desirable outcomes, and how the different outcomes should be valued.
The SDGs set the agenda for a better world by 2030 and put forward challenges to development practitioners and evaluators. By taking them on, evaluation can make significant contributions to changing the understanding of development processes and their outcomes.