Conflicting Results: Measuring outcomes in situations of conflict
Understanding and measuring the difference between targeted results and overall achievements in fragile situations can be challenging.
Understanding and measuring the difference between targeted results and overall achievements in fragile situations can be challenging.
By: Lauren KellyDaniel Nogueira-BudnyJeff ChelskyIn conflict affected situations, sometimes just “keeping the lights on” can be considered a success. In these situations, important outcomes can be achieved by simply maintaining development gains. But these “successes” can be missed by performance tracking systems that are designed to measure the results of development assistance under normal conditions. For example, only 18% of countries affected by fragility, conflict and violence are on track to meet the Sustainable Development Goals related to basic needs. In these countries, however, there are many tangible and intangible achievements – related to legitimacy, peace, and security – that are not easily captured by typical results frameworks. They can also be difficult to measure.
Two cases illustrate the challenge of understanding and measuring the difference between targeted results and overall achievements, and critical importance of monitoring for both intended and unintended consequences in highly fluid, conflict affected situations.
Sometimes, just the very act of engaging in conflict affected situations may lead to important and positive outcomes. This includes signaling to the domestic and international community that the country is open for business, maintaining public service delivery to the most vulnerable populations, or 'keeping the lights on.'
In 2017, the World Bank provided quick-disbursing budget support to the beleaguered government of Chad. By one measure, the operation delivered poor results: the public wage bill was not reduced by targeted amounts and there were significant delays in ensuring oversight of state-owned enterprises and freezing of tax exemptions. However, the operation is also rightly credited with helping to shore up the country’s fiscal situation in the context of severely deteriorating security conditions along the borders with Nigeria and Cameroon, due to Boko Haram, and fragile situations along the borders with Libya, Central African Republic, and Sudan, to a lesser extent. According to the report on the project’s results, failure to address the fiscal crisis could have translated into a “broader social, security, and humanitarian crisis, as the deteriorating security conditions in Chad could have serious repercussion on the already fragile sub-regional economic and social situation, with potentially very high long-term costs.”
While in Colombia, the conditional cash transfer (CCT) program supported by the World Bank, Familias en Accion, whose outcome was rated ‘satisfactory’ – for achieving school attendance and food consumption increase targets –was also shown to have increased overall violence by insurgents. While results attributable to the CCT were measured and reported on relative to project targets, unintended outcomes were not. In this case, in some municipalities, the project’s anti-poverty subsidies incentivized information sharing with the Government, increasing its territorial control and encouraging the Revolutionary Armed Forces of Colombia (FARC) to turn to increased and more indiscriminate acts of violence to recapture lost territory. Similarly, in the Philippines and elsewhere, development assistance was shown to have paved the way for looting or predation of project resources (see, for example, Berman et al. 2012; Wood 2014), or led to preemptive sabotage to prevent the supposed loss of support that such projects would bring (Crost, Felter, and Johnston, 2014), thereby increasing insurgent violence.
The effectiveness of development interventions is rightly assessed using project results indicators. This is done to ensure that interventions are well designed and implemented so that limited resources are spent efficiently and effectively. However, as suggested by the above examples, when working in conflict-affected states, measuring project level results relative to explicit project objectives can hide the reality of both intended and unintended consequences, positive and negative. Indeed, project development objectives should be realistic and clearly articulated; however, assessing impact or performance through mechanistic use of project ratings can provide a distorted picture.
Sometimes, just the very act of engaging in conflict affected situations may lead to important and positive outcomes. This includes signaling to the domestic and international community that the country is open for business, maintaining public service delivery to the most vulnerable populations, or, as mentioned above, “keeping the lights on.” Furthermore, in situations of conflict, with fragile institutions and fluid power dynamics, sometimes unintended outcomes are achieved alongside project objectives. In such contexts, more so than elsewhere, development partners need to identify and understand conflict drivers and mitigate the risk of exacerbating existing societal cleavages or otherwise stoking the flames of conflict. As such, secondary outcomes need to be captured and assessed alongside the achievement of explicit project objectives or targets.
The Independent Evaluation Group will be addressing many of the questions raised here in an upcoming evaluation of the World Bank’s Engagement in Situations of Conflict. The evaluation was launched to coincide with the rollout of the World Bank’s new Strategy for Fragility Conflict and Violence, and will build on the body of IEG work including World Bank Assistance to Low-Income Fragile and Conflict-Affected States (2014) and World Bank Group Engagement in Situations of FCV (2016).
We look forward to your comments as we conduct this evaluation, the preliminary findings of which will be available later this year.
This blog is part of IEG's Measuring Up series, designed to spark greater knowledge sharing in the development community about what success looks like in Global Development, as well as they ways we measure, assess, and evaluate it. We hope you will share your views and perspectives in the comments to this blog post.
Comments
The challenges of measuring…
The challenges of measuring results in conflict-affected and fragile states are immense, in view of numerous constraints in planning, implementing and monitoring of development interventions. Even if the theories of change or logical frameworks have been appropriately developed to guide interventions, these become outdated in view of rapidly changing environment.The security restrictions affect both the quality and timelines of monitoring and henceforth limit the opportunities for adaptive programme management and necessary adjustments in implementation arrangements. In highly sensitive political context and sensitivities, stakeholders may have divergent and conflicting viewpoints and perception of intended and achieved programme and development results
In order to address the challenges of measuring development results in crisis and fragile situations, organizations should ensure that:
1. Conflict analysis is undertaken not only to inform design of the programmes, but also conducted periodically to assess whether programme strategies and approaches are sensitive to the changes in conflict/fragility settings;
2. Measurement of results need to be conflict sensitive as well. Monitoring and evaluation approaches should be conducted in a manner that "Do No Harm" and take into account ethical considerations, and potential vulnerabilities and risks (e.g. security risk for informants releasing sensitive information; potential implications for vulnerable groups (minority or indigenous population), as key informant; gender-related concerns).
3. Involvement of stakeholders. The full diversity of local stakeholders need to be involved not only in undertaking, but also in designing/planning phase of the evaluation. The development of the scope and methodology of evaluation should be transparent and follow participatory approaches, to ensure that the evaluation benefits from the local views and understanding of the local context. Engagement of stakeholders from different sides of conflict or with different viewpoints on the context is important to capture the linkages between the development intervention and diverse contextual, political and other factors.
Kind regards,
Serdar Bayryyev
Senior Evaluation Officer
UNFAO
Add new comment