Facing off: Accountability and Learning - the Next Big Dichotomy in Evaluation?
Are we heading into a big split in the evaluation profession?
It wouldn't be the first or second time that we take this route. In the 1990s we saw extensive and heated discussions about whether quantitative methods trumped qualitative ones, or the other way round. That phase was followed by a decade of debates on whether randomized control trials were the only "true" evaluation of results. A claim that was countered by evaluators committed to other evaluation methods that are more participatory and capture qualitative evidence.
The good side of these debates has been a testing of boundaries of existing methods. By engaging in deep arguments about the value and quality of one method over the other, evaluators were able to put their methods to the test and improve them. Debates like this can be a way to grow and strengthen the evaluation profession and practice.
But anyone who has followed the profession for the last 30 years also knows the expense. Deeply entrenched debates have absorbed a lot of energy about who is right and who is wrong. Investments have been made on which evaluation method will provide the ultimate insights or become the panacea that answers all questions. Today it is clear that no one direction will do.
Instead of this partisan approach, I am a staunch supporter of using a mix of evaluation methods. Each one has its advantages and disadvantages. Combined, and used for the right purpose they can shed more light and generate deeper insights into what happened and why as a result of an intervention, or other factors.
So where is the next dichotomy?
Over the last couple of years, I have heard more and more often that accountability and learning are irreconcilable, that evaluation needs to take a stance to adopt or focus on one or the other.
I am concerned that the dialogue will lead to unnecessary divisions in the evaluation profession rather than to "the day when evaluation will be considered as the intelligence of an organization, the wisdom of a society" as Ian Davies put it in the February 2016 edition of the EES Newsletter.
In my contribution to the same newsletter, I argued that the answer to this question depends on how accountability is exercised. If it is used to blame people for poor performance and shortfalls in results, if it becomes a threat that is used by evaluators in their work, no doubt accountability will trigger defensiveness rather than learning.
But, in my view this kind of definition and behavior is not what accountability is about. And it is not what evaluation is about. At least not in my book.
Instead, evaluation brings together learning and accountability in that we are looking back at projects, programs, policies that have been implemented. The intention is to understand what has happened: did we reach our intended goals? If not, what took us off-course? Should we set different goals, or manage better the implementation process? All of these evaluation questions combine queries that serve accountability - in the sense of whether we delivered what was expected - and learning - in the sense of how can we replicate successful experiences, how can we avoid mistakes that have been made before.
Jacques Toulemonde, in the same edition of the EES Newsletter, points to valid issues along the accountability and learning axis. Accountability for results is affected by the complexity of other intervening factors so that the project designer or manager cannot be directly held accountable. Likewise, he raises the shortcomings of learning - lessons that are called "learned" even if no-one absorbs and implements them. I agree with these points. And with his reiteration of Cheryl Grey's argument that evaluation should aim for accountability for learning.
But, I am equally certain that this is not the end of the debate. I just hope that we use it to constructively explore the boundaries of this (supposed) dichotomy, the associated methodological challenges, and contribute to making the evaluation profession stronger rather than unnecessarily dividing it.
P.S. Readers may also want to read this WEF article, Is your team in 'psychological danger'? The article speaks directly to the issues of blame culture that I highlight above showing why they matter for teams and organizations.