Boldly Going Where No Evaluator has Gone Before - A Framework for the Evaluation of Service Delivery

We adopted a definition of service delivery that places fundamental emphasis on meeting the expectations of citizen beneficiaries.

We recently launched a series of blogs that explore the adequacy of traditional approaches to evaluation, as part of our commitment to innovation, and to developing evaluation models that adequately address complexity. IEG’s recent work on Service Delivery represents one such foray into the unknown, destined to challenge evaluation practice to better enhance development effectiveness.

Prologue

A while back, we decided to build part of our work program at IEG under three Strategic Engagement Areas to help inform choices about what to evaluate, build synergies between individual evaluations, and to enhance engagement with the World Bank Group’s management. Under the sustained services for the poor strategic engagement area, we produced working papers on Behavioral Change (we’ll blog on behavioral change soon) and Service Delivery. The service delivery paper set out to define key concepts and features, and to identify how the World Bank Group supports service delivery. It also set out to develop a framework and analytical protocol to evaluate service delivery as part of three ongoing sector evaluations (urban transport, water and sanitation, and health services).

Take off

Following extensive research into the cross disciplinary literature, as well as engagement with colleagues across various units inside the World Bank Group (such as, the Global Delivery Initiative (GDI), Governance, Social Protection, Education, Development Economics) we adopted a definition of service delivery that places fundamental emphasis on meeting the expectations of citizen beneficiaries.

To echo the World Bank Group President, Jim Kim, we defined service delivery as a series of highly localized actions by agents in public agencies or private enterprises to provide needed goods and services to citizen beneficiaries in a way that meets their expectations.

Accountability to Citizens is an Overarching Service Delivery Principle

What also became apparent is the importance of a systems approach to service delivery. We found that service delivery is much more than any given delivery mechanism, or interface, between the service deliverer and the citizen beneficiary. In fact, effective service delivery is always subject to culture, and its design spans an appreciation, as our evaluation framework identifies, of enabling conditions, ensuring appropriate service delivery inputs, and implementation mechanisms, with a view to securing relevant outputs and outcomes.

That sounds like a tall order – and it is! But, it’s also clear that no one entity, let alone single project, can be responsible for ensuring the integrity of all parts of a complex service delivery effort. That said, the framework assumes that any intervention has to be explicitly aware of its place in the delivery system, and of the extent to which all other moving parts are in motion, recognizing that a flaw in any one part of the system is likely to affect the whole. That’s just the nature of systems.

Beyond Definitions

We’re currently testing our approach on a number of fronts. First, we’re analyzing a sample of project appraisal documents that form part of the portfolios under scrutiny for the urban transport and water and sanitation evaluations. Initial findings are interesting, particularly regarding the limited extent to which all-important accountability and feedback mechanisms are evident, something that will be of interest to colleagues working on the forthcoming Citizen Engagement evaluation. We’re also planning a similar, but more developed engagement with the evaluation of health services. We’ll share more on this once we’ve fully crunched the data.

And we’re involved in an exciting learning engagement with our partners at the GDI through which we aim to further socialize our approach with colleagues across the World Bank Group that work on sectors with significant public service delivery focus such as transport, water and sanitation, education, and health and nutrition services. We’ll also share insights to feed into supports and resources made available to operational staff and mangers. Finally, we will deliver a session in May as part of Human Development Week at the World Bank Group.

So, a lot happening, and a lot to deliver! Maria Gonzalez de Asis will blog in a few weeks to fill you in on developments under the GDI, and we’ll be back later to let you know how things are going at our end. In the meantime, please share your experience with service delivery in development and/or evaluation, as well as your views on our work to date.