Back to cover

Meta-Evaluation of IEG Evaluations

Chapter 1 | Introduction

Background, Objectives, and Scope

Since 2005, the Independent Evaluation Group (IEG) has been subject to independent external reviews assessing the credibility, utility, and independence of its work.1 To support the next review, a meta-evaluation of IEG evaluations was conducted in 2020–21. More specifically, the purpose of the meta-evaluation was the following:

  • To provide inputs on the quality and credibility of IEG’s evaluations for IEG’s upcoming independent external review, and
  • To provide IEG’s leadership team an external perspective and suggestions on how to improve the quality and credibility of evaluations.

IEG conducts independent evaluations of the World Bank Group’s interventions and processes mainly at three levels of analysis:

  • Major or thematic and corporate process evaluations with a global or regional reach,2
  • Country Program Evaluations, and
  • Project-level evaluations.

The meta-evaluation covered the first category of IEG’s evaluations, programmatic and corporate process evaluations,3 completed between fiscal year (FY)15 and FY19.

Questions

The meta-evaluation was guided by the following questions:

  1. Can the meta-evaluation appraise the quality and credibility of IEG evaluations according to a dedicated assessment framework? How would such a framework be operationalized?4
  2. Which data are required for such an assessment framework?
  3. Which methodological approaches (both standard and broadened) were used in the 28 IEG evaluation reports published between FY15 and FY19? How did the methods used in the evaluation reports compare with what was initially proposed in the Approach Papers guiding the evaluations? Did the evaluations explicitly discuss elements of research design?
  4. What are the results of the in-depth review of 8 selected IEG evaluations?
  5. What do evaluation reports, Approach Papers, and interviews with IEG staff tell us about the use of innovative methods in the context of evaluation in IEG?
  6. What conclusions may be derived from the inventory, in-depth review, and interviews? What suggestions can be made for future IEG evaluations?

Approach

The meta-evaluation relied primarily on a desk review of evaluation reports (and their corresponding Approach Papers) and was complemented by selected interviews. The assessment focused on the credibility of evaluations (excluding utility and independence). More particularly, it focused on aspects of credibility that could be gleaned from the reports and Approach Papers. The assessment framework was developed according to the guidelines of the American Evaluation Association, the Organization for Economic Co-operation and Development’s Development Assistance Committee, and the Evaluation Cooperation Group. It was further supplemented by standards from various professional evaluation societies, selected international development organizations, and applied behavioral and social science research.

The analysis was conducted in three phases. The first phase (inventory stage) focused on the rationale and scope of all 28 IEG evaluations within the universe of evaluations published from FY15 to FY19. The inventory also appraised the evaluation reports and Approach Papers in terms of various research design attributes, the reliability of the evaluation approach, and the use of innovative (also referred to here as broadened) methods. An inventory of core attributes provided insights on credibility, research design, and methodological diversity across all reports in the universe. A combination of manual and automatic content analysis was used to tabulate the prevalence of conventional (standard) and innovative (broadened) evaluative methods, comparing the methods suggested in Approach Papers with those used in the evaluation reports.5

In the second phase (assessment stage), an in-depth review guided by the assessment framework was conducted to assess the quality and credibility of a stratified random sample of eight evaluations. The review assessed evaluations according to their scope and focus, reliability, validity (including construct, internal, external, and data analysis validity), and consistency. Special attention was also given to the use of innovative evaluation and research methods. Finally, the analysis was supplemented with interviews with IEG team leaders and evaluation officers to obtain contextual information on the design and implementation of evaluations within IEG.

The remainder of the report is structured as follows: Chapter 2 presents the assessment framework, outlining the operationalization of concepts and the set of guidance used to assess the various attributes under consideration. The chapter also provides a brief overview of the ways in which the data were collected and analyzed. Chapter 3 describes the output from the inventory exercise, covering 28 IEG evaluations.6 Chapter 4 describes the results of the in-depth review of eight selected IEG evaluations. Chapter 5 elaborates on the use of innovative methods in IEG evaluations, building on insights from the inventory, interviews, and in-depth review of selected evaluation reports and Approach Papers. Chapter 6 draws conclusions and presents some suggestions to IEG.

  1. The previous self-evaluation was conducted in 2015. The 2020 review was postponed as a result of the coronavirus (COVID-19) pandemic. Historically, meta-evaluations can be traced back to the 1960s when evaluators such as Scriven, Stake, and Stufflebeam began discussing procedures and formal criteria of this genre of work. The term “evaluation of the evaluation,” however, was most likely coined by Orata in 1940. A checklist for conducting meta-evaluations can also be found in Scriven (2015).
  2. We use the term programmatic evaluations in this report.
  3. When we use the term IEG evaluation, we refer to the subset of programmatic and corporate process evaluations.
  4. An internal working document on the development of the assessment framework and other guiding templates was prepared for the meta-evaluation.
  5. Conventional (standard) methods included interviews, focus groups, questionnaires, surveys, traditional document analysis, case studies, descriptive statistics, regression analysis, and literature reviews. Innovative (broadened) methods included machine learning, network analysis, geospatial data analysis, social media analysis, process tracing, qualitative comparative analysis, theory layering (including nested theories of change), and (quasi-) experimental methods.
  6. These are programmatic and corporate process evaluations.