Apply mechanisms for strategic identification and prioritization of IEs at the World Bank and IFC to balance learning and results measurement objectives: - At the World Bank, develop a strategic approach to guide IE selection across sectors and regions. Introduce guidelines to implement the strategic framework building on a framework developed in the context of IDA 16.
1. Application of a strategic approach to identify IEs In the World Bank: Rapid growth in IE activity in the World Bank has been accompanied by efforts to improve strategic selection and coordination of IEs, led by DIME and SIEF. There is emerging evidence that IEs initiated under more strategic and better coordinated approaches over the last 3 to 4 years have better quality assurance mechanisms, greater engagement of clients and project teams, and are aligned with a broader set of sector and knowledge priorities. However, in some sectors, particularly non-HD sectors, a strategic framework for IE selection from an operational and knowledge perspective is less prevalent. These IEs were more often selected opportunistically, often due to skills and funding availability, and ease of evaluation The importance of strategic IE selection has been further underscored during the IDA 16 replenishment, in which IDA deputies requested that the World Bank increase the number of IEs and deploy a more strategic approach to selecting projects for impact evaluation.
Original Response: WB: The IE strategic selection framework developed as part of the IDA16 commitments, aims at increasing the coverage and learning from IDA's program of impact evaluations. With regard to IBRD, the DIME steering group will continue to support Sector Boards and thematic groups in identifying priorities and in compiling MIC IE financed by others, notably partner countries themselves. MICs serve an important role in IE communities of practice, generating lessons applicable in both MICs and IDA countries.
Specific Action: Develop and implement a strategic approach to guide IE selection across sectors and region in IDA countries
Baseline: Strategic selection approach adopted ; 44 IDA project selected for FY12-13as a balanced sample across regions and sectors representing 10 percent of IDA projects;
Indicator: periodic reports to IDA deputies
ï¾ 22 IDA projects to be selected for FY14 using the strategic selection framework
Timeline: June 2013 for FY14 selection
IEG welcomes and gives highest commendation to Management for their recent commitment to CODE to further boost impact evaluations (IEs) and learning generated from them in a more systematic manner in response to IEG's recent ROESES evaluation. Just as encouraging is the mandate given to OPCS to lead the formation of an IE Working Group to take stock of, learn from, and make strategic use of IEs. It is IEG's sincere hope that the promise of this direction will be realized. Indeed, the structure and mandate of this Working Group is very much in line with IEG's recommendations five years ago and it is heartening to see these actions finally being taken. However, while these promises of future action are certainly desirable and at some future date may well address all the concerns raised in Recommendation 96 (and 98 and 100), IEG notes that these actions were taken more recently than FY16 and so, regrettably, fall outside of the scope of this MAR update.
As in previous years, Management update gives information for three impact evaluation hubs at the World Bank (i2i, SIEF, and the Africa Gender Innovation Lab). This leaves out the IEs from the Health Results Innovation Trust Fund (aside from lumping HRITF IEs in with an aggregate count), the Gender Innovation Labs from the other regions, project-funded IEs (those not receiving support from IE hubs), and DEC (outside of DECIE/DIME). The update also inconsistently gives updates from DIME outside of i2i: Sometimes DIME's non-i2i results are given, but not always.
The fact that the Management update sequentially presents the work of each of these three IE hubs without giving an aggregating summary underscores the perception of fractionalization across the hubs.
The IE hubs included in the update (i2i, SIEF and the Africa GIL) have established a strong selection apparatus. Unfortunately, because there is no update on the selection process of many of the World Bank's IE producers, neither the Bank nor IEG can determine the extent to which a strategic selection approach has been adopted. Even so, though it seems reasonable to believe that these hubs account for a lower bound of 60%, allowing for a Substantial rating but less than the 90% required for a rating of High. The Management revised update asserts that 89% of IE products come from i2i, SIEF, Gender LAB, HIRTF and Jobs(IEG's calculations with the data provided by OPCS indicates that 86% of ongoing IEs come from these groups). Even so, this may be an upper bound. The numbers are drawn from projects with an IE tag and the IE hubs are much more likely to appropriately tag their products and are more likely to subject those products to a strategic selection process than are ad hoc IEs funded by projects which may not be tagged in the system. Management updates have never presented evidence of the selection process of HRITF or Jobs.
Regardless of the de jure strength of these processes, they to continue to result in a portfolio that is imbalanced by sector and regionthe dimensions originally pointed out by IEG 4 years ago. IEG estimates that half of the 18 Global Practices and Cross-Cutting Solutions Areas combine to account for only 15% of the 58 impact evaluation concept notes approved in FY16. Moreover, Africa was disproportionately overrepresented as fully half of the newly-approved concept notes were from that region even though it accounts for one third of the world's poor (PREM estimates). Conversely, South Asia and East Asia and the Pacific continue to be underrepresented: Though the regions combine for more than 60% of the world's extreme poor, it is slated to benefit from less than 14% of the new IEs. The Middle East and North Africa region is the median region in terms of poverty, but has historically had very little local IE evidence to inform policy; that trend looks to continueless than 3.5% of the newly-approved IEs will be going to that region. Also concerning is the fact that comparing the FY15 update to the FY16 update IEG notes that growth of IE coverage seems to have slowed significantly.
Overall comment. Similar to issues raised in this IEG evaluation, IEG's report Behind the Mirror: A Report on Self-Evaluation Systems of the World Bank Group(or so-called ROSES) discussed at the CODE meeting on June 29, 2016, raised similar issues. There was consensus that the WBG has comprehensive self-evaluation systems in place which adhere to international good practice standards, however, there is a need to further amplify learning and strengthen the quality of Monitoring and Evaluation in operations. As part of that commitment to CODE, Management is taking steps to further boost Impact Evaluations (IEs) and learning generated from IEs in a more systematic manner, and has delegated OPCS with the mandate to lead the formation of an IE Working Group (starting from September 2016) focusing on the following tasks:
a) Stocktaking of IEs: This is a rudimentary but important starting point given that currently there is no central database on IEs conducted across the Bank. We would like to have a clearer picture in terms of: quantity (number of IEs), classification (e.g., stand-alone IE to address a specific knowledge gap, IE integrated into the project design), and financing (e.g., financed through a TF grant, Client-executed loan financing). As part of the stocktaking exercise we will also reach out to the various IE hubs (e.g., i2i, SIEF, Africa Gender Action Lab, Health Results Innovation Trust Fund).
b) Learning from IEs: The WG will explore options to make existing IEs more easily accessible to staff to help knowledge sharing and learning (including how IEs are used to inform policy dialogue with Clients, recalibration of existing operations, or design of new operations).
c) Strategic use of IEs: Currently IEs are primarily financed through TFs tied to sectors/themes/regions, and availability of funds greatly influences when to undertake IEs. It is not feasible nor necessary for every new operation to have an embedded IE, however, GPs could identify a small subset of new operations to be prioritized for rigorous evaluation. OPCS will help shape guidance/criteria how to promote more strategic use of IEs, and identify any constraints (e.g., financing, technical capacity) and proposed next steps/actions.
This explicit IE mandate, and identification/prioritization of tasks, will more than address all the concerns raised in Recommendation 96, and in the other recommendations. This overview frames all of the Management responses to the recommendations of the report, including the requests for additional information from the Health Results Innovation Trust Fund and other GILs.
i2i, SIEF, Gender LAB, HIRTF and Jobs represent at least 89% of IE products. All of these have gone through rigorous selection processes. Of the remaining 11%, the great majority come from the HD sectors. Some of them may have gone through other selection processes.
For more updates, please see attached CODE2012-0018.WB_MAR_FY16_96 document
The OPCS update gives information for three impact evaluation hubs at the World Bank: i2i, SIEF, and the Africa Gender Innovation Lab. In general, these hubs have done outstanding work in making progress on this recommendations, each in its own way. Still, these hubs do not constitute the whole of World Bank IE activities; while they undoubtedly cover the majority, it is not clear what share of Bank IE activities is covered by the incomplete OPCS update.
The OPCS update does not indicate what the strategic priorities for selecting impact evaluations for FY15 were. It does detail the selection process of three IE hubs.
Even so, the World Bank, and i2i in particular, has done excellent work in expanding the scope of regions and sectors for projects to be impact evaluated to Fragility and Conflict Situations, Agriculture, and Energy and Environment in FY15. The total of 59 impact evaluations representing 27 IDA countries is excellent progress. Though it is unclear how many individual projects are represented by these 59 IEs, it is likely that tally exceeds the FY14 target of 22 projects from IDA countries.
The selection process for SIEF involves ex ante identification of knowledge gaps, and then strategically places calls to fill them. Strategies relying on ex post identification can be responsive to needs from the GPs and Regions, but are also more exposed to the risk of being selected ad hoc. IEG encourages the other IE hubs to include clear ex ante statements on the strategic selection priorities by region and sector.
Because OPCS did not provide aggregated counts of approved impact evaluations, IEG has used the nearest available proxy of IE concept reviews undertaken by Region and by GP to assess geographical and sectoral selection in the Bank's IE portfolio. Regional representation is still skewed away from MENA, which by IEG's count accounted for only 2% of impact evaluation concept reviews between 2013 and 2015; though this trend is persistent, it is not dissimilar to trends of non-Bank IEs. Africa accounted for 57% of IE concept reviews but 34% of the world's poor (PREM estimates), while South Asia and East Asia and the Pacific combined for only 25% of IE concept reviews, even though these two regions account for more than 60% of the World's Extreme Poor.
Sectoral representation is certainly improving along with the growth of IEs, but there is still significant work to be done. The Macroeconomics and fiscal management, Energy, Finance, Transport, Poverty, and Environment Global Practices accounted for only 10% of all IE Concept Reviews.
The OPCS update indicates that Macroeconomics and Fiscal Management and Public-Private Partnerships are not covered by the three organizations represented. Moreover, it asserts that HRITF is not an impact evaluation hub, even though HRITF has 29 impact evaluations currently in its portfolio. Finally, it does not describe the existing efforts and plans of the other regional Gender Innovation Labs. Without information on strategic selection for these areas, the OPCS update is incomplete. Moreover, because OPCS does not realize that HRITF is substantially engaged in impact evaluations or that the other regional Gender Innovation Labs are in the process of ramping up their efforts, it is difficult to assert that there has been a high level of coordination in strategic deployment of impact evaluations across the whole of the World Bank.
Still, the three IE hubs OPCS does list have clearly, if separately, made substantial improvements by developing a cogent process for selecting IEs, even if the selection criteria and strategic priorities have not always been made clear.
See attached file
Additional Information from Management:
Identify the areas in GP/CCSA, if any, that are not covered by i2i, SIEF and GIL
I2i/DIME response:
All GPs and CCSA are covered by i2i, SIEF, and GIL, with the exception of:
o Macroeconomics and Applied Fiscal Management
o PPP
Describe how the topics of the calls for expressions of interest are decided. What factors are considered when deciding which knowledge gaps to fill. Or are the IE entities simply reacting to the gaps identified by applying Principle Investigators?
I2i/DIME response:
The GP and regions play a key role in identifying knowledge gaps. At the thematic level, program-wide research teams work with GPs to structure the learning agenda and identify knowledge gaps. Besides coordinating regional/GP IE activities on an ongoing basis, regional and GP Focal Points play a pivotal role in identifying projects for IE and in supporting the process for ranking policy relevance, which feeds directly into funding decisions.
Affirm that each region and each GP and each CCSA has an impact evaluation focal point for impact evaluations, as is implied.
I2i/DIME response:
All GPs and CCSA that have specific programs within i2i have IE focal points, including Agriculture, Energy and Extractives, Finance and Markets, Governance, Social, Urban, and Resilience, Trade and Competitiveness, Transport and ICT, Water, and all regions.
The Bank is commended for establishing a selection strategy for the 5 thematic areas and 2 cross-cutting themes included in the envelope of the i2i umbrella fund.
However, it is unclear what influence, if any, this selection strategy has on the other thematic areas / global practices / cross-cutting solutions areas at the Bank, or on the other impact evaluation hubs at the Bank that are not included under i2i (e.g. HRITF, SIEF, GAFSP) or impact evaluations that are conceived and executed within the funding envelope of a project.
While the i2i documentation and governance structure successfully included representation from across regions and indicates that it will cover IEs from IDA (as well as non-IDA) countries as required by WB Action A, the Bank has not indicated which or how many IDA countries will have an IE. Furthermore, the Bank has not provided evidenceas towhether 22 IDA projects were selected for FY14 using the i2i strategic selection framework, as per WB Action A.
Bank-wide consultations were undertaken during 2013 to establish a Bank-wide governance structure for strategic identification and prioritization of IEs at the World Bank. The new structure was put in place in 2014 under the new impact evaluation umbrella, called Impact Evaluation to Development Impact (i2i). The i2i governance comprises a Steering Group, a Council, Regional and Global Practice Focal Points, a Technical Committee, Technical Reviewers, and a Managing Unit (DECIE). The Steering Group comprised of donors, partners, and internationally recognized experts, provides broad upstream guidance on the strategic direction of the i2i program. The Council comprised of Chief Economists and Directors of Operations from relevant VPUs reviews and advises on the overall i2i work program and progress, and provides guidance on strategic focus. At the thematic level, program-wide research teams work with GPs to structure the learning agenda, and identify knowledge gaps. Regional and GP Focal Points coordinate regional/GP IE activities on an ongoing basis, help identify projects for IE and support the process for ranking policy relevance. They also control quality at entry by reviewing concept notes for most impact evaluations in the regions. External Technical Reviewers review and rank i2i proposals for technical quality and their importance for filling knowledge gaps. A Technical Committee comprised of internal IE technical experts provide technical expertise, guidance and advice on technical quality in the selection of IEs to ensure quality and technical rigor of i2i IEs. DECIE manages and delivers the i2i work program.
The approach is being implemented in the roll out of i2i and contributing to a strategic approach to guide IE selections across sectors and regions. The objective is to expand the use of IE and especially in areas that have been under-evaluated. These areas include fragile and conflict situations and crime ; violence, energy, environment, transport, public sector governance, and gender. Furthermore, within each area, the idea is to focus on a priority set of questions that can significantly increase results on the ground.
In the Africa Region, the Gender Innovation Lab (GIL) established a Steering Committee comprised of policymakers and researchers, as well as a systematic selection process for new impact evaluation activities to ensure that resources are focused on the most critical questions and potentially promising solutions, with the goal of building a strategic body of evidence and increasing the take-up of effective policies addressing the underlying causes of gender inequality in the productive economic sectors.
IEG agrees that impact evaluation (IE) use is increasing, in line with the agreement under IDA16. IEG acknowledges the efforts of management to convene a panel of external experts to advise the Bank on establishing a strategic framework for selecting IDA projects for impact evaluation. However, the findings from this panel have not been made public, raising questions about Bank transparency and accountability. Moreover, IEG has not been able to find a copy of the strategic framework for IE selection or the list of FY12-13 projects selected using the framework and reported to have resulted in a balanced sample across regions and sectors. IEG has also been unable to locate the list of 22 IDA projects selected for FY14 using the strategic selection framework which was intended to be done before June 2013. At a minimum, these documents should be archived so as tobe easily found and to guide selection work and indicate progress. The lack of available documentation prohibits IEG from verifying the existence of the strategy or assessing the strength of its application in selection.
Indeed, the new IE selection strategy wasnotcitedin the most recent paper on IDA strategy, "The Demand for IDA 17 Resources and the Strategy for their effective use" (June 19, 2013). The paper mentions impact evaluations in two passages: One is on partnering with others to evaluate interventions that may help to meet the health MDGs (which may be part of a strategy); the other is an accounting of completed and ongoing IEs in Africa and on gender.Noindication is given of how those were selected. Similarly, the document "IDA17 Overarching Theme: Maximizing Development Impact" notes that IE will be an important component of the IDA monitoring and evaluation strategy, but mentions nothing of a selection strategy. This leads IEG to conclude that the Strategic framework for IE selection is not widely disseminated, visible, or understood within the Bank or externally.
IEGtakes note ofManagement's response that a new IE selection framework has been deployed that results in a balanced sample of operations across regions and sectors,and is encouraged to read that regions are aware of these changes and have been reporting on these IEs. However, IEG has not been able to find documentation of the strategic framework for selecting IDA projects for impact evaluation.In keeping with the intent of the recommendation, Management should formalize, prominently publicize, and make easily accessible the new IE selection guidelines.
While the more than 20% increase in the number of impact evaluations directly associated with IDA operations is impressive, the real year-on-year increase of 3 impact evaluations is less so, especially considering that recent trends in impact evaluations indicate that this increase would have been reached without special management attention. Compared to the large number of IDA operations potentially amenable to an IE, the increased number represents a small fraction, [perhaps less than 5%, of IDA operations].
IDA is increasing the use of impact evaluation (IE). Based on the recommendations of a panel of external experts, Management has established a strategic framework for selecting IDA projects for impact evaluation. The new framework has been deployed in selecting FY12-13 projects for impact evaluation and has resulted in a balanced sample of operations across regions and sectors for IE. Regions and Networks have used the strategic selection framework note as a guide as they proceeded through the IE selection process started in FY13 in developing the FY14 pipeline of IDA projects for impact evaluation and updating the existing FY13 pipeline. Regions have assigned contact persons who are responsible for monitoring the portfolio of IDA projects with impact evaluations in their respective regions. They have been reporting periodically on the status and budgeting of these IEs to OPCS. Management has increased the number of IEs directly associated with IDA operations from an average of 14 per year over the FY09-FY11 period to an average of 16 per year over the FY10-12 period, and exceeded the IDA16 performance standard of 17 in FY12-13 period with an average of 20 IEs.