Improve the integration of IEs into design and review of projects at the World Bank and IFC to sharpen the focus of project operations on results by: Stating clearly in the implementation design of the IE how it will achieve operational usefulness, serve the key decision points of the project, engage operational teams and local counterparts, and disseminate the findings to the relevant audience, in particular local counterparts. Strengthening the use of IE evidence in the appraisal and ex-post assessment of World Bank and IFC projects, wherever such information is available. Effectively communicating IE evidence to the global audience through maintaining a central repository of IEs and undertaking thematic syntheses of existing IE evidence. Building capacity of project teams and other local counterparts to understand and integrate IE evidence in program and policy decisions.
3. Integration of IEs into project design In the World Bank: Overall, there is a modest feedback loop between IE production and project operations and learning. There are notable examples of IE influence on development practice, including project assessment, decisions to design and sustain evaluated and future projects, raising the profile of certain types of interventions, informing policy dialogue and institutional strategies, and building local M&E capabilities. Such examples indicate that, overall, IE is regarded as a valuable tool to increase development effectiveness through better evidence. But in some instances, even when IEs have been relevant and of good quality, they appear to have had limited use and influence due to varying reasons: poor timeliness, failure to engage project teams and decision makers, or lack of dissemination. However, there are signs of improvement, including for example dedicated SIEF support for results dissemination as well as closer collaborations with operations and clients in design of ongoing IEs.
Original Response: WB: On the Bank side, the integration of IE into the design and review of projects is part of the IE strategic selection framework developed as part of the IDA16 commitments. Identified projects will explicitly acknowledge requirements for impact evaluation, describe the primary questions to be answered, and the impact evaluation plan will include baseline and endline data collection activities and an ex ante projection of the counterfactual. DIME will provide technical advice and guidance as needed. Bank Management notes that the use of existing evidence in areas relevant for proposed operations is applicable to all knowledge products. As part of the overall strengthening of operational quality assurance, the role of Network anchors is being strengthened; notably they will be accountable for providing the relevant sector knowledge, including on IE findings, to task teams. Progress will be monitored as part of regular reporting on modernizing knowledge services. The Bank maintains an internal and external database of impact evaluations, notably through DIME, which has an active dissemination program. The Bank is exploring options for developing an open IE knowledge market and will report progress in the context of regular reporting on modernizing knowledge services.
Improve staff and units capacity to integrate IEs into project design and reviews
Indicators: Number of workshops to integrate IE evidence in project design, build capacity and introduce testing to provide feedback into project implementation.
Target: at least 6 workshops per year
Indicator: DIME and Networks reports on the number of IE workshops held annually
Time Line: Ongoing (with annual reporting).
Development new IE Product Portal
Indicator: New portal launched and operative
Target: New IE product portal will ensure full access to information on IE products. (This covers all IE with an IE-code) Baseline:
IE Repository (DIME will continue to maintain); IE data catalog (DECDG will continue to maintain); No portal with full information access
Time Line:
ï¾ Portal launched FY13
ï¾ Portal Phase 2 (Detailed guidelines, Concept Note model including dissemination plans, IE reviews list etc.) FY13-14
Improve integration of IEs into project design and reviews
Indicators: annual IDA reporting
Target: increase the number of impact evaluations directly associated with IDA operations by 20 percent (to at least 17 per year during FY12-14)
Baseline: annual average of 14 per year
Time Line: IDA 16 period.
The lack of reporting details on IE activities beyond i2i, SIEF, and the Africa GIL makes overall progress of the Bank on Recommendations 98c and 98e unknown. The OPCS revised update asserts that 89% of IE products come from i2i, SIEF, Gender LAB, HIRTF and Jobs(IEG's calculations with the data provided by OPCS indicates that 86% of ongoing IEs come from these groups), but does not give detail on the progress of HRITF or Jobs regarding any of the elements of recommendation 98. Consequently, the overall progress on recommendations 98c and 98e are still unknown.
The efforts made individually by the three reported hubs to improve staff capacity to i2i, SIEF, and the Africa GIL integrate IEs into project design and reviews is excellent. Each has elements worth emulating by the others. The i2i/DIME training for teams embarking on the process of doing an impact evaluation is thorough and effective and, along with the SIEF workshops, demonstrates learning improvements. The SIEF efforts to summarize findings for policy makers and SIEF's aggressive outreach program that reviews Bank IE work and findings, including briefs and workshops, is excellent. Finally the Africa GIL's work to document its project influence on design of subsequent projects is innovative and has the potential to catalyze greater influence precisely because it is attempting to measure its influence.
The sections covering both SIEF and DIME emphasized that although they are working to catalogue IEs within their focus area, they do not have the financing or mandate to require registration by TTLs or to serve as a central repository for IEs. This in fact highlights the challenge of fractured funding and coordination for IE work at the Bank. The previous system of an offline, by-request largely static database that presupposes knowledge of both its existence and the appropriate contact person was clearly sub-optimal. IEG takes heart in learning of the renewed effort to catalog Bank IEs in a repository maintained by DIME. The 2011 IEG evaluation's recommendation for a role for central coordination function is still germane. The OPCS IE Working Group resulting from the IEG ROSES evaluation holds future promise because OPCS presumably does have the mandate to be coordinate across the fractured IE landscape to encourage stocktaking and learning through a central IE window. Although this latest development unfortunately cannot be included in this year's MAR feedback, IEG aims to continue monitoring progress on this remaining work.
Inspection of the IE data catalog shows it to be progressing nicely, though Management did not report on how complete the catalog is. The suggestion from DIME's entry for Recommendation #99 is that only one third of IEs with available results are posted on the Microdata catalog, citing a clause allowing data to be embargoed by research teams for an unspecified period of time or until the authors have completed their report. OPCS did not indicate the particulars of this embargo policy or provide evidence that it is being complied with. IEG views that there is still room for growth in this area.
The action plan target for the number of IDA IEs has been met through i2i alone, though growth appears to be slowing. For example, it appears that i2i did not expand into any new IDA countries (remaining at 27, or 35% of all IDA countries) and a net expansion of only 4 World Bank projects in FY15 (59 to 63) in IDA countries.
OPCS overview comment. Please note the overview comment in recommendation 96, concerning the requests for additional information from the Health Results Innovation Trust Fund and other GILs.
Bank-wide stocktaking of IEs was never an explicit DIME/DEC mandate - there was no such mandate in this intuition. That is why the OPCS-led IE Working Group has been delegated such an explicit mandate to coordinate an exhaustive stocktaking, among other critical IE-related task. Again, this is the first time there is such a initiative in the World Bank, it will take time to systematically coordinate these activities, however, this is start in the right direction.
Building capacity of project teams and other local counterparts to understand and integrate IE evidence in program and policy decisions.
Building capacity was one of the explicit purposes of the creation of the Impact Evaluation to Development Impact (i2i) program. Every year i2i organizes several workshops, which gather policymakers, World Bank operational staff, external researchers, and subject experts to advance knowledge on the research agenda and identify new IE products for financial and technical support. These workshops serve three main purposes.
- First, government counterparts and other partners receive training in IE methods and are given tools to allow active participation in the development and implementation of their IE and to become better informed consumers of knowledge, whether generated through IE or other methods.
- Second, participants are exposed to the latest available evidence in each focus area, with a view to incorporating relevant evidence in their own program and policy design.
- Third, each country team is paired with one or more IE researchers to develop a prospective IE that will answer research questions of direct relevance to their specific program. These researchers continue working with each team throughout the process of IE implementation. Researchers working hand-in-hand with operations are an effective mechanism for discussing and operationalizing evidence into project design.
i2i, SIEF, Gender LAB, HIRTF and Jobs represent at least 89% of IE products. Data sharing is compulsory for ALL i2i /SIEF/GIL supported IEs. However research teams are allowed to embargo their data on intellectual property grounds. The data is provided but not made publically available until the authors have completed their report.
For more information, please see attached document - CODE2012-0018.WB_MAR_FY16_98
As elsewhere, IEG notes the lack of a report of IE activities beyond i2i, SIEF and the Africa GIL, rendering the OPCS update incomplete.
The training efforts by the Bank, and i2i in particular, are exemplary in nearly every way. They are thorough and effective. IEG continues to be concerned, however, by the regional representation of the IEs proceeding from those workshopsEast Asia and the Pacific, Middle East and North Africa, and South Asia combine account for nearly two thirds of the World's extreme poor but less than one third of the IEs participating in i2i workshops, while the LAC region represented 30% of the IEs in those workshops but accounts for just 3% of global extreme poverty. More training and better evidence is likely of greatest value where the problems of poverty are the worst.
All three reported hubs have solid strategies for disseminating IE findings to the project, country and government teams subject to a particular impact evaluation, and all produce policy notes and maintain an online presence of their activities and findings. SIEF's strategy for disseminating findings beyond the target countries is extremely helpful, and the Africa Gender Lab's move to develop synthesis reports is an excellent move; the highly popular Smack Downseries from the Africa Chief Economists office can be a model for other regions.
Yet there continues to be a troubling trend in the abandonment of the commitment for DIME to maintain an IE repository and DECDG to maintain an IE data catalog. This represents a tremendous missed opportunity to facilitate learning, and is something that the World Bank can and should do. To compensate for not having a global solution, several regions and GPsamong them South Asia and Social Protectionas well as the three IE hubs included in the management update each maintains its own database. This is highly inefficient and does not take advantage of opportunities that the World Bank has to distinguish itself amongst bilaterals and other multilaterals by cross-pollinating ideas that work across regions and sectors. The lack of a dedicated, central repository of IEs, as recommended in 2011, and undermines communication and learning of IE evidence to World Bank TTLs and to the global audience. 
See attached file
Additional Information from Management:
(0098-02 WB Action D) There is currently no comment at all on effectively communicating IE evidence to the global audience through maintaining a central repository of IEs and undertaking thematic syntheses of existing IE evidence.
I2i/DIME response:
The i2i and DIME team regularly collects information on the status and technical details of each IE in its portfolio. This information is collected once or twice a year depending on the phase the IE falls in, and covers over 100 indicators related to:
o General information: i.e. GP, topic, status, timeline, budget etc.;
o Evaluation design: i.e. intervention, outcomes, methods, unit of intervention and analysis, treatment arm(s), mechanism(s) tested, sub-group analysis, etc.;
o Data collection: i.e. details on survey rounds and timeline;
o Technical quality and best practices: i.e. whether it has ethical clearance, a study registry, etc.
o DIME involvement: whether DIME helped with fundraising, providing a TTL, securing client engagement, assembling research team, collecting data, etc.;
o IE influence on program/policy: counterpart details, whether the IE provided high-quality baseline and follow-up surveys, attrition and take-up rates, whether the IE led to improved counterpart M;E, rationalized policy design, motivated the scale-up or down of policy, etc.;
o Documentation: i.e. baseline report, policy brief, working paper, final paper / report, publications, Microdata catalog link, etc.
Reporting on all these indicators is mandatory for all IEs seeking to renew i2i funding (every 6 or 12 months). The i2i/DIME team is currently in the process of setting up of a new online-based monitoring system for the purpose of tracking this information. The system will serve both as a data collection as well as a data storing and management tool. It will greatly ease the current reporting burden on researchers, thus ensuring high compliance, as well as enable the effortless generation of reports and analytics. The system is projected to be ready by the end of 2015. This is a further step towards the end goal of having a public repository. In the meantime, the IEs portfolio database is shared by the team at the request of any researcher / interested party.
I2i has committed to DFID, our donor, to produce knowledge reviews for policymaking, including mechanisms, sector-to-sector and country-to-country adaptation lessons: this is an indicator in the i2i log-frame. In FY15, no output was expected, but i2i over-delivered by producing a paper answering the questions: Does research add value to aid? Specifically, does impact evaluation research help or hinder the delivery of development projects?The paper can be found here and a policy brief can be found here.
(0098-03, WB Action E) Beyond the significant contributions of the 3 IE entities discussed, please indicate how each of the Global Practices, Regions and Cross-Cutting Solutions Areas have approached this recommendation of improving the integration of IEs into design and review of projects at the World Bank in the use of IE evidence in appraisal and ex-post assessment.
Action C: The number of training workshops carried out by DIME and SIEF (9) alone exceeds the minimum target of 6. These appear to be well-received and useful in imparting knowledge transfer on how impact evaluations are to be done and improve the likelihood that a project will undertake an impact evaluation.
Regional balance of projects represented at the workshop is good but could be improved. The workshops have quite rightly focused on projects for the Africa and South Asia. However, no ECA project attended an IE workshop, and of all projects represented at an IE workshop, only 13% came from the populous and challenging EAP region and only 7% came from MENAï¾a region consistently lagging behind in IE production.
While these workshops have improved IE capacity among attendees, Managementï¾s update has not presented evidence that IE evidence is better integrated into project design and reviews generally. Relying on workshops alone would seem an inefficient way to achieve that objective; a multi-pronged approach may be appropriate.
Action D: While a database of impact evaluations does exist within the project portal, its accessibility and utility is clearly a decline from the previous year when DIME hosted a dedicated IE repository and DEC hosted a dedicated data catalog. This is especially true for external parties who cannot access any form of database for the World Bank IEs because the web presence of the database from last year has been deleted (and would now be outdated).
Even so, if Management can make the database more accessible for both internal and external audiences and include additional filter options (such as ï¾outcomesï¾ to find, for example, all Bank projects that have an IE estimate on poverty reduction or by Cross-Cutting Solutions Area (e.g. Gender, Jobs)), the comprehensiveness of the current database in having both all IEs and all documents related to each IE can be a tremendous boon to improving the integration of IE findings into project design.
Action E: In this last period of review for Action E, the target has been exceeded: Management reports that 22 of the projects approved in FY14 in IDA countries had impact evaluations, exceeding the minimum of 17 per year.
The World Bank has made good progress against its committed actions.
The World Bank set a target to deliver 6 workshops to integrate IE evidence into project implementation.
Delivered in FY14: 9 workshop
In FY14, DIME conducted 6 cross-country workshops with a participation of a total of 82 projects. The workshops were advertised with all pipeline projects in the relevant sectors across all regions (expect for the health and agriculture workshops in Senegal that were specific regional requests) and several team discussions were held in advance of the workshop. Each region appointed an IE focal point and a working group to work with our researchers to define priorities and identify cases.
· Improving life in Senegal: Investing in Health (5 AFR projects, Dakar, Oct 2013)
· (with SIEF) Quality education, skills and productivity among youth in Africa (23 GLOBAL projects, Dakar, Oct 2013)
· Impact Evaluation 4 Peace (20 GLOBAL projects, Lisbon, Mar 2014)
· Innovations for Agriculture (8 AFR projects, Dakar, Apr 2014)
· Impact Evaluation for Legal Empowerment (8 MENA projects, Jordan, May 2014)
· Innovations for Agriculture (18 GLOBAL projects, Kigali, June 2014)
In addition, SIEF conducted 3 cross-country workshops:
· Delhi, India - April 23-25, 2014 (20 SAR projects)
· Dhaka, Bangladesh - April 27-29, 2014 (26 SAR projects)
· Seoul, South Korea - May 5- 9, 2014 (20 mostly EAP projects)
In total 151 teams participated across the 9 workshops. The regional split of projects in DIME and SIEF workshops held in FY14 is as follows: EAP - 20, MENA - 11, ECA - 0, SAR - 49, AFR - 60, and LAC - 11.
The workshops presented the latest evidence and used a minimum of one researcher per project to facilitate the process of introducing evidence in project design and help the projects develop an impact evaluation design. These researchers continue working with each team throughout the process of IE implementation. The workshops remain our main way of incorporating evidence into projects. This is because project teams donï¾t have the time to keep up with all the literature. Researchers working hand-in-hand with operations are an effective mechanism for discussing and operationalizing evidence into project design. As a result 42 percent of DIME projects report having introduced IE evidence in their project design, 53 percent introduced a structured learning strategy to compare alternative implementation modalities and plan for mid-course corrections and 58% improved projectï¾s design based on a better understanding of the causal chain (source: DIME portfolio review July 2014). Please see attached note on monitoring framework for statistics on policy influence.
Data from knowledge tests administered on the first and last day of the two agriculture workshops shows that knowledge on IE methods significantly improved among participants. Prior to the workshop, participant on average scored 66% in Senegal and 61% in Kigali. After the workshop these scores improved to 74% and 72%, respectively.
The first two i2i windows for new IEï¾s were launched in FY14. The technical review process consists of a double-blind review by two external peer-reviewers. The results show a significantly higher rating for proposals submitted by project teams that participated in a DIME workshop. Proposals are ranked on 0-3 scale. Average scores for teams that participated in the workshop were 2.25 for the FCS and 2.06 for the agriculture window. The average score for proposals from teams that did not participate were 2.01 and 1.78, respectively.
The World Bank also committed to develop new IE Product Portal Indicator
In the operations portal, please search by key word impact evaluation and select product line impact evaluation: http://isearch.worldbank.org/cprojects?qterm=impact+evaluation;sitename…
In addition, one can filter by country and sector. The portal has full information on IE products including all attachments (CN, Reports, auditsï¾
) as well as peer reviews, management comments, timeline, ect. You can also identify in the preparation summary which project is associated with the IE. Effort is being made to make the information, partially, publicly available. All IEs are required to have an IE code to apply for i2i grants. This should contribute towards making the IE portal a comprehensive source of information for the whole IE production at the Bank.
The move to the IE portal is significant. It requires all units to follow the same guidelines just like they would for operations and for ESW and other knowledge products. While the operations portal may not be perfect, it is the system the Bank uses for all its products. Enquiries and improvements should be directed to OPCS.
Units also develop monitoring systems for their own activities because bank systems may not address their particular needs. DECIE maintains its own monitoring system and framework for the products it is responsible for.
The World Bank set a target to increase the number of IDA operations by 20%.
Annual target: 17 per year.
Delivered in FY14: 22
As of September 2014, OPCS had verified that among IDA operation approved in FY14, 22 had incorporated an IE in their project. These were not counted in earlier reports. This exceeds the agreed target of 17. The 22 listed are only those that are part of the IDA commitment to do IE, not all IEs in IDA countries.
WB Action C
Efforts of DIME and the Africa region to integrate IE results into project design is much appreciated and highly commendable. Still, while DIME and Africa are critical agents at the Bank, the lack of information about the integration activities from other regions and IE nodes is still unsatisfying. It is unclear if workshops alone are sufficient to ï¾provide feedback into project implementation.ï¾ IEG encourages management to explore this link further and develop indicators on how well IE findings are being ï¾fed backï¾ into project implementation.
WB Action D
The launch of the IE Product Portal is a strong first step towards ï¾effectively communicating IE evidence to the global audience.ï¾ IEG recognizes the need for the planned Phase 2 activities for FY13-14, particularly the dissemination and outreach components, to fully realize the potential of this portal.
IEG notes, however, that further work needs to be done to make this portal fully inclusive of all IEs executed at the Bank or on Bank projects. Management should clarify the set of IEs the portal is meant to catalog (the current description implies it is a repository of DIME Impact Evaluations, though the portal seems to contain IEs beyond that set as well) and work to make the catalog more comprehensive to include IEs from across the World Bank, including those from other funding hubs (for example SIEF, HRITF) and research outside of DIME/DEC. IEG also notes that Management may wish to coordinate this IE Product Portal with similar efforts elsewhere at the Bank (for example, the IE database being built by the Gender and Development group).
Management could also work toward integrating the two main elements of the portal: the IE Repository and the IE data catalog. Currently these appear to be two separate but cross-linked portals. Better would be the repository of IEs with links to the relevant data from the data catalog.
WB Action E
Again, IEG commends Management for exceeding the target number of IEs in design by as much as 50% and acknowledges the wisdom of providing some margin for project cancellations while ensuring the target will be met or exceeded.
Still, as the number of IDA projects including an IE in their designs is small in absolute terms (20 and 21) and as a fraction of all IDA projects. IEG encourages Management to stretch further.
Overall:
IEG applauds the significant progress made, especially by DIME, and acknowledges that Management has largely met or exceeded the targets management established for WB Actions 0098-01 through 0098-03. However IEG feels that there are some specific areas in which the targets do not fully encompass IEGï¾s recommendations or Managementï¾s Response.
In addition to the specific issues mentioned above for WB Actions C through E, IEG also recommended that Management ï¾strengthen the use of the IE evidence in the appraisal and ex-post assessment of World Bank projects, wherever such information is feasible.ï¾ To which Management responded that ï¾the use of existing evidence in areas relevant for proposed operations is applicable to all knowledge products. As part of the overall strengthening of operational quality assurance, the role of Network anchors is being strengthened; notably they will be accountable for providing the relevant sector knowledge, including on IE findings to task teams.ï¾
IEG encourages Management to establish indicators and targets for integrating IE evidence into ex-ante project appraisal and design as well as ex-post assessment of World Bank projects. Noting the IEG findings that few projects cite IE evidence in appraisal and less than half of projects that have had an impact evaluation cite the findings of that evaluation in the implementation completion report, there is clear room for improvement in this area.
WB Action C
During FY13, 9 impact evaluation workshops were conducted.
DIME conducted 6 workshops on impact evaluation with the participation of 67 projects in ECA, EAP, LCR and AFR.
Agriculture (2 projects, Haiti)
Sustainable development (4 projects, Vietnam, Oct 2012)
Urban crime ; violence (12 projects, Brazil, Mar 2013)
Investment climate (12 projects, Global, Paris Nov 2012); this is a WB/IFC collaboration
Health (24 projects, Africa, Cape Town Dec 2012)
Health and social protection (9 projects, Nigeria, May 2013)
SIEF conducted 3 workshops on impact evaluation.
Human Development (13, South Korea, Dec 2012)
•Human Development (India, Mar 2013)
•Human Development (Chile, Apr 2013)
The Bank has significantly scaled up the integration of IEs into design and review of projects and done so increasingly at very early stage of project design. The Africa region, for example, has worked with DIME to systematize the use of IE evidence in the design of all its health, education, social protection and finance and private sector operations, by assigning research teams to all these operations to help them operationalize relevant evidence and transform it into project features. It has also made important strides in nontraditional sector for IE including infrastructure and public sector governance. All DIME IEs have been designed in close consultation with operations and government counterparts to answer policy relevant questions. The central objective of these IEs is to provide evidence for policy decisions. All baseline reports and IE reports are presented to the local implementing agency and wider audience. Further innovations in communication with operations include the Africa Region Slum Dunk BBL series in which a panel of TTLs and a panel of IE researchers hold a lively discussion on IE results and operational imperatives. The series is always oversubscribed. A second innovation is the use of Ignite 7 minute presentations both inside the Bank and around the world to communicate IE results concisely to elicit response and action. These have been implemented during the Big Ideas seminar, the IE presentations to the Board, DIME presentations to IEG, The Nigeria workshop on health and the upcoming education/social protection and health workshops in Senegal among others. These combined efforts have strengthened the use of IE evidence in the appraisal of World Bank projects. In addition DIME newsletter and brief series, the IE blog and HDN evidence series provide a continuous source of information on new IE results.
WB Action D Update: The new IE portal and new IE product guidelines were launched during FY13. The portal is online and fully operational
WB Action E Update: During FY12, 21 IDA projects included IE in their designs. 20 FY13 IDA projects included IE in the designs. Management has increased the number of IEs directly associated with IDA operations from an average of 14 per year over the FY09-FY11 period to an average of 16 per year over the FY10-12 period, and exceeded the IDA16 performance standard of 17 in FY12-13 period with an average of 20 IEs.