Adopt and consistently apply good practice quality standards to the conduct of all IEs at the World Bank and IFC, including independent peer review protocols. Additionally, ensure data availability for replication.
4. Adoption of quality standards In the World Bank: Over half of completed IEs were of high quality and another two-fifths of IEs met medium quality standards. World Bank IEs go through varying degrees and types of quality assurance processes, especially IEs not initiated under IE initiatives, such as SIEF, which has formal and standardized quality review controls. In addition, there is low availability of IE data for replication, which can help to ensure quality.
Original Response: WB: As part of its work on accountability and decision-making Bank Management has clarified that sector units in the Regions are responsible for operational and Analytic and Advisory Activities (AAA) quality. Any movement away from that framework dilutes accountability. As noted above, Network are responsible for providing relevant sector knowledge, including on IE. Networks also have the responsibility to identify qualified and independent peer reviewers. DIME will support sectors and task teams by identifying qualified peer reviewers and consultants who can assist with IEs. Management will report on progress in the context of quality reporting. The Bank's open data and open knowledge initiatives ensure the general availability of data for replication, with very limited exceptions as set out in the Bank's access to information and Creative Commons policies. This very openness helps enhance quality because of the (welcome) external scrutiny of Bank-supported IE.
Implement good practice quality standards
Indicator: DEC, in consultation with networks/regions, develops norms and standards , including protocols for peer reviews, for IEs as part of the modernization of knowledge products;
Baseline: No norms and standards
Target: Norms and standards developed; at least 2 networks implement a review and monitoring framework of IEs using the norms and standards
Timeline: FY14
The quality assurance processes outlined in the management response are generally strong and warrant a SUBSTANTIAL rating. For example, i2i's practice of requiring a review of the coding by a fellow RA is commendable. Even so, there is little information provided on the level of compliance with those standards generally (outside of the assertion that no exceptions are made for i2i's compliance standards).
IEG had registered concern regarding the independence of i2i's peer review process if the product is reviewed solely by a fellow economist (presumably within DECIE) and the DECIE manager (and the reported process does not seem to require external peer reviewers). There was considerable potential for a conflict of interest on which and whether results were published. However, the additional information provided that double blind external review is applied to three stages of i2i IEs assuages those concerns.
The fact that only a third of the products have data posted to the microdata catalogue restricts the scope for independent replication and validation work.
SIEF reports its policy for data publication but does not report the compliance for actual publicationthat is, how many SIEF datasets have been published as a share of those that should have been? Similarly, the Africa GIL indicates that 27 datasets have been published through the microdata catalog but does not report compliance in terms of the number that should have been published to date.
The lack of detailed information on IEs generated outside of the 3 reporting organizations calls into question the consistency with which good practice quality standards have been applied to the conduct of ALL impact evaluations at the World Bank, as per the original recommendation. Management has provided no information on overall compliance to quality standards across all IEs produced at the Bank.
There is likely more work to be done in this area, but given that the reported data is incomplete over multiple dimensions, it is impossible to determine how much work remains.
Please note the overview comment in recommendation 96, concerning the requests for additional information from the Health Results Innovation Trust Fund and other GILs.
The Impact Evaluation to Development Impact (i2i) upholds high standards throughout the life cycle of the IE. It supports high research quality and policy relevance to ensure that the answers provided are actionable. To do so i2i works to secure the quality of: (i) methods used for causal analysis; (ii) data generation; (iii) study protocols and implementation; (iv) analysis and peer review; and (v) collaboration and policy dialogue to ensure policy actions at multiple stages of the policy cycle.
New IEs apply through Calls for Proposals by sending Expressions of Interests: if selected, they receive $25,000 preparation grants, which support IE teams as they develop a Concept Note. Ongoing IEs, or new IEs that have completed Concept Note, apply for implementation funding by submitting their Concept Note: if they pass technical quality review, they receive yearly implementation grants of $50,000 for up to 3 years. Data collection grants and dissemination grants are also available. As a first quality check, all IEs must select a TTL with substantial IE expertise in order to be eligible for i2i funding. Several further quality checks are applied at all stages of the IE life cycle, with a particularly emphasis on the IE design stage, given its overriding importance in the context of prospective evaluations.
Additional information:
- i2i, SIEF, Gender LAB, HIRTF and Jobs represent at least 89% of IE products.
- All i2i IE comply with the set standard. No exceptions are made.
- The i2i has three rounds of external double blind external review for every product at EoI, CN and report stages. In additional all IEs developed since the launch of i2i are required to be registered in an online trial registry.
For more information, see the attached document - CODE2012-0018.WB_MAR_FY16_99
The OPCS update gives information on adopting and applying good quality assurance standards for three impact evaluation hubs at the World Bank: i2i, SIEF, and the Africa Gender Innovation Lab. these hubs have done commendable work in making progress on this recommendation, and, as importantly, in harmonizing their approach by using consistent standards. Still, these hubs do not constitute the whole of World Bank IE activities, and it is not clear what share of Bank IE activities is covered by the subsequently incomplete OPCS update.
The laudable adoption of a common set of good practice quality standards that has been applied to all IEs produced by DIME/i2i, SIEF and Africa GIL no doubt covers the vast majority of impact evaluations produced by the Bank, but the fact that OPCS was unaware of the HRITF's substantive stake in the production of impact evaluations indicates that those impact evaluations are not subject to the same standards (though they may be high quality). Likewise, there is no reporting on the quality control of IEs produced by the other Gender Innovation Labs or by DEC, for example.
The impending advent of data availability for i2i IEs through the Microdata catalogue is a good start which IEG encourages the other IE hubs to follow if they have not already. The question of data availability for SIEF, HRITF and others is also left unanswered in the OPCS update.
See attached file
Additional Information from Management:
Please provide more detail on ensuring data availability for replication. For example, OPCS indicates that all data used in i2i IEs are expected to be made public, but does not indicate if it actually is made public, or where it would be found. Please briefly describe the DEC data embargo rules.
I2i/DIME response:
The data from i2i IEs will be made public through the Microdata catalogue. This is furthermore a requirement for IEs that receive i2i dissemination grants. The main point of the Research Department's Policy on the Release of Data are:
1. DECRG Management views the public release of primary data collected or assembled by DECRG research staff (with support from the Bank or external research funding agencies) as a key part of our mission as well as an important aspect of staff performance. Management also recognizes, however, that the incentive for researchers to collect and assemble innovative datasets is undermined if researchers are not granted sometime after data-collection to carry out and publish research based on their data before making the data publicly available for others to use.
2. Primary data collected through DECRG research activities are to be made public, unless they are subject to a Bank policy restricting access, or if the permission to collect them by the respective government included an explicit restriction on public release, or the data were purchased subject to such restrictions. Public release of data should ideally be in the form of a fully validated ("cleaned") and documented dataset downloadable from the Bank's research site and/or the Bank's microdata catalog. In some cases, however, it will be deemed sufficient that the dataset is made available to other researchers on request, with an announcement to that effect on the research web site.
3. The researchers involved in the collection of the data should be given an opportunity to analyze the data prior to their being made public. Data should be released not later than 9 months after the publication in a journal or book of the first research paper based on those data, and not later than 24 months after all data collection is completed. DECRG management can grant an extension based on special circumstances.
4. The most appropriate form of the data will naturally vary, but a good guiding principle is that it should be possible for a user to replicate the means and covariance matrix (standard deviations and correlation coefficients) of all the variables actually used in the paper. (More complicated calculations from the data may not be replicable without supplementary software and appropriate training of the users, and these are not the direct responsibility of the DECRG researcher.) The form in which the data are released can either be a primary sourcedataset (actual data, or a link to that data) or it will be the paper's own constructed dataset, as derived from the primary dataset. In either case there should be adequate scientific documentation of how the researcher derived the paper's dataset from the source data.
5. Particularly large or complex datasets may be released in two or three batches, where each batch contains a "clean" subset of the data on which DECRG researchers have already done analysis. This applies also in the case of datasets collected over a period of time, such as in longitudinal datasets where data may be released wave by wave or after the endline data have been analyzed.
The guidelines developed by DIME and adopted into the concept note template for i2i are an excellent step. Its integration into the Africa Region Gender Innovation Lab concept note template is equally welcome. However, Management has not clearly indicated how these standards have been adopted by 1) the other IE hubs, or 2) the Regions.
Still, Management has demonstrated that they are in compliance with the minimum targets for this recommendation. Both the PREM and FPD networks seem to have incorporated norms, protocols and standards for quality control. It is unclear from Managementï¾s update whether the PREM and FPD standards are harmonized.
The World Bank committed to implement good practice quality standards
Target: at least 2 networks implement review and monitoring framework using the norms and standards by FY14
The FY13 IE guidelines and use of the IE portal for processing all IE milestones are compulsory for all IE products in the Bank. Other standards and practices (including concept note, ethical standards, blind external peer review, monitoring framework, etc.) are being adopted by the rest of the Bank through their participation in i2i programs and new and ongoing financing windows. This currently represents a large of projects in (formerly) non-HD sectors.
In addition in FY14, PREM, in collaboration with DIME, launched ieGovern, the Bank impact evaluation program on public sector governance, and established a working group with representation from management, operations and research to define knowledge and operational priorities in civil service reform and the decentralization agenda. The working group supports the management and operationalization of the IE program. Through this collaboration, PREM adopted norms, standards and protocols for high quality impact evaluation for their operations. In FY15, the Governance practice in collaboration with DIME appointed an IE focal point, and expanded the ieGovern working group and the ieGovern program to include procurement and financial management issues.
Similarly, FPD and IFC in FY14, in collaboration with DIME, continued the implementation of the FPD (started in 2010) and Business Climate (started in 2012) programs, managed by a FPD/DEC working group with representation from management, operations, research and donors to define knowledge and operational priorities, competitively select IE cases, monitor progress, and discuss and disseminate results to the practices. Through this collaboration, FPD adopted norms, standards and protocols for high quality impact evaluation for their operations. In FY15, the program was transferred to the Trade and Competitiveness practice which an appointed IE focal point and working group.
In FY15, this model is being replicated in the other GPs and CSSAs, with ongoing arrangements in Agriculture, Environment, Energy, Transport, Water, FCS, PPP, and CC, in addition to Governance and C.
IEG endorse the new IE guidelines and agrees that if they are applied as conceived, the quality of Impact Evaluations at the World Bank will be high. However, while supportive of the objective that all IE products meet quality standards, it is not clear how Management will implement this or what its record has been to date. IEG commends DIME for supplementing quality standards with independent review at concept and report review and encourages such efforts to be applied across the Bank. While the quality standardsguidelines appear on the OPCS IE web page, a robust strategy should be established for monitoring and reporting compliance with these quality standards.
Furthermore, as IEG found that ï¾there is low availability of IE data for replicationï¾ and recommended that Management ï¾ensure data availability for replication,ï¾ Managementï¾s response that ï¾the Bankï¾s open data and open knowledge initiatives ensure the general availability of data for replicationï¾ is unsatisfying. While the new IE Portal in Action 0098-02 references a data catalog, this catalog could be expanded to include links to the data (and preferably the data cleaning and analysis program files) used for the IEs used in the IE Repository.
Finally, DEC developing a log frame to analyze the portfolio of IEs across all sectors is useful, but the does not merit a rating of High as it does not detail what the rest of the World Bank is doing on their IEs nor does it address the Action to "implement a review and monitoring framework of IEs" for "at least 2 networks,"though this may be anticipated for the FY14 timeline.
WB Action F
Quality standards: The new IE guidelines establish quality standards for all IE products. All IE products are required to meet IE products milestones and procedures. Peer reviewers must include at least one sector and one IE specialist. The list of IE specialists is available online. As part of the forthcoming IE UF, DIME will supplement the process with independent reviews at concept and report review stages to obtain standard technical quality ratings across the IE portfolio.
DEC developed a logical framework to analyze the portfolio of IEs across all sectors. This includes 1. The questions of the IE associated with the hypotheses and underlying theory of change; 2. The mechanisms being tested; 3. Some distributional elements; 4. Quality of data collection; 5. Level of institutional support to the IE; 6. Composition of research team; and 7. Links to World Bank projects. DEC researchers conducted a portfolio review of all DIME IEs (160 IEs) and reported the results to the Director of Research. DIME responded with a suggested set of actions. This analysis forms the basis upon which action F Norms and Standards will be developed. The action is expected to be achieved in FY14 as per target.