Regularly incorporating, where feasible, analytical elements, such as analysis of heterogeneous program impacts and cost-benefit analysis, in the design of all World Bank and IFC IEs.
5. Incorporating analytical elements that enhance operational relevance In the World Bank and IFC: There has been mixed coverage of analytical elements relevant for operational needs, such as analysis of distribution of program impacts; cost-benefit or cost effectiveness analysis of interventions; mapping of the causal chain from program inputs to outputs to outcomes; and measuring the contribution to impacts of individual components of program design. At the World Bank, IEs initiated in recent years appear to pay greater attention to some of these dimensions, and this trend should be sustained in future IE efforts. Similarly, these elements should be included in the design of future IFC IEs.
Original Response: WB: For projects, the quality of design of the results framework is a key element in the overall quality framework being developed for implementation this year. That framework gives sector anchors the responsibility for assisting task teams on results frameworks and indicators and monitoring arrangements, including these analytical elements. Their assistance and support from DIME, notably in Quality Enhancement Reviews (and at the concept stage), will be the setting to assess the importance and feasibility of these analytical elements. The new knowledge quality framework under development will be the setting for IE as a stand-alone core knowledge product.
WB Same as Action F: Implement good practice quality standards
Indicator: DEC, in consultation with networks/regions, develops norms and standards , including protocols for peer reviews, for IEs as part of the modernization of knowledge products;
Baseline: No norms and standards
Target: Norms and standards developed; at least 2 networks implement a review and monitoring framework of IEs using the norms and standards
Timeline: FY14
The World Bank has done commendable work of raising the issue of increasing the share of Bank IEs that examine heterogeneity of program impacts. The update is inconsistent in its report of the level to which the Bank has been able to achieve those aims.
DIME's policy of mapping IEs to the causal chain is innovative and needed for the field. In terms of exploring heterogeneity. Management has clarified that gender-analysis is conducted in 50% of cases for i2i and 100% for the Africa GIL. DIME's efforts to prepare a toolkit to provide guidance on the measurement of heterogeneous effects promises to improve beneficiary analysis, but was not disseminated within the FY16 period to be considered in this update.
DIME continues its assertion that it does not have funding to support cost effectiveness work, though such work typically costs only a small fraction of the cost of a typical impact evaluation. DIME indicates that it is open to providing methodological guidance in doing cost effectiveness work, but does not indicate the frequency with which it has given such guidance. IEG acknowledges the complexity of recovering cost data from Bank projects, though that complexity does not reach the level of difficulty and time required to estimate benefits, as done with IE work. In contrast, OPCS reports that both SIEF and the Africa GIL have taken credible steps to work around capacity constraints to improve the integration of cost-benefit analysis in their impact evaluation work.
IEs funded by SIEF are required to report cost-effectiveness estimates and SIEF is piloting a costing template for ECD interventions. However, the update did not give an indication of the number of IEs that have complied with this requirement or have made use of the template.
The Africa GIL has laid out plans for studying heterogeneity of impacts and has requested all impact evaluations to include ex ante cost analysis to facilitate ex post cost analysis. However the update again does not give an indication of the level of compliance with these plans and requests.
At this stage management should have moved beyond planning and into implementation, particularly for cost benefit analysis which does not require ex ante planning (though it is extremely helpful). For all of the plans and policies associated with heterogeneous effects and cost benefit analysis, there appears to remain considerable work to be done in effectively implementing these plans and policies. This in combination with the lack of overall reporting on the degree to which Bank IEs as a whole now explore heterogeneous effects and cost effectiveness keeps the IEG rating at SUBSTANTIAL.
Please note the overview comment in recommendation 96, concerning the requests for additional information from the Health Results Innovation Trust Fund and other GILs.
The Impact Evaluation to Development Impact (i2i) and DIME support the inclusion of analytical elements in all IEs in their portfolio. For example, about half of i2i IEs have conducted or will conduct a gender analysis.
Distributional impact analysis
The research coming out of the DIME portfolio investigates the statistical feasibility of distributional analysis. Such analysis is performed where the data allows for adequate statistical inference. DIME monitors the number of IEs that plan to perform a sub-group analysis: as an example, over two thirds of i2i-supported IEs will perform a gender analysis.
As explained in the FY14 update, DIME is also currently conducting a study on Distributional Impact Analysis aiming to both improve the quality of the analysis and our policy advice. One of the outputs of this initiative is to develop a toolkit on state-of-the-art knowledge on how to perform distributional impact analysis under different scenarios. The goal of this toolkit is to provide guidance for researchers who want to use experiments to answer questions for which the mean is insufficient as an answer.
To do so, the plan of the study is first try to clarify the questions that call for DIA. Different questions require different methodological approaches, so we group them into (non-exhaustive) categories according to the parameter of interest including (i) impacts on outcome distributions, (ii) distribution of impacts, and (iii) subgroup analysis. The study plans to then discuss key methods for each category. That is, in addition to providing the reader with a set of core methods for each type of question, it seeks to clarify the required assumptions to allow the researcher to choose a reliable methodology or turn to some of the more advanced methods that we will refer to.
The central objective is to provide a set of baseline methods, discuss common problems and detection methods, and shed light on practical issues. Topics include required sample sizes for statistical inference, power calculations and estimation of standard errors in (potentially small or dependent) samples, as well as practical applications, and a set of statistical programs to use the methods discussed in the toolkit.
**please see the attached for the full update***
Additional information:
- For the i2i portfolio: Gender specific interventions are tested in 18 percent of cases and Gender analysis is conducted in 50 percent of cases. These figure are 100% for the GIL
- DIME conducts cost effectiveness studies of the IE products not of the underlying interventions. IEG recommendation is based on the experience of small academic trials that study interventions implemented by NGOs. These ca
The OPCS update does not indicate the number or share of impact evaluations completed in the last FY that estimate heterogeneous effects or introduce a cost benefit analysis. Most progress appears to currently be in the planning or gestational stage. Nor does the OPCS update give the share of World Bank IEs initiated in FY15 and that include such analytical elements.
Plans by SIEF and i2i-supported IEs to undertake efficiency calculations or distributional analysis, respectively, are poised to receive high marks once those IEs are completed, and the results have the potential to support better decisions and intervention designs by policy makers.
The knowledge products on these two topics produced by the Bank are important and welcome. When deployed, DIME's forthcoming toolkit on performing Distributional Impact Analysis will be an exciting resource for evaluators at the Bank and beyond. The plan is indeed impressive. The current outreach by David Evans is an excellent start, and IEG encourages the Bank to build upon his efforts.
There is insufficient detail from the HRITF or the Africa or other Regional GILs to assess their contribution to this recommendation.
See attached file
Additional Information from Management:
For each of the hubs, how many and what share of IEs currently estimates heterogeneous program impacts or has a cost-benefit analysis?
I2i/DIME response:
We regret not to be able to calculate this in the given timeframe. We do know, however, that over two thirds of new, i2i-supported IEs plan to conduct disaggregated gender analysis (at least 49).
How does i2i make the tradeoff between deciding to survey for multiple arms versus Distributional Impact Analysis?
I2i/DIME response:
Surveying for multiple arms and Distributional Impact Analysis Research do not necessarily present trade-off. Nonetheless, contingent on a limited budget, teams work with the clients to jointly define priorities.
Please provide more detail on the level and types of activities of the Africa GIL related to this recommendation
WBG Action G indicates that good practices, norms, and quality standards for execution of cost-benefit analysis, distributional analysis, and mapping the causal chain be done in consultation with networks and regions. While the OPCS supplementation to the Management Response points out that the governance structure of i2i has focal points and council members appointed from all VPUs, it is not clear whether the norms developed for i2i will be used for IE work beyond the purview of i2i.
Distributional Analysis and Heterogeneous Effects
The study being conducted by DIME on Distributional Impact Analysis, together with the promised toolkit, has great promise in being able to shape the application of this analytic element. Finally, IEG notes that distributional analysis and heterogeneous effects have taken on even greater importance over the last year with the emergence of the Bankï¾s ï¾twin goalsï¾, the first of which being to ï¾improve shared prosperity.ï¾ IEG looks forward to the progress of these efforts with great interest.
Cost-Effectiveness
The sound analysis executed by Evans and Popova was completed in FY15 and cannot be counted towards the FY14 MAR. The promised toolkit developed in conjunction with JPAL appears promising but is as yet not available. The inclusion (but not requirement) of a cost benefit component for SIEF proposals is a good start but inadequate for the whole of the World Bankï¾s IE activities. Moreover, there appears to be little guidance on how such analysis should be done within SIEF-funded IEsor what the consequence might be for doing it perfunctorily or not at all.
Theory of Change
IEG welcomes the required inclusion of a theory of change in the i2i IE concept note template, together with an example and guidance. Management has not indicated the level of adoption of this analytic element outside of i2i, and as i2i is new and had not yet funded a full IE proposal within FY14, real change for the inclusion of a theory of change appears to still be in its infancy.
Distributional Analysis and Heterogenous Effects
DIME is currently conducting a study on Distributional Impact Analysis aiming to both improve the quality of the analysis and that our policy advice. Specific objectives of this exercise are to:
i. Investigate what can be done to develop a better understanding of the distributional impacts of public policies (e.g., moving beyond comparisons of aggregate distributions);
ii. Discuss what it would take to include distributional analysis in the evaluation design of future projects, and
iii. Create guidelines for conducting distributional analysis based on (i) and (ii).
One of the outputs of this initiative is to develop a toolkit on state-of-the-art knowledge on how to perform distributional impact analysis under different scenarios. The toolkit is directed to the IE community inside and outside the Bank, particularly but not exclusively, focused on IE for policymaking. This toolkit will include case studies using data from WB RCTs to show how distributional analysis can work in practice. The work will include different approaches for conducting distributional impact analysis. Currently, the structure and content of the Toolkit are being defined jointly between a team from DIME and 2 senior experts (specialists in different approaches from the University of Chicago and the Funda��o Getulio Vargas), who have published extensively in top econometric journals and are supporting the initiative. A Draft of the Toolkit will be available in the first trimester of 2015.
Cost-effectiveness
The DIME team recognizes that an essential complement to impact evaluation is cost-effectiveness or cost-benefit analysis, so that policymakers can account for costs as well as benefits. A new working paper from the team discusses issues crucial to comparing cost effectiveness estimates across countries and for policymakers and task team leaders to be able to effectively use such estimates in their operational work. (Evans, David ; Anna Popova, ï¾Cost-Effectiveness Measurement in Development: Accounting for Local Costs and Noisy Impacts,ï¾ World Bank Policy Research Working Paper, September 2014.) This complements recent work from the Poverty Action Lab (JPAL), highlighting the sensitivity of cost-effectiveness estimates to key assumptions about discounting, exchange rates, etc., (Dhaliwal, Iqbal, Esther Duflo, Rachel Glennerster, and Caitlin Tulloch. 2012. ï¾Comparative Cost-Effectiveness Analysis to Inform Policy in Developing Countries?: A General Framework with Applications for Education.ï¾) and detailed new work demonstrating the construction cost estimates for education interventions.( J-PAL. 2014. ï¾Student_Learning_cea-Data-Full-Workbook-1ï¾. Jameel Abdul Latif Poverty Action Lab.)
In addition to this methodological work, research teams will need a clear, step-by-step toolkit to be expected to actually implement cost-effectiveness analysis. This especially true because cost-effectiveness analysis is principally useful if the costing methods are common across evaluations. The DIME team considered the creation of such a toolkit; however, upon investigation, the team discovered the JPAL has already drafted such a toolkit. JPAL agreed to share the toolkit with DIME to provide feedback, and DIME agreed to disseminate the final product to help research teams more effectively.
Theory of Change
The new template for IE concept notes requires a section on the theory of change. The annex also provides an example and guidance for teams to follow.
IEGï¾s Findings in the IE Relevance and Effectiveness report listed four elements of further analysis that should be considered for inclusion in future IE work: 1. distribution of impacts/heterogeneous effects, 2. cost effectiveness, 3. data-backed mapping of the causal chain, and 4. contribution of components in program design. In Management's response, none of these is addressed comprehensively for the Bank as a whole, but it is encouraging that some entity at the Bank is touching on some aspect of each.
1. Distribution of Impacts and Heterogeneous Effects. DIMEï¾s revised estimate of incorporation of gender analysis into 37% of its IEs is encouraging. While acknowledging the increased cost in data collection to have sufficient statistical power for distributional analysis, the information from such analysis can be critical to informing policy makers on the design and scale-up of interventionsï¾as evinced by the Kondylis and McKenzie research cited by Management. Other decompositions of the distribution of effects, for example by elements of socio-economic status or locality, may be as or more salient than gender in a particular context. While mindful of costs and policy priorities facing clients, tracking and reporting on these elements for all of the Bankï¾s IEs will contribute to the Bankï¾s status as a knowledge leader.
2. Cost Effectiveness. While the SIEFï¾s request of 100% of the IEs it finances to conduct cost-effectiveness analysis is laudable, it is important to also know how many actually follow through on that request and provide high-quality cost-effectiveness analysis. If in 4-5 years there are 42 impact evaluations with solid cost effectiveness analysis as anticipated, that will be a clear improvement over the current state. IEG would hope for similar efforts from other IE hubs as part of a coordinated Bank-wide strategy.
3. Data-backed mapping of the causal chain. It is not clear whether Managementï¾s plans to incorporate a section on theory of change into the Concept Note template for IEs refers to Bank-wide efforts (it is assumed so). Such a step is productive as a necessary initial step for being able to map the causal chain. Further steps of collecting data on intermediate indicators along the causal chain would fulfill this analytical element.
4. Contribution of components in program design. DIMEï¾s approach of combining accountability of whether or not an intervention had an effect with learning why and how the results came to be is a useful upstream approach to understanding the separable or complimentary effects of individual components in a program package. Understanding the behavioral biases, incentives, and delivery, accountability and constraint-relaxing mechanisms is useful for understanding elements of the causal chain (to point 3, above) of any intervention. The recommendation of evaluating the contribution of individual components for packaged interventions still holds.
IEG strongly endorses the efforts of DIME and SIEF in beginning to integrate parts the elements that allow for greater understanding of distributional effects, cost effectiveness, causal chains, and component contribution. However, Managementï¾s response does not include the efforts of other impact evaluation producers at the World Bank who are not associated with SIEF or DIME. IEG encourages Management to track and report the inclusion of these analytic more completely, perhaps as an added utility to the new IE Portal, to be able to generate reports on the distribution of IEs produced in the last year containing each of the four analytical elements.
WB Action G
See Action F update ï¾ Additionally, DIME conducted a portfolio review of its IE products. The review finds that 29% of the IEs measure heterogeneous effects across the gender dimension and 40% of IEs are multi-arm studies unpacking intervention effects. Furthermore 100% of the new SIEF-financed IEs are requested to conduct cost-effectiveness analysis.
Additional Information for the benefit of the IEG team on Action G
Distributional analysis. IEG should be made aware of the cost of doing distributional analysis. Every subgroup added to the analysis requires a significant sample size. For this reason ï¾trueï¾ distributional analysis is very expensive with every subgroup nearly doubling the cost of the IE. Nonetheless,
Very significant progress was made at the Bank in introducing gender analysis in IE in the form of both differentiated impacts and gender-differentiated interventions. The idea is to understand the economic basis for investing in gender specific interventions. From a baseline of close to zero in 2008, DIME has 37% of its IEs incorporating gender analysis in 2013. Results from the first IEs completed suggest that there exist significant differences in economic constraints and outcomes between men and women (e.g. Kondylis 2013 and McKenzie 2013).
In 2011 DIME started a program financed by the KCP called MEASURING INEQUALITY AND INEQUALITY OF OPPORTUNITY USING DIME MICRODATA. The development objective was to promote the use of high-quality distributional analysis within the context of impact evaluation of scaled programs in order to inform the intervention design and related policy decisions to improve the equity and social welfare impact of programs. For this purpose, data of two DIME-supported impact evaluations have been used in country-specific concept notes of equity analysis that generate evidence on gender inequality of outcomes. The evaluations are for a vocational training for vulnerable youth program in Malawi and for an irrigation and drainage project in Ethiopia. The gender equity analysis developed enabled task teams and policy makers to fine-tune interventions so that development resources are spent not only effectively but also in an equity-enhancing manner. Rigorous analysis of the distributional effects of development programs is scant. The impact evaluation in Malawi provides the first rigorous evaluation of a skills training youth employment program in Africa, and the gender equity analysis finds, unlike the programs in Latin America (where the limited evidence comes from), negative effects on female trainees. The gender equity analysis of the Malawi program provides evidence that these programs can have negative effects, analyzes the causes of failure of the program, and provides possible solutions for future interventions
In 2013, the IE of Financial Literacy in Brazil conducted an analysis of distributional impacts of test scores that revealed that the intervention shifted the whole distribution of test scores and benefited both low and higher achieving students, and both Bolsa Escola recipients and more well off students.
In DIME ongoing work distributional impact analysis is being incorporated systematically whenever data allows for adequate statistical inference.
Cost effectiveness. In the case of the SIEF, IE recipients are required to estimate cost effectiveness. As part of the SIEF program we expect 42 impact evaluations to conduct cost effectiveness analysis in the next 4-5 years. This could provide the basis of learning the value of these exercises.
Theory of change is being incorporated in the new Concept Note template for IE. From a baseline of close to zero in 2013, we expect the proportion of IE CNs that include a theory of change to increase steadily. This will also be a requirement for IEs to receive funding from the new IE umbrella facility and incorporated as a feature in the corresponding Grant Funding Requests.
Effectiveness of components of programs:
A critical feature of DIMEï¾s approach to IE is the combination of accountability (ï¾whatï¾) and learning (ï¾why/howï¾) that results from the way we conduct IEs. DIME evaluates the effectiveness of packages of interventions (ï¾whatï¾) and experiment with mechanisms (ï¾why/howï¾). The packages are the combination of policy interventions to address a specific objective (e.g. training program, registration reform, or agricultural technology program). The experiments with the mechanisms are designed to understand why a policy package works or how it could be improved. For instance, when people face limited-attention problems (a common behavioral bias), one mechanism is to send them reminders; another is to automatize their behavior. These mechanisms have been proven highly effective in increasing health-related take-up treatments, technology adoption, tax payments and the use of welfare-enhancing financial instruments.
In practice, the selection of mechanisms to be tested requires a careful analysis of the logic of each intervention and discussion with policymakers and implementers, who together with the IE team identify the problems and develop hypotheses as per which mechanism may help achieve the intended goals. The figure and table below present the distribution of IE questions in the DIME portfolio along a proposed classification of mechanisms.
DIME Impact Evaluations Questions by Type - % of IE Questions
Packages ï¾ 27%
Delivery Mechanisms ï¾ 21%
Constraints ï¾ 21%
Incentives ï¾ 12%
Behavioral biases ï¾ 9%
Demand-side accountability ï¾ 7%
Top-down accountability ï¾ 3%
On average each study answers 1.6 IE questions. Of these questions, 27% evaluate policy packages and 73% policy mechanisms classified as delivery mechanisms, constraints, incentives, behavioral biases and accountability.
Behavioral biases include limited attention, self-control, cognitive capacity, and understanding, and asymmetric valuation of gains and losses. These biases may exacerbate other problems, e.g., market failures/inefficiencies, but are also present in perfectly functioning markets. Mechanisms tested include choice simplification, default options, information, framing, demonstration, saliency, pre-commitment devices, and reminders.
Delivery mechanisms include targeting rules, centralized/decentralized modality of public service delivery, collective vs. individual or peer-to-peer, private sector vs. public sector delivery, paid vs. voluntary, and the use of media and technology.
Accountability mechanisms include top-down accountability (audits, inspections, supervision, performance assessment and feedback, laws and regulations) and demand-side accountability (information, report cards, user participation and monitoring).
Incentives conditional monetary and non-monetary incentives applied either to the supply or the demand.
Constraint-relaxing mechanisms target the range of possible constraints to capital formation and productivity including credit, financial, cash-flow, financial literacy and life skills, information, inputs, institutional, legal, tax burden, corruption, managerial, property, public goods, risk-management, skills, technology, and transaction costs.