Develop and adopt explicit evaluation protocols for piloted interventions to capture lessons from experience on poverty reduction, with a view toward opportunities for scaling up successful interventions.
Pilots are used to strengthen the design, implementation, and scaling-up of projects and to enhance the poverty focus of the Bank's projects. Early intermediate outcomes can attract additional resources leading to scaling up. The impact of Bank support for poverty reduction will depend in part on successful use of pilots to leverage and crowd-in external resources to scale up. However, evidence on whether pilots are successful and serve to leverage non-Bank resources is rare.
Action 8a: Toolkit for distributional impact analysis.
Indicator: Toolkit issued and disseminated.
Baseline: No comprehensive source of know-how exists.
Target: a) Working Paper prepared, b) Internal workshop to disseminate the handbook internally.
Action 8b: Improve monitoring of pilots designed for scaling up; and capturing lessons learned from these pilots.
Indicator: Update ï¾Results Framework and M&E Guidance Note to provide advice on monitoring of pilots designed for scaling up; and update ICR guidance note to capture lessons learned from these pilots.
Baseline: No guidance on monitoring of such pilots or systematic capture of resulting learning
Target: Update ï¾Results Framework and M&E Guidance Note and ICR guidance note.
Action 8a: Management indicates that a draft toolkit exists, although this was not attached so it was not possible to verify to what extent it is helpful for distributional impact analysis. It has not been yet issued and disseminated.
Action 8b: No response has been provided by management. As IEG highlighted when the action plan was under preparation, an essential step for the implementation of this recommendation involves setting up a repository for monitoring pilots, in order to be able to identify them and document their results.
The first draft of the Toolkit has been prepared. The toolkit covers the distributional consequences of programs/policies due to participants' heterogeneous responses or participation decisions. In particular, it examines features beyond the gross total gain of a program by studying where the gains/losses of a program - if any - were produced, or who wins or loses as a result of program participation. Sections include: (i) Questions of Interest and Definitions; (ii) Impact on the Outcome Distributions for RCTs with full compliance and RTCs with less than full compliance; (iii) Distribution of Treatment Impacts incl. Bounding the Distribution of Treatment Impacts and Point Identification of Features of the Distribution of Treatment Impacts; (iv) Subgroup/Conditional Analysis (Subgroup Means and Machine Learning); (v) Applications; (vi) Practical Issues incl. Sampling/Power, Inference and Multiple Hypothesis Testing; and (vii) Appendices and Code Samples.
The Toolkit will be circulated for comments in August 2016 and planned to be submitted to the Working Paper series in the Fall.
The internal training will be delivered late Fall or early Spring.
8b. Management has undertaken steps towards reforming the ICR, which is expected to be completed by the end of FY17. Once the decision is made how to reform the ICR's content and process, a new ICR guideline will be developed and the M&E Guidance Note will also be updated.ï¾
WB: Agrees. Each piloted intervention, particularly those that aim to reduce poverty or enhance shared prosperity, should develop a well thought-through process and protocols for evaluating its impact and create the evidence base for scaling-up.