Organization
World Bank
Report Year
2015
Accepted
Yes
Status
Active
Recommendation

Develop and adopt explicit evaluation protocols for piloted interventions to capture lessons from experience on poverty reduction, with a view toward opportunities for scaling up successful interventions.

Recommenation Adoption
IEG Level of Adoption by Year: mar-rating-popup NTNNTNT Management Adoption by Year: mar-rating-mng-popup NTNNTNT
CComplete
HHigh
SSubstantial
MModerate
NNegligible
NANot Accepted
NRNot Rated
Findings Conclusions

2016

Original Management Response

Pilots are used to strengthen the design, implementation, and scaling-up of projects and to enhance the poverty focus of the Bank's projects. Early intermediate outcomes can attract additional resources leading to scaling up. The impact of Bank support for poverty reduction will depend in part on successful use of pilots to leverage and crowd-in external resources to scale up. However, evidence on whether pilots are successful and serve to leverage non-Bank resources is rare.

Action Plans
Action 1
Action 1 Number:
0376-01
Action 1 Title:
Action 8a: Toolkit for distributional impact analysis.
Action 1 Plan:

Action 8a: Toolkit for distributional impact analysis.
Indicator: Toolkit issued and disseminated.
Baseline: No comprehensive source of know-how exists.
Target: a) Working Paper prepared, b) Internal workshop to disseminate the handbook internally.
Timeline: FY17

Action 2
Action 2 Number:
0376-02
Action 2 Title:
Action 8b: Improve monitoring of pilots designed for scaling up and capturing lessons learned from these pilots.
Action 2 Plan:

Action 8b: Improve monitoring of pilots designed for scaling up; and capturing lessons learned from these pilots.
Indicator: Update モResults Framework and M&E Guidance Note to provide advice on monitoring of pilots designed for scaling up; and update ICR guidance note to capture lessons learned from these pilots.
Baseline: No guidance on monitoring of such pilots or systematic capture of resulting learning
Target: Update モResults Framework and M&E Guidance Note and ICR guidance note.
Timeline: FY17

Action 3
Action 4
Action 5
Action 6
Action 7
Action 8
IEG Update:
No Updates
Management Update:
No Updates
IEG Update:
No Updates
Management Update:
No Updates
IEG Update:
No Updates
Management Update:

Action 8a: Management indicates that a draft toolkit exists, although this was not attached so it was not possible to verify to what extent it is helpful for distributional impact analysis. It has not been yet issued and disseminated.

Action 8b: No response has been provided by management. As IEG highlighted when the action plan was under preparation, an essential step for the implementation of this recommendation involves setting up a repository for monitoring pilots, in order to be able to identify them and document their results.

IEG Update:

The first draft of the Toolkit has been prepared. The toolkit covers the distributional consequences of programs/policies due to participants' heterogeneous responses or participation decisions. In particular, it examines features beyond the gross total gain of a program by studying where the gains/losses of a program - if any - were produced, or who wins or loses as a result of program participation. Sections include: (i) Questions of Interest and Definitions; (ii) Impact on the Outcome Distributions for RCTs with full compliance and RTCs with less than full compliance; (iii) Distribution of Treatment Impacts incl. Bounding the Distribution of Treatment Impacts and Point Identification of Features of the Distribution of Treatment Impacts; (iv) Subgroup/Conditional Analysis (Subgroup Means and Machine Learning); (v) Applications; (vi) Practical Issues incl. Sampling/Power, Inference and Multiple Hypothesis Testing; and (vii) Appendices and Code Samples.

The Toolkit will be circulated for comments in August 2016 and planned to be submitted to the Working Paper series in the Fall.

The internal training will be delivered late Fall or early Spring.

8b. Management has undertaken steps towards reforming the ICR, which is expected to be completed by the end of FY17. Once the decision is made how to reform the ICR's content and process, a new ICR guideline will be developed and the M&E Guidance Note will also be updated.ヤ

Management Update:

WB: Agrees. Each piloted intervention, particularly those that aim to reduce poverty or enhance shared prosperity, should develop a well thought-through process and protocols for evaluating its impact and create the evidence base for scaling-up.