Address the issue of timing of IFC's Advisory Services self-evaluation system to ensure project are sufficiently mature to assess more meaningfully their development results. In doing so, IFC might either consider conducting self-evaluation two to three years post completion, possibly on a sample of projects as done for XPSRs, or launching a post completion system based on clear selection criteria for projects to be included.
6. Capturing Advisory Services projects' outcomes and impacts.
- ï IEG could not assign Development Effectiveness ratings to 18 percent of projects selected for evaluation, in most instances (65 percent of the cases) because projects had not achieved results at the time of IEG evaluation and, in 35 percent of instances, because of insufficient information and lack of credible evidence. Moreover, even among those projects for which IEG assigned Development Effectiveness ratings, 41 percent could not be rated at the impact level because impacts had not been achieved by evaluation/project closure or because there was insufficient information and evidence to assign a rating.
- ï Given the limitations of PCR instrument to adequately assess outcomes and impacts at project closure, IFC may not count on sufficient evidence to systematically evaluate completed advisory services projects, and provide insights into the causal relationships between interventions and longer term results. IFC has attempted to capture longer term results through impact and other types of evaluations as well as through some ad hoc post completion monitoring efforts. However, this has not been done in a systematic way across IFC and is largely de-linked from IFC's self-evaluation system. Since FY10, Advisory Services has revised its project objective-setting approach to determine what is achievable within the project timeframe and budget, and stated that they are aiming to capture intermediate results of projects. This practice may be strengthened and supplemented by a systematic, sample-based post completion evaluation system, aimed at capturing impacts.
Agree: Amongst the many improvements to results measurement for AS activities in recent years, reforms have included measures to ensure project objectives are realistic, and at least the intermediate impacts of the projects are assessed. In tandem, Management has been reviewing the array of tools it has to ensure rigorous and cost-effective evaluation of longer-term results and impacts. This includes the new IFC evaluation strategy, and work towards an appropriate AS post completion monitoring system which we expect to roll-out by the end of CY13.
Action 6.1: Roll-out pilot of new post-completion monitoring system for AS.
Indicator: Launch pilot of new AS post completion monitoring system.
Baseline: NA.
Target: Pilot of new post completion monitoring system launched.
Timeline: by end of CY13.
Action 6.2: Continue to conduct evaluations and related studies.
Indicator: Number of AS related evaluations and related studies launched each year.
Baseline: Approximately 10 AS evaluations and related studies launched each year.
Target: At least 15 AS, or joint AS-IS, evaluations and related studies launched each year.
Timeline: by end of FY15.
No management actions. Based on past years' adoption records, IEG considers the recommendation to be made inactive.
IEG acknowledges rollout of Post Completion Monitoring and related evaluations were conducted. However, project results information may not be updated or revised once a PCR is approved. The Development Effectiveness Rating may not be changed, nor is a second Development Effectiveness Rating assigned during PIM. IEG rates this as "High." However, IEG notes that the activities have not yet addressed to a significant degree key areas of the recommendation, that is to address the issue of evaluating a project's development results after completion in a systematic fashion..
Update on 6.1 - Post Implementation Monitoring (PIM) for AS was completely rolled out in late CY13. A PIM flag has been active in the system and aligned to the indicators that have end dates beyond the project completion dates. A final decision of implementing the PIM for a project is discussed and determined at the PCR review meeting. A PIM questionnaire is also available in the ASOP. The questionnaire will be filled after 2 years of the project completion by the project committed to a PIM.
Update on 6.2 - 29 AS related evaluations and studies were launched in FY15. Of 29 evaluations, 10 were completed, and 19 are still active.
IEG notes the launch of PIM. However, creating a link between PIM and PCR or validation of self-evaluation is missing. IEG rates Medium as the activities are not to a significant degree in key areas of recommendation, that is to address the issue evaluating the projects' development results post-completion in a systematic fashion. IEG rates this as "Medium" because the activities have not yet addressed to a significant degree key areas of the recommendation, that is to address the issue of evaluating a project's development results after completion in a systematic fashion.
Post Implementation Monitoring (PIM) for AS projects was officially launched in October 2013 following the endorsement of the AS management on the PIM Guidelines. PIM has been implemented for (i) projects tracking at least one indicator beyond the project completion and (ii) a sample of 25% of completed projects annually that are identified based on the strategic criteria for which a PIM questionnaire with questions regarding sustainability and lessons will conducted 1-2 years after completion. As of the end of FY14, 59% completed projects were committed to the PIM and the PIM questionnaire was piloted with 18 projects closed in FY12.
In FY14, IFC launched 15 joint AS-IS evaluations and related studies, of which 9 joint evaluations are still active and 2 have been completed.