Streamline the project-evaluation approach and process to align more closely with MIGA's business model and conditions on data gathering.
Reduce cost burden on project evaluation, possibly by strengthening periodic collection of project data in line with industry practices.
Increase coverage of evaluated projects in order to enhance the ability to derive meaningful results at the corporate level.
MIGA 10. Remaining gaps in MIGA's self-evaluation - ï MIGA has mainstreamed self-evaluation of its guarantees and has strengthened some aspects of its project monitoring. However, the coverage of MIGA projects through self-evaluation can be strengthened to enhance the ability to assess MIGA's overall development performance. As a development institution, MIGA should be able to know the development effectiveness of its portfolio.
As FY15 update, MIGA made rationalization of self-evaluation process and procedures. Based on this, IEG considers to retire this recommendation. However, as mentioned in the FY15 comment, IEG still undertakes a substantial part of project evaluation by covering the cancelled projects. This is rather misaligned with the other World Bank Group institutions which conducts self evaluation with IEG validation as the norm.
IEG acknowledges MIGA's progress on streamlining PER. It is important that MIGA's streamlining of self-evaluation should not undermine the quality and rigor of assessment of MIGA's PERs.
On cost saving, MIGA is seeking efficiencies and is changing field visit policies aimed at reaping mission cost savings. This is positive and certainly aligned with IEG's recommendation to reduce the cost burden of self-evaluation. It is not clear, however, if these efficiencies also entail the number of MIGA staff working on each self-evaluation.
MIGA now covers the 100% of the mature projects for evaluation. However, IEG still undertakes a substantial part by covering the cancelled projects. This is rather misaligned with the other World Bank Group institutions which conducts self evaluation with IEG validation as the norm.
- During FY15, MIGA sustained the progress in streamlining project evaluations achieved during FY14. MIGA's FY15 self-evaluation program consisted of project evaluation reports (PERs) for eight projects (i) National Cement Share Company, Ethiopia: (ii) Jordan Bromine Company, Jordan (iii) Turkmenistan Coca-Cola Bottlers Ltd., Turkmenistan (iv) Hitachi Construction Machinery, Zambia (v) ProCredit Group, World (vi) OrPower 4, Inc., Kenya (vii) Khauzak-Shady-Kandym, Uzbekistan (viii) Kadikoy-Kartal-Kaynarca Metro Project, Turkey. For all self-evaluations, the assessments were focused on key issues related to project outcomes, achievement of objectives, and ratings justification. As a result of the streamlining measures, MIGA has succeeded in shortening the length of PERs to 8-10 pages on an average. In the case of the Joint IFC-MIGA ProCredit Group project, MIGA's PER Team worked closely with the IFC Team (IFC is yet to undertake a self-evaluation of the project). MIGA also relied on the IFC repository of documents as a source of information for the self-evaluation. In the case of the Kadikoy-Kartal-Kaynarca Metro Project, MIGA self-evaluation was informed by IFC support for the earlier phase of the project (though not a formal Joint IFC-MIGA project) as well as IEG evaluation of the cancelled guarantee for the earlier phase. In addition, the self-evaluation featured the piloting of the draft evaluation guidance for NHSFO (Non-Honoring of Sovereign Financial Obligations) projects that was crafted by a joint MIGA-IEG Working Group. The streamlining of project evaluations has also been strengthened by process improvements by way of self-evaluation teams engaging with IEG and the original underwriting teams on a consistent basis. Finally, the FY15 self-evaluations were also guided by a Tip Sheet that was developed for MIGA self-evaluation teams for the express purpose of streamlining PERs, based on comments and feedback from previous self-evaluation teams.
- With regard to reducing the cost burden on project evaluations (second part of the recommendation), the streamlining initiatives in FY15 addressed this objective as well. Most of the self-evaluations were undertaken by two-person teams. In the case of both Jordan Bromine and ProCredit Group PERs, the self-evaluation teams relied on information gathered through phone discussions and electronically with the guarantee holders and project enterprises, but without undertaking self-evaluation missions. In the case of the Khauzak-Shady-Kandym PER, there was only an E&S field visit that was also used for gathering relevant information for the self-evaluation. The National Cement Share Company (Ethiopia), Hitachi Construction Machinery (Zambia) and OrPower 4 (Kenya) PERs were task-led by one MIGA Staff and hence benefited from the cost savings of one self-evaluation mission that covered all three projects. Going forward, MIGA may also use relevant project-level data from the Development Effectiveness Indicator System (DEIS) as the DEIS coverage is ramped up and the system is stabilized.
- With regard to the third part of the recommendation, since FY12, MIGA and IEG have agreed on full (100%) coverage of MIGA guarantee projects. This practice was implemented for the FY15 MIGA evaluation program as well, with MIGA undertaking eight self-evaluations for the very first time since the inception of the self-evaluatiion pilot in FY10.
IEG notes increased coverage of evaluation (Item 3). On Items 1 and 2, in May 2014, IEG and MIGA joint working group completed the review of MIGAâs project evaluation program that identified cost-saving areas by partnering with other WBG information as well as benefitting from other MIGA information gathering. MIGA management accepted the recommendations including simplification/streamlining of MIGA project evaluations, including focusing of self-evaluation to key issues, achievement of objectives, and justification of ratings, as well as shortening the length of the self-evaluation report. Recent PERs reflected these changes. IEG rates âSubstantialâ as MIGA made progress in follow-up actions.
MIGA has made significant progress in streamlining project evaluations during FY14. MIGA's FY14 self-evaluation program consisted of four projects: (i) West Africa Gas Pipeline (ii) Cotecna Togo (iii) Cotecna Congo and (iv) ProCredit. In the case of the ProCredit project, MIGA has produced a Methodological Note for evaluating the Capital Optimization product that will inform the FY15 self-evaluation of the ProCredit projects on a programmatic basis, covering 14 ProCredit projects approved in FY11 under one master contract and four additional projects in FY12. With regard to the other three projects, the self-evaluations have been were focused on key issues related to project outcomes, achievement of objectives, and ratings justification. As a result of the streamlining measures, MIGA has succeeded in shortening the length of PERs to 8-10 pages. In the case of the Joint WB-MIGA West Africa gas pipeline project, MIGAâs PER and WBâs ICR Teams worked closely and undertook a joint evaluation mission in Ghana, the portion of the project covered by the MIGA guarantee. MIGA also relied on the WB team for the E&S monitoring of the project including the E&S assessment for the PER. In the case of the Cotecna Congo and Togo PERs, the evaluations were done in conjunction with a due diligence mission in Congo for another project. Also, the E&S assessment for the PER was based on information gathered during E&S monitoring missions for the two projects earlier.
With regard to the second part of the recommendation, the streamlining initiatives in FY14 discussed above were also intended to reduce the cost burden on project evaluations. In the case of the West Africa Gas Pipeline project for instance, MIGA relied on WB primarily for information related to the self-evaluation, given the Joint WB-MIGA nature of the project. In case of the Cotecna projects, previous E&S monitoring missions, as well as MIGAâs familiarity with the scanning sector and long-term partnership with Cotecna, helped to save costs for the self-evaluations. Going forward, MIGA may also use relevant project-level data from the Development Effectiveness Indicator System (DEIS) as the DEIS coverage is ramped up and the system is stabilized.
With regard to the third part of the recommendation, since FY12, MIGA and IEG have agreed on full (100%) coverage of MIGA guarantee projects.