IFC should conduct selective tests and reviews to validate information provided by clients. For unaudited information, selective direct data verification is needed to enhance the credibility and reliability of data supplied by companies. Any assumptions and data limitations or biases should be publicly disclosed.
The external assurance provider's mandate should be expanded to include assessment of the credibility and attribution of data particularly related to IDGs appropriate verification, and whether IFC is effectively disclosing data limitations or biases.
3. Enhancing the credibility of DOTS indicators.
- ï DOTS indicators are based in part on data from audited financial reports, company annual reports, and other validated sources. However, other data are based on assumptions by client and IFC staff, and IFC does not have a process to verify data integrity other than through a desk review of information received. IFC has pioneered external assurance of its development results reporting. However, this review has been limited to a small portion of the information only ex post micro, small ,and medium Enterprises loan data are externally assured that is, only 1 of 15 measures of Development Reach by IFC investment clients. Moreover, similar to IFC's internal CDI quality control, the assurance provider's review does not include contacting clients, visiting projects, or communicating with field-based staff. Direct data verification for some data that are based on less credible sources would enhance the credibility and reliability of data supplied by companies and staff, and any assumptions and data limitations or biases should be publicly disclosed.
Agree on selective data validation
Disagree on expanding the external assurance provider's mandate
As the BROE report pointed out, IFC is the leader among MDBs with regards to tracking and monitoring development results. IFC has continued to refine our quality review process and further strengthened the results measurement network, in order to improve staff understanding of development impact as well as the data quality.
Over the last fiscal year, IFC has conducted a pilot verification of AS outcome data quality in the Africa region. We will incorporate lessons learned from that experience. In FY14, we will implement a similar verification process of a sample of IS projects to assess any data quality issues on the investment side.
We disagree with the recommendation to expand the mandate of the external assurance provider to include direct assessment of the credibility and attribution of results. We see no evidence that it is necessary and it would be more costly to clients. We already have stringent data quality controls in place through different levels of data scrutiny including:
1) Portfolio officers collect and enter data into DOTS and upload the sources of information in iDesk
2) DOTS champions and results measurement staff review data quality within a deadline given by CDI
3) CDI provides weekly update of reach data quality focusing on year -on-year variations (large increases/decreases or no variations), big contributors, zero values, and accuracy of gender data component
4) CDI reviews DOTS ratings and indicators quality against IFC's official guidelines as well as reach data quality and
5) DOTS champions request portfolio officers to make any changes and to provide any additional information that has been requested through this thorough review process.
We are additionally undertaking pilot quality control tests, in FY13 and FY14, and will then determine whether we need any additional data quality controls. At that point, we will consider asking the external assurance provider to review the appropriateness of these additional control processes. We do not consider it necessary or helpful to ask the assurance provider to undertake direct assessments of the quality of the data itself - in effect, this would be a third or fourth layer of checking - which would create unnecessary additional intrusion for IFC's clients.
Background: Subsequent to the issuance of the Management Response in February 2013 and in the context of the World Bank Change Process, IFC is changing its IS monitoring system to be more client-centric. Once the new approach is implemented, we plan to verify data quality by conducting a data verification pilot accordingly.
Action 3: See Action 1
Based on "Independent Assurance Report on a Selection of Sustainable Development Information" in the IFC Annual Report 2017, "Reach indicators are directly collected from clients and data received can correspond to data that differ from IFC indicators definitions. IFC enhanced its internal controls on the data received, especially from major contributors to the Reach numbers, by comparing, for instance, reported data to publicly disclosed information, when available." Thus, some enhancement of quality control was observed. Also, the independent assurance provider noted significant reduction in the number of extrapolations used for micro-loans and SME-loans indicators. At the same time, the provider commented that the use of country averages in assessing the final number of beneficiaries (i.e. average number of people per households) to be refined in order to better reflect the evolution of demography at country level.
There is no management update on this recommendation. In FY16 IFC Annual Report, Ernst & Young, the external assurance provider pointed out that "clients may provide requested data using
definitions that differ from IFC definitions. Significant divergence has been observed in the students reached indicator results and corrected to align with the IFC's definition. IFC should continue to conduct quality checks in order to ensure that the data reported is consistent with its own definitions and calculation methodologies." Moreover, while E&Y recognized improvement in IFC methodology, it voiced that "IFC should consider limiting the use of extrapolation for micro-loans and SME loans Indicators where data from clients is not available. The extrapolation for a new top CY15 contributor indeed led to a higher share of the reach data extrapolated this year. " Because of the remaining substantial areas of weakness in quality of publicly disclosed development indicators, IEG considers this recommendation is still relevant and urges management to continuously focus on improving reliability of its reporting.
IEG welcomes evaluations by IFC, which provided opportunities to validate data received from clients. IFC findings are very significant about the reliability of the data supplied by the clients, implies clients may consistently exaggerated labor productivity or wage per labor, by under-reporting direct number of employees. It also raises concern about reliability of other reporting, for example, whether the companies are honest about labor related payments such as social security contributions. These disconnects indicate there is a need for third-party verification of DOTS data supplied by the client companies. Based on this, IFC may wish to reconsider adopting the part of original IEG recommendation categorically rejected by the IFC management âexpand the mandate of the external assurance provider to include direct assessment of the credibility and attribution of results.â Moreover, as some XPSR project evaluations pointed out, there are possible trade-off between quantity and quality of labor, and IEG noticed some companies which had high employment under the IFC evaluation also had deteriorating working conditions (in terms of compliance to the Performance Standard 2).
As mentioned in the previous year's response, IFC conducted data verification exercises as part of the Tourism and Poultry evaluation, with inquiries on numbers that are reported in DOTS such as number of direct jobs, female jobs, and number of farmers reached. The results are indicated below:
For Tourism study which includes three hotels: one of which (Mali) was closed so no DOTS data was available for comparison
For Ghana (KHI #30029), in DOTS total employment was 350 and 150 for female employment while from our questionnaire, total employment was 300 and female employment was 113.
For Maldives (Maldives SL #25498), in DOTS total employment was 475 and 108 for female employment while from our questionnaire, total employment was 583 and female employment was 81.
The poultry evaluation found that a much larger number of indirect jobs are supported by a poultry producer in its suppliers and clients than the direct jobs provided by the company. In addition to the 8,000 jobs directly supported by PRONACA's poultry supply chain, another 24,000 indirect and induced jobs are associated with the company's activities. SIPRA supports a total of 1,100 direct jobs, but for every worker directly employed by the company and its contractors, an additional 3.6 will be employed in the rest of the economy. Jobs supported by and associated with PRONACA and SIPRA's poultry operations are quite diverse ranging from unskilled to highly skilled, and also provide important employment opportunities for women, particularly in processing. The evaluation suggests that employment figures reported in IFC's Development Outcome Tracking System (DOTS) vastly understates actual employment because it does not consider employment through companies closely associated with a client through contracts, nor indirect and induced impact.
As a result, we have concluded that there is no concrete data issue in DOTS system. As noted above, DOTS actually understates total employment. Given the burden these tasks are for IFC's clients together with our stringent data quality review process, we suggest that this recommendation is considered as fully implemented and to be made inactive for next year.
IEG acknowledges that piloting is planned and, once achieved, is likely to influence future ratings. IEG also noted the comments made by the Independent Assurance Report in the 2014 IFC Annual Report stating "additional tests on a few major contributors of the Reach data should be performed to increase data reliability." Plans and actions for adoption are in a very preliminary stage and concrete plans are only forthcoming, thus IEG rates progress as Medium.
Development Impact department has continued to implement stringent data quality controls during the portfolio reviews and to have an external assurance provider to review development data. With additional regional results measurement staff and greater involvement from industry strategists/economists, 2014 data was even more scrutinized.
However, with internal changes and the proposal of improvements to IFC's results measurement, CDI plans to conduct data verification pilot for investment projects in FY15.