The private sector plays a critical role in promoting sustainable and inclusive development across the world. In view of its importance, a range of actors invest now in promoting the private sector in developing countries and regions. These actors span across the World Bank Group’s private sector arm, the International Finance Corporation (IFC), and risk guarantee arm, the Multilateral Investment Guarantee Agency (MIGA) and a growing number of impact investors, along with a host of bilateral, private and multilateral organizations in between. One question they all face, is how to measure the development outcomes of their investments.
One approach adopted by both IFC and MIGA, and in some form by several other investors, is to identify expected development outcomes ahead of time, and use them as the basis for a framework to monitor and evaluate the progress and outcomes of investment projects. As a result of these various efforts, there has been increasing focus on improving the measurement of development outcome and the harmonization of the indicators and methods used, which would facilitate the comparison of outcomes. It has also led to a better understanding of the types of development outcomes and the key stakeholders who benefit from investments to support the private sector.
The development outcome types are as varied as the range of investors. One typical outcome involves increased access to goods and services. Other frequently found outcome types include improved quality of goods and services or enhanced capacity of private sector stakeholders. There is also a variety in the level and complexity of outcomes, such as improved living standards of individuals and increased productivity of enterprises, although these higher-level outcomes are not widely and systematically tracked for private sector investments. These would include higher-level outcomes at the sector and country level, such as increased competition, and the improved function and integration of markets. Yet despite their diversity, there is one trait they all share for the moment: little is known as to whether certain outcome types are hard to achieve, and if so, which ones?
An analysis of outcome types of IFC projects
The 2021 Results and Performance of the World Bank Group report included an analysis of the varying degrees of difficulty in achieving different outcome types. The main purpose of the analysis was to assess the relationship between certain outcome types and evaluation ratings and to determine what ratings actually reveal about the outcome types, and if higher ratings necessarily meant higher outcomes.
For IFC, the assessment focused on a group of IFC projects which were evaluated between 2012 and 2019, and to which the IFC’s newly created ex-ante and monitoring framework -AIMM (Anticipated Impact Measurement & Monitoring) system - was retroactively applied. The findings indicate that outcome types related to higher level, market outcome are, in general, harder to achieve compared to project level outcomes or harder to measure. This is because the achievement of market-level outcomes depends on the broader market environment and external factors such as the government actions. It is important to mention that in the analysis, the outcomes are only considered achieved if there is sufficient evidence to verify their achievements. In this regard, market outcomes are particularly hard to measure because of their longer-term nature, lack of good indicators, and the relatively smaller impact of one specific project. Among market outcomes, those related to enhanced competitiveness, integration, and sustainability in the market had lower achievement rates – and evidence for such outcome is hard to be obtained.
Among the outcome types which would impact directly on project stakeholders, outcome types related to increased access to goods and services, particularly those related to micro, small and medium enterprises (MSMEs), are relatively hard to achieve, compared with outcome types related to the quality of goods and services or increased linkage with suppliers and distributors. For example, increasing access to goods and services for MSMEs requires expanding lending to MSMEs, enabling them to borrow from financial institutions, which is not entirely within the control of the project.
Some limitations of the analysis.
The above results should be taken with certain caveats. The assessment was based on the projects to which the AIMM system was retroactively applied, meaning the rigor that the framework should bring might not have been present at project approval. In addition, IEG’ assessments of the achievements do not necessarily coincide with the monitoring results based on the AIMM system, where different timing between the monitoring and evaluation could be one of the reasons for the difference. Therefore, along with ongoing efforts to understand the varying challenges presented by different outcome types, it would also be helpful to continue the assessment of different degrees of achievements per outcome type, especially when projects with original AIMM assessments are evaluated.
Another important aspect to be considered is the relationship between achievements of outcomes and evaluation ratings. The RAP assessment showed that harder-to-achieve development outcomes are not necessarily accompanied by lower evaluation ratings in the IFC’s projects, because the evaluation ratings do not only take into account the achievement of development objectives but other factors, such as industry benchmarks and unintended outcomes (For more detail, please see the RAP.) In some other evaluation approaches, on the other hand, the achievement of higher development outcomes that are aligned with the original project objectives may have higher weights in assigning the rating. Therefore, the relationship between the achievement of development outcomes and evaluation rating needs further analyses and further refinement of evaluation systems may be warranted.
Lessons for future approaches
With the above caveats in mind, the outcome type analysis has several implications:
- Factoring in risk: Understanding the different degree of achievements per outcome type would help in identifying the risks in achieving specific outcome types. If achieving access to finance for MSMEs proved to be more challenging, for example, there should be a recognition of higher risk of achieving such an outcome at project approval so that an informed decision can be taken, or mitigation actions taken. If an ex-ante scoring/rating tool incorporates risks for achieving outcomes, it would benefit from the information on the difficulty of achieving certain outcome types.
- Ratings/scores for development outcomes: Currently, there are various approaches to assigning scores and ratings for the expected and actual development outcome. In the approaches taken by IFC’s AIMM system, the ratings and scores are based on the magnitude of a project’s contribution to address a country’s development challenges. For example, power generation capacity added by the project is assessed against the country’s power deficit. However, in most cases, rating or scoring mechanisms are not necessarily differentiated by the type of outcome and their level of challenges. For instance, expanding access to finance for MSMEs would not be assigned a higher weight than improving the quality of the access to finance for MSME. Adding these aspects into the rating/scoring mechanism might lead to ratings and scores that offer a more comprehensive picture of the outcomes assessed.
- Enhancing results measurements: As mentioned before, the analysis of outcome achievement identified a higher share of outcomes for which properly measuring the development outcome is a challenge. This is particularly the case for market outcomes, where nearly a third of outcomes lacked proper evidence. Results measurements have been more advanced for outcome types which are considered more immediate or direct outcomes, such as increased direct employment. Further efforts are needed for the effective monitoring of higher-level outcomes, which are related to the living standards or productivity of intended beneficiaries as well as market level outcomes. The ongoing efforts promoted through AIMM in IFC is encouraging, as it identifies clear outcome claims and related indicators and monitor them. This is expected to enhance result measurement overall and allow better identification of different degrees of challenges to achieve certain development outcome types. In addition, further harmonization or alignment of ex-ante and monitoring frameworks among development finance institutions and impact investors would allow comparison and benchmarking of the results achieved by different players, further enabling better understanding of the nature of development outcomes and the impact of investments in the private sector.
Add new comment