Back to cover

The Development Effectiveness of the Use of Doing Business Indicators

Chapter 3 | The Effectiveness of Doing Business: Is It “Doing Things Right”?

Highlights

Doing Business (DB) reports make many claims linking reforms measured by its indicators to outcomes, such as job creation and economic growth. Only a minority of these claims can be confirmed through articles published in leading journals, and a smaller share have been replicated or confirmed through a rigorous methodology. Weakly evidenced claims can create a reputational risk for the World Bank Group and its Development Economics Vice Presidency.

Although many countries and Bank Group projects use movement in DB indicators as an outcome measure, such movements are inconsistently linked either to reform implementation or to economic outcomes, such as increased investment, employment, or productivity. The Independent Evaluation Group’s country case studies show both strong movement of DB indicators and tenuous links to measurable development outcomes.

Three-quarters of evaluated country strategies with DB objectives mostly or wholly achieved those objectives, but only 45 percent also showed improvements in DB indicators. In some countries, impact was limited by a failure to address binding constraints.

The Bank Group DB-informed project portfolio is generally successful in achieving project objectives but less so in three areas that require deep institutional reforms (enforcing contracts, registering property, and resolving insolvency). Although client commitment and capacity are important, most factors of success—including complementary analytic work, client engagement and follow-up, effective coordination, and monitoring and evaluation—are largely within the Bank Group’s control.

Bank Group project experience shows that effective practices include a focus on binding constraints; use of a strong interagency coordination unit; timely availability of expertise; long-term, comprehensive engagements; public-private dialogue; client capacity building and knowledge sharing; and peer-to-peer learning.

This chapter examines the extent to which activities informed by DB are yielding intended objectives and results—their immediate and intermediate outcomes. This evaluation is ultimately concerned with the development effectiveness of how DB is used by client countries and the Bank Group—the extent to which program, policy, and project reforms achieve their objectives and results, systemically and for specific beneficiary groups. A key question concerns how much benefit an improvement measured by DB yields in terms of actual results and development outcomes such as increased growth, investment, business entry, productivity, or employment.

Research Evidence

IEG’s desk-based review of literature identified by the DB team finds significant associations of what DB indicators measure with outcomes but does not establish causation. The desk review took stock of evidence on DB effectiveness based on a database of 426 articles from 100 leading journals. The database was organized by indicator. Some areas had abundant coverage (starting a business, protecting minority investors, and trading across borders), but many others had much lighter coverage (for example, DWCP and getting electricity). The desk review did not assess the methodological rigor of each article but did consider whether the article shed light on the relationship between what a DB indicator measures and an outcome, thus limiting its reporting to 75 articles of strong relevance. Findings confirmed by multiple articles were considered to have “strong evidence” (box 3.1). Where the findings of two or more articles contradicted each other, the evidence was classified as “mixed.” On this basis, very few findings were confirmed by multiple articles.

IEG conducted a second rigorous SLR, which found that although there is a reasonable depth of coverage of some indicators, others have sparse evidence causally linking outcomes to reforms tracked by DB. To capture a broader body of literature and understand rigorously established relationships between DB-related action and outcome, the SLR followed methodological criteria consistent with norms established by leading practitioners of systematic reviews (appendix F) to identify studies in English meeting its inclusion criteria for populations, interventions, comparison groups, outcomes, and study designs. The SLR cast its net broadly, initially identifying 9,221 studies from multiple data sources that pertained to the 10 indexed DB indicator areas. After applying the filters for inclusion, the SLR identified 103 studies of relevant reforms meeting its criteria, including that allocation to intervention and control groups was random, and selection bias had been addressed by design (figure 3.1).

Box 3.1. Four Doing Business Outcome Findings Evidenced by Multiple Articles Published in Leading Journals

Starting a business: Higher entry costs or more steps or documents are associated with less firm creation, growth, or profitability.

Getting credit: Stronger legal rights are associated with more lending, more financial activity, or lower interest rates.

Paying taxes: Higher tax rates constrain entrepreneurship (rate of business creation) and investment rate.

Trading across borders: Better trade facilitation as measured by Doing Business shows a correlation with increased trade flows.

Source: Independent Evaluation Group, desk review of Doing Business literature database..

The SLR confirms several of the relationships between the laws or regulations tracked by DB and significant development outcomes, while also raising some cautions and identifying gaps. In general, research findings do not necessarily use DB’s definition or base case to measure critical phenomena. For example, there is evidence that low business entry costs encourage firm entry or formalization and growth, but not all findings are connected to DB’s specific formulation of entry. Sparse evidence confirms that the accessibility of land services and the process of transferring property is positively related to increased activity in commercial rental and property markets and access to credit. Mixed evidence generally confirms (with considerable nuance) a relationship of improved credit information and more expansive collateral laws to increased credit to private firms. Evidence supports the relationship of tax administration reforms to enhanced tax compliance and reduced business perceptions of tax administration as a constraint. Evidence supports a positive relationship between trade facilitation reforms and trade flows. Evidence on contract enforcement supports a general relationship of improved contract enforcement and judicial efficiency to broader outcomes such as investment, productivity, and profits; yet this evidence does not link to the specific good practices advocated in DB. Evidence on insolvency supports the idea that improvements in the efficiency of insolvency can lower the cost of borrowing and enhance private firm access to credit.

Figure 3.1. Rigorous Studies of Outcomes Identified by Structured Literature Review Relating to Indexed Doing Business Indicators

Image

Figure 3.1. Rigorous Studies of Outcomes Identified by Structured Literature Review Relating to Indexed Doing Business Indicators

Source: Independent Evaluation Group, structured literature review.

The SLR did not find empirical evidence supporting outcomes linked to the DB measures of DWCP (construction permits) or of getting electricity, and it raises some cautions about unintended consequences of reforms encouraged by DB. The cautions arise from findings in the literature on the potential for (i) gender and environmental consequences of simplified business entry; (ii) “getting credit” reforms to increase the exclusion of some borrowers; (iii) protection of minority shareholder rights to increase the cost of equity, debt, and audit fees; (iv) efficiency/ quality trade-offs in judicial efficiency; and (v) insolvency reforms to promote capital intensity with potential labor market effects. On trade facilitation, there is a lack of robust evidence about the implications of reforms for other public objectives (for example, public health, safety, the environment, and reducing informality).

Effectiveness Claims Made in Doing Business Reports

DB reports frequently associate the DB reform agenda with the Bank Group’s institutional twin goals of ending extreme poverty and promoting shared prosperity. The association of DB reforms with reduced poverty and enhanced prosperity can be found in even the earliest DB report but was quite explicit in DB2019, in which then–World Bank president Jim Young Kim stated: “The reforms that the report inspires will help people reach their aspirations; drive inclusive, sustainable economic growth; and bring us one step closer to ending poverty on the face of the earth” (World Bank 2018b, v). Earlier, in DB2017, the World Bank’s then chief economist stated, in relation to the twin goals, “Doing Business helps us make progress on one crucial strategy for meeting these goals—offering market opportunities to everyone” (World Bank 2016c, vii). DB2010 (and others) linked its measured reforms with the ability of people with lower incomes to find jobs and escape poverty.

IEG finds that DB reports make claims that improving the legal and regulatory conditions for businesses as summarized by DB’s general index will benefit a variety of development outcomes. To better understand claims for explicit links of reforms tracked by DB with development outcomes, IEG used supervised machine learning and human review to identify claims about outcomes in DB reports issued from 2010 to 2020. This review identified 89 affirmative claims linking improvements in what the DB indicators measure to better development outcomes (or, conversely, worse DB-tracked conditions with worse development outcomes). Of these, 23 (26 percent) concerned the general association of improvements in DB areas (including the overarching EoDB indicator and distance to frontier) with improvements in outcomes. Like the claims in the previous paragraph, such claims do not map a reform path, but focus instead on outcome claims.

Of 66 remaining claims pertaining to a specific indicator area of DB, most (51 percent) did not reference the immediate outcome of the claim, leaving the causal links to the described outcome or impact undefined. Of those that did define an immediate outcome, they were most commonly in the categories of reducing the cost of doing business, followed by streamlining of procedures and reducing days to complete procedures. The overwhelming majority of all the 89 claims identified (68) did refer to intermediate outcomes, led by an increase in business entry (33 percent of claims), an increase in investment (30 percent), or an increase in formality (20 percent).

IEG also identified within the 89 claims 97 asserted links associating DB-related reforms with identifiable development impacts, led by job creation and economic growth (figure 3.2).1 As an example, DB2015 states that “research provides strong evidence that reforms making it easier to start a business are associated with more firm creation, which in turn is strongly associated with job creation and economic growth” (World Bank 2014a, 33). Impact claims of increased job creation were most commonly associated with intermediate outcome claims to increased business entry, formality, and investment. Impact claims of economic growth were mostly linked to intermediate outcome claims regarding increased business entry, with a minority also related to claims about increased formality and investment.

Although some of the claims contained in DB reports meet a high standard of evidence, most claims do not, which can create a reputational risk to the Bank Group. IEG found that only 13 percent of articles cited as evidence used robust methods and study designs meeting the criteria of the SLR, and 10 percent of results cited had been replicated in multiple articles in DB’s own literature database.2 If claims for evidence of outcome made in the report are not held to a consistent and high standard, or if they are cited selectively only to support the case for reform, the risk is that the DB report, DEC, and the Bank Group may be increasingly regarded as advocates more than as trustworthy interpreters of evidence (box 3.2). Although the standards need not be as rigorous as those used in IEG’s SLR, the principle is that transparent and systematic reporting of the available evidence with attention to nuance and complexity is necessary to guard against potential bias, oversimplification, overgeneralization, and reputational risk.

Figure 3.2. Claims Linking Doing Business–Related Reforms to Development Impacts

Image

Figure 3.2. Claims Linking Doing Business–Related Reforms to Development Impacts

Source: Independent Evaluation Group, analysis of DB reports 2010–20.

Note: Bar labels denote percent of all claims..

Box 3.2. Comparing the Doing Business 2020 Literature Review with the Independent Evaluation Group Structured Literature Review

The second chapter of Doing Business (DB) 2020 presents a literature review citing 53 articles that together cover all DB areas and were published between 2013 and 2019. Of the articles, 90.5 percent supported the arguments of DB by confirming the relevance of the DB reforms’ agenda or finding that positive results associated with specific undertaken reforms were consistent with what DB indicators reward. (The Independent Evaluation Group’s structured literature review [SLR] found that 85 percent of rigorous articles are associated with positive benefits of reforms in terms of an economic or development outcome.) Nevertheless, these articles mostly did not meet the criteria of the Independent Evaluation Group’s SLR. Of 35 articles overlapping with the coverage of the SLR, only 4 were included. These four related to the relationship of customs delays to export volume, of contract enforcement to entrepreneurship among individuals with higher levels of education, of credit information to loan defaults, and of the improved legal rights of creditors to lending activity and credit terms.

Source: Independent Evaluation Group, desk review of Doing Business 2020 (World Bank 2019a).

Country Reforms

Although many countries, donors, and Bank Group projects use movement in DB indicators as an outcome measure, such movements are inconsistently linked both to reform implementation and to economic outcomes. A review of IEG’s 10 case studies reveals strong movement of DB indicators in most countries but tenuous links to measurable development outcomes. Some countries show little increase in investment, employment, or productivity. Overall, IEG finds a mixed picture of indicators’ links to country outcomes (appendix C):

  • In Morocco, despite both an improvement in ranking from 128th of 183 countries in EoDB in 2010 to 53rd of 190 countries in EoDB in 2020, and the utility of the indicators in setting concrete targets, a key World Bank economic expert interviewed observed “the impact gap between advances in [DB] rank and growth, employment and productivity. … Growth slowed. Job growth slowed.” Nor did investment clearly respond to improved indicators. A World Bank private sector development expert interviewed by IEG observed, “We don’t see a correlation between movement up in the [DB] ranking and the daily life of enterprises.”
  • In India (which moved from 133rd in DB2010 to 63rd in DB2020), business groups and foreign investors interviewed stated that they did not feel the benefits of strong movements in the DB indicators. One complaint was that the scope of DB reforms did not address a number of binding constraints, ranging from input markets (land, labor) to bureaucracy to constrained competition.
  • For Rwanda, which moved from 67th in DB2010 to 38th in DB2020, the persistent structural constraints cited in the Country Program Evaluation (CPE) explained why, despite many measured reforms, gross domestic product per capita had not responded as intended and the Vision 2020 objective of becoming a middle-income country was not achieved (World Bank 2019c). Persistent structural constraints narrowly concentrated the benefits of economic growth and major DB achievements. They included a “complex political economy environment … limiting fair competition and effective implementation of regulatory reforms … with companies closely affiliated with the government, the ruling party, and the military playing a dominant role in the private sector” (World Bank 2019c, 3). As a result, despite reforms, there was limited development of manufacturing, diversification of exports, and development of financial and services sectors. This disconnect was apparent in access to credit, where there were seven recorded DB reforms and where Rwanda rose to the rank of fourth in the world. A 2019 enterprise survey indicated that only 12 percent of firms reported having bank financing for investment (14 percent for working capital), and access to finance was the most cited leading constraint.
  • In China, which moved from 89th in DB2010 to 31st in DB2020, analytic efforts were unable to capture the economic impact of remarkable progress in DB-related reforms. Donors, foreign investors and experts interviewed indicated that the DB agenda did not capture some key business environment challenges, ranging from issues in the judiciary system to intellectual property rights and competition to difficulties with the financial system. World Bank concluded it was time to move the agenda forward to business environment challenges beyond the DB agenda. It also emphasized the need to strengthen access to data, and feedback from the private sector.
  • In the Democratic Republic of Congo, where EoDB rank was 182nd of 183 in 2010 and 183rd of 190 in 2020, lackluster overall performance was attributed to “negative reforms” for which some DB indicators showed a worsening of conditions, as well as to a communication deficit between the government and private sector on implementation of reforms, changes in the DB methodology during the case study period, and the failure of DB ratings to reward “complementary reforms.” Between DB2010 and DB2020, the Democratic Republic of Congo was credited with 27 positive reforms and 8 negative reforms.
  • In Jordan, which moved from 100th to 75th in EoDB, the case study found it hard to identify direct economic benefits of the DB-informed reforms. Cited factors include Jordan’s susceptibility to regional shocks, macroeconomic policies, and the nature of economic growth, which, in turn, has not been in labor-intensive sectors. The study found that although DB-informed projects target areas that need to be reformed, reforms in some cases improved ranking without addressing the private sectors’ needs.
  • Russia moved from 120th in EoDB in 2010 to 28th in EoDB in 2020, yet the case study found that movement of DB indicators was not informative on intermediate outcomes and impacts of reforms. Even though Russia moved from 182nd of 183 countries in DB2010 to 26th of 190 countries in DB2020 for DWCP, some indicators of constraints increased in business surveys. Bank staff pointed to a host of priorities outside even the expanded regulatory agenda growing out of DB, including SMEs’ ability to engage with global value chains, competition (antimonopoly) regulation, investment promotion, and more.

Similarly, IEG’s deep dives found mixed effectiveness in specific indicator areas (table 3.1). Except for the paying taxes subindicator on tax rates, the reforms encouraged by DB-related activities were beneficial. At the same time, benefits could be limited when the reforms focused only on what was captured by the indicator and the base case scenario rather than addressing the broader regulatory area.

Table 3.1. Key Findings on Effectiveness from Doing Business Indicator Deep Dives

DB Indicator

Summary Findings

Starting a business

  • Country experiences in reforms are decidedly mixed given the many constraints to entry and disincentives to formalization.
  • There is general empirical support in the literature for reducing the cost and complexity of business entry, but many countries experience only short-term or limited benefits from reforms.
  • For informal firms, there is evidence that the cost of registration alone may not tip the decision to formalize, unless it is accompanied by additional incentives such as simplified or reduced taxation or enhanced access to finance and land (World Bank 2013b).

Getting credit

  • Useful in focusing reforms on movable collateral and credit information.
  • Deeper reforms take time. Where client countries focused on carrying out quick reforms to improve DB rankings, it overshadowed the long duration of time needed to set up adequate credit infrastructure. In Morocco, collateral law and registry reforms took nine years. It takes up to two years to set up a credit registry, and it is recommended to collect at least two years’ worth of data before launch.
  • The outcome of reforms depends heavily on whether there are other factors limiting credit, such as heavy presence of the state, weak competition, other distortions in credit markets, or demand-side factors constraining the flow of credit-worthy projects.

Trading across borders

  • The trade facilitation features captured by this indicator are linked to trade flows.
  • By focusing only on a single export and single import at the largest port in the largest business city in most countries, the trading across borders indicator is less effective at capturing implementation progress in trade facilitation reform.
  • A lack of granularity means many reforms may not be picked up. For example, over the evaluation period, thorough trade reform in the Lao People’s Democratic Republic yielded only two recorded trading across borders reforms in DB.

Dealing with construction permits

  • In several countries, DWCP has proven to be a useful starting point for conversations about construction regulation.
  • Because important aspects (including corruption and informality) are missed, reforms are often not experienced as improvements by firms.
  • There is a risk that countries focus only on improving their DWCP ranking rather than overcoming persistent industry obstacles.
  • In China and the Russian Federation, it led to successful reform efforts. Those experiences indicate that considerable supplemental analysis, expertise, and local understanding at the municipal level must be mobilized to successfully introduce reforms.

Source: Independent Evaluation Group; World Bank 2013b.

Note: DB = Doing Business; DWCP = dealing with construction permits.

IEG’s exploratory econometric analysis suggests that it is difficult to find significant, systematic relationships between changes in DB indicators and measurable outcomes such as gross domestic product growth, employment, foreign direct investment, trade, or labor productivity. The econometric exercise tested the attribution of economic outcomes to movements of DB indicators. Many apparent correlations vanish when control variables are introduced. There is a high sensitivity to which variables were included (with an implicit sensitivity to omitted variables) and to small changes in model specification. For example, although one model specification showed a significant relationship between the protection of minority investors indicator and foreign direct investment, another showed no significance. In some cases, DB indicators bore a counterintuitive but significant negative relationship with some outcome variables (for example, a negative relationship between resolving insolvency and employment, and between registering property and employment). Simple before-and-after analysis does not control for a host of explanatory factors, yet it can also be hard to specify a control group required to apply a more rigorous difference-in-difference technique. For these reasons, IEG is not offering its own econometric treatment of these relationships, deferring instead to the literature.

World Bank Group Country Strategies

An examination of a sample of IEG’s CPEs and reviews of Completion and Learning Report Reviews shows both the power and limits of the use of DB in country strategies.3 IEG’s review of a sample of IEG-evaluated country strategies indicates that, of 38 that proposed a DB-related work program, 74 percent of them achieved or mostly achieved the corresponding DB objectives. The highest success rate was achieved in low-income countries (86 percent), followed by lower-middle-income countries (79 percent), and then upper-middle-income countries (64 percent). By region, although Latin America and the Caribbean, Sub-Saharan Africa, and Europe and Central Asia saw more than 75 percent of their relevant interventions achieve their objectives, this rate fell to 67 percent in South Asia, 50 percent in Middle East and North Africa, and 33 percent in East Asia and Pacific.

However, achievement of project or country program objectives often does not translate into improvements in DB indicators. Although DB indicators are the most popular source used in country strategy documents to show progress on the business environment, they do not always prove responsive to the reforms being supported. Despite the 74 percent success rate of DB-informed country strategies in achieving their business environment reform objectives, only 45 percent also showed improvements in DB indicators.

A wide range of country experiences link DB-related reforms with improved business environments and sought-after economic benefits, but they also reveal mixed evidence about the effectiveness of DB-led reforms and the extent to which actual economic progress is achieved:

  • DB’s direct influence was evident in the Philippines. With IFC support, the country enacted a law known as the Ease of Doing Business and Efficient Government Service Delivery Act of 2018, which aimed to reduce processing time, cut bureaucratic red tape, and eliminate corrupt practices. Romania’s Completion and Learning Report Review cites DB-inspired reforms as evidence of an improved business environment.
  • Some other countries achieved desired DB-related reforms but without evident economic benefits. The Mexico CPE notes that DB-informed national and subnational initiatives seem linked to “a trend improvement” in the EoDB as well as to other reforms, including in competition policy (IEG 2018, xii). (Nonetheless, the country suffered from persistent low total factor productivity.) In Mauritius, DB reports and investment climate assessments were found to be “instrumental in highlighting areas of weakness and strength in the regulatory environment” and defining reform priorities. Mauritius improved its DB ranking yet “[d]espite this progress, there remain areas of weakness in the regulatory environment” such that reforms “have not resulted in a surge in business registrations” (IEG 2016, 32–33).
  • Despite a focus on improving DB standing, some countries slipped back in DB ranking. In Bhutan, a 2017 performance and learning review found that “While Bhutan’s 2017 Doing Business ranking is the highest among South Asian countries, its drop in ranking from 71 to 73 (out of 190 countries) suggests that continued effort is needed to improve the business climate” (World Bank 2017c, 9).
  • In Zambia, the country achieved measured DB reforms with World Bank and IFC support, yet there was “a marginal decline in Zambia’s distance to frontier score for overall ease of doing business” (World Bank 2019e, 6). For Benin, the DB target of reducing days to enforce a contract was detached from the country program in that “no program interventions could reasonably be identified with … number of days to enforce a contract—and hence any attribution is an issue” (World Bank 2018a, 6).
  • In other countries, the focus on DB had limited impact because it failed to address binding constraints. As described in chapter 2, this was the case in Rwanda, where the CPE found that, in spite of its top “DB reformer” status, “sustaining growth and poverty reduction—from already impressive achievements—will require significant structural change in the economy” (World Bank 2019c, 3). In Albania, which reached the top half of the global ranking in Doing Business by DB2018, the CPE says it did not address important private sector constraints, including weak institutions and the absence of a reliable and affordable power supply and adequate roads. In small states, IEG’s clustered program evaluation found that the Bank Group “needs a sharper focus on the most binding business constraints, using sector-specific lenses. Engagement based on the DB framework and similar cross-cutting approaches led to useful reforms … but engagement did not always focus on the most binding constraints.” (World Bank 2016a, 47). In the Kyrgyz Republic, a key lesson was that: “overarching PSD [private sector development] reforms have greater impact than changes in specific doing business indicators” and selective interventions (World Bank 2018c, 2).
  • In some countries, although the EoDB ranking showed progress, there was no clear evidence that the investment climate improved. The evaluation of Tunisia’s country program through 2013 references business surveys and analytic work that “pointed to the heavy handed and pervasive influence of the state, and to the lack of serious reforms in the onshore sector” (World Bank 2014b, 48). A key problem was “privilege and unequal application of the rules of the game” that constrained competition. Weak governance was manifested in “discretion in the application of laws and regulations, inefficient procurement processes, rigged privatization, declassification of public land and assets and improper use of public banks.” These factors were “binding constraints on domestic private investments.”

The project portfolio informed by DB is generally successful in achieving stated project objectives. Of 137 IEG-validated evaluated projects involving 291 interventions (components), 87 were World Bank lending projects and 50 were IFC AS.4 (World Bank advisory services and analytics projects have no validated evaluation framework.) At the intervention level, the World Bank had a success rate of 85 percent (134 out of 157). At the project level, this success rate was 70 percent. Within World Bank lending instruments, interventions in development policy loans and investment project financing had a similar success rate (85 and 86 percent, respectively), while specific investment loans were more successful, with 91 percent achieving their outcomes. Success rates were virtually identical in low-, lower-middle-, and upper-middle-income countries. For IFC AS, the success rate for DB-informed interventions was 78 percent, yet at the project level, it was 54 percent. Thus, for both, the overall success rate of DB-informed components was higher than that of the projects that they were a part of. This suggests DB-informed component objectives may point to one appeal of DB-informed components—their high likelihood to succeed.

In the evaluated portfolio, success rates were lower in 3 of 12 specific indicator areas requiring deep institutional reforms—enforcing contracts, registering property, and resolving insolvency (figure 3.3). Projects on contracting with government and employing workers were extremely rare and even projects dealing with getting electricity and protecting minority investors were not common.

Regarding how DB indicators were used, projects using the indicators as justification were the most successful, followed by those using DB as a project indicator. Projects using DB as an objective or to generate indicators had somewhat lower rates of success (table 3.2). In terms of component objectives, setting up a reform agency was markedly less successful than others. Although components with other objectives were between 71 and 100 percent effective, reform agency components were only 57 percent effective for Bank lending, and 50 percent effective for IFC AS.

Of the 291 evaluated interventions, 262 had data on immediate outcomes of the work, ranging from more transparent tariffs to improved systems to reduced costs of compliance. In general, the data showed better success in achieving project component objectives than in improving immediate outcomes. The measurements least likely to show the desired change were improved administrative procedures (59 percent) and reduced days to complete procedures (63 percent).

Fewer than half of the evaluated interventions provided data on intermediate outcomes (figure 3.4). The most common of these to be reported were an increase in credit, cost savings for businesses, an increase in investment, and an increase in business entry. Of these, investment seemed to respond best to reforms (88 percent successful), followed by improved trade volume (67 percent) and business entry (64 percent). Other categories of intermediate outcomes, including higher tax compliance or revenue (44 percent), cost savings for businesses (25 percent), and an increase in formality (17 percent), indicated success less often.

Figure 3.3. Project Success by Doing Business Indicator Area, Fiscal Years 2010–20

Image

Figure 3.3. Project Success by Doing Business Indicator Area, Fiscal Years 2010–20

Source: Independent Evaluation Group, portfolio review analysis.

Note: Figure is based on 291 interventions (which excludes 12 interventions with no data regarding their intervention outcomes). Interventions may be counted more than once since they can support multiple business areas. The share of successful interventions is defined as the proportion of interventions that achieved or mostly achieved their intervention outcomes.a. Denotes n < 5 interventions.

Table 3.2. Success Rate of Doing Business–Related Interventions by Type of Use of the DB Report, Fiscal Years 2010–20

Use of DB Report or Indicators

Indicators (no.)

Success (%)

As justification for project

183

82

As project indicator

142

80

As project objective

50

72

Source: Independent Evaluation Group, portfolio review analysis.

Note: Table is based on 291 interventions (which excludes 12 interventions with no data regarding their intervention outcomes). Interventions may be counted more than once since they can use DB reports in multiple ways. DB = Doing Business.

Figure 3.4. Success Rate of Doing Business–Related Intermediate Outcomes, Fiscal Years 2010–20

Image

Figure 3.4. Success Rate of Doing Business–Related Intermediate Outcomes, Fiscal Years 2010–20

Source: Independent Evaluation Group, portfolio review analysis.

Note: Figure shows the percentage of interventions above the line, World Bank Group validated by the Independent Evaluation Group. Figure is based on 150 intervention with outcomes (which excludes 153 interventions with no data regarding their immediate outcomes). Interventions have multiple intermediate outcomes. The share of successful interventions is the proportion of interventions that achieved or mostly achieved their intermediate outcomes.a. Denotes n < 5 interventions.

Learning from Factors of Project Success

IEG’s review of evaluated projects found 696 references to factors to which project success or failure were attributed, most within the Bank Group’s control. Eighty-four percent of factors associated with project success and 82 percent of factors associated with project failure related to either quality at entry, project supervision, or monitoring and evaluation (figure 3.5):

  • Regarding quality at entry, the two most important factors were the role of accompanying or prior analytic work and proper identification of risks at appraisal. A negative factor was design complexity that exceeded implementation capacity.5
  • During supervision, key factors included client engagement and follow-up, effective coordination with internal and external stakeholders, and flexibility of implementation.6
  • The quality of monitoring and evaluation could contribute to or inhibit success.7
  • Two external factors—client commitment and public sector institutional capacity—figured most importantly, although agency coordination and political economy factors also mattered in select cases.8
  • Advisory services and analytics self-evaluations indicate success is more subject to external factors, led by political economy and agency coordination, and client commitment and capacity. Realism of the timetable, design simplicity, and elements of supervision ranging from client engagement to team composition were important internal factors.9

IEG’s econometric analysis finds several factors significantly predictive of project success (appendix H). Contextually, the multivariate logistic regression found that, although country income level and region were not predictors of success, the degree of political stability was. Key internal factors were good up-front analytic work, strong coordination, and appropriate team composition. IEG did not gain insight into DB country reform development outcomes applying similar econometric analysis.

IEG’s deep dives into Project Performance Assessment Reports and IFC AS projects highlighted several lessons of success and failure, reinforcing the analysis above but adding nuance (table 3.3; appendix I). They add five key success factors: focus on binding constraints, a strong interagency coordination unit, timely availability of expertise, longer-term and more comprehensive engagements, and public-private dialogue. The analysis also shows how capacity could be built, through learning by doing, sustainable funding mechanisms, timely and appropriate expertise, and careful selection of contractors. Value is found in the use of complementary indicators and analytics, as well as knowledge sharing and peer-to-peer learning.

Figure 3.5. Factors Influencing Outcomes in IEG-Evaluated Projects, Fiscal Years 2010–20

Image

Figure 3.5. Factors Influencing Outcomes in IEG-Evaluated Projects, Fiscal Years 2010–20

Source: Independent Evaluation Group, analysis of evaluated projects and portfolio review analysis.

Note: IEG = Independent Evaluation Group; M&E = monitoring and evaluation.

Table 3.3. Lessons from World Bank Lending and International Finance Corporation Advisory Services Projects

Success

Failure

1. Strong ownership/commitment from the government, coordinating ministry key (PPARs) needed to overcome inertia, vested interests.

1. DB provides limited evidence on relevance, priority of reforms

2. Strength of interagency coordination unit is key.

2. Mismatch between project complexity and client capacity hinders success.

3. Capacity is built by learning by doing, sustainable funding mechanisms; and with timely and appropriate expertise and careful selection of contractors.

3. Failure is more likely where there is a lack of focus on binding constraints.

4. Having the right technical expertise at the right time and place matters.

4. Governments need a proper framework for inter-governmental cooperation across agencies and central/regional government.

5. World Bank and IFC can complement each other through collaboration. World Bank Group organizational changes at times helped or hindered success.

5. One-stop shops and single windows need authority over the functions they combine, requiring process simplification and “back-office re-engineering,” not just a simplified interface.

6. Deeper reforms require comprehensive and long-term engagement. Repeat interventions can be strategic or merely opportunistic.

6. Discontinuity of counterparts, regime change, and shifts in influence of champions can disrupt progress.

7. Many IFC AS projects use other primary indicators due to limited DB relevance, timeliness.

7. Global indicator standardization under DB may be out of alignment with industry standards.

8. Emerging lessons point to value of knowledge sharing, peer-to-peer learning.

8. Lender preference for immovable collateral, distrust, and technical issues may limit uptake of collateral registries.

Source: Independent Evaluation Group.

Note: AS = advisory services; DB = Doing Business; IFC = International Finance Corporation.

  1. As indicated in figure 3.2, there were 24 additional claims of benefits to Doing Business reform that did not specifically identify what those benefits were.
  2. Of the 89 claims, 14 (16 percent) did not mention a source, and 8 mentioned prior World Bank publications (including DB reports), while the remaining 67 claims referenced specific papers. These 67 claims made 117 references to published literature; however, 39 of those references were mentioned in multiple DB reports, leaving only 77 unique sources. Excluding eight general claim references related to overall improvements in areas tracked by the ease of doing business index, the total number of papers referenced is 69. Of these 69 references, only 31 could be found in the bibliographic database maintained by the DB team from 100 leading journals. Of these, 24 could be mapped to a specific business area where a clear link of intervention to outcome might be established. Of these, 10 were identified by the Independent Evaluation Group in its desk review of literature as providing relevant evidence on outcomes for specific indicators, while seven of these 10 were validated by multiple articles (box 3.2). Two of the 12 suffered from mixed evidence where one finding contradicted another in the literature pool. Taking the structured literature review as a basis for rigorous study of outcomes, only 8 (13 percent) of the 69 references to published literature can be found in the structured literature review.
  3. This section draws from the following Country Program Evaluations (CPEs) and Completion and Learning Report Reviews (CLRRs) and a Performance and Learning Review: Albania CPE (World Bank 2021a), Benin CLRR (World Bank 2018a), Bhutan Performance and Learning Review (World Bank 2017c), Cluster CPE on Small States: Organisation of Eastern Caribbean States (World Bank 2016b), Kyrgyz Republic CLRR (World Bank 2018c), Cluster CPE on Small States: Mauritius (World Bank 2016a), Mexico CPE (World Bank 2018d), Philippines CPE (World Bank 2019d), Romania CLRR (World Bank 2018e), Rwanda CPE (World Bank 2019c), Tunisia CPE (World Bank 2014b), Zambia CLRR (World Bank 2019e)
  4. Information was drawn from Implementation Completion and Results Report Reviews, Expanded Project Supervision Reports, and evaluative notes. Effectiveness, learning, and environmental and social aspects sections apply only to projects evaluated by the Independent Evaluation Group. Twelve additional interventions were identified within these projects for which no relevant data were available on effectiveness.
  5. Positive examples: Senegal Economic Governance Project (P113801; analytic work); Nigeria’s Lagos State Development Policy Operation II Project (P123352; identification and mitigation of risks). Negative examples include Côte d’Ivoire’s Second Poverty Reduction Support Credit Project (P143781) and Pakistan’s ADR Phase 2 (inadequate prior analysiswhere the project suffered from insufficient up-front analysis of stakeholders as part of its due diligence, and Sierra Leone Financial Sector Support TA Project (P121514) which suffered from design complexity.
  6. Positive examples: IFC Liberia Investment Climate Advisory Services Phase 3 Project (577647; engagement with counterparts); IFC’s Costa Rica Secured Transactions and Collateral Registries Project (coordination with a broad range of stakeholders); Tajikistan Private Sector Competitiveness Project (P130091; adaptation to challenges).
  7. Positive example: Mauritius Fourth Trade and Competitiveness Development Policy Loan (P116608; well-designed monitoring and evaluation [M&E] system). Negative examples: Kyrgyz Republic’s Development Policy Operation 1 Project (P126034; weak M&E coordination and integration); Togo’s Private Sector Development Support Project (P122326; weak M&E oversight and coordination).
  8. Positive examples: Ukraine’s 2015 Second Development Policy Loan (P151479; client commitment); Colombia’s 2014 Taxes Program (599785; client commitment, stakeholder buy-in). Negative examples: Senegal Economic Governance Project (P113801; client commitment); Philippines’ 2014 Third Development Policy Loan (P147803; capacity constraints); IFC Investment Climate in the Caribbean Advisory Services Project (567627; limited counterpart skills, capacity).
  9. Unlike World Bank lending and IFC advisory services, the framework for World Bank advisory services and analytics self-evaluation has neither been agreed to nor validated by the Independent Evaluation Group. Furthermore, the potential biases of unvalidated self-evaluation are evident from the overwhelming reported effectiveness rate of reimbursable advisory services (96 percent).