The Committee on Development Effectiveness requested that the Independent Evaluation Group (IEG) assess the effectiveness of using Doing Business (DB) and its strategic relevance to countries’ reform priorities and the World Bank Group’s development agenda. It parallels work by the Bank Group’s Group Internal Audit Vice Presidency on process and data integrity (World Bank 2020) and an external expert review commissioned by the Development Economics Vice Presidency focused on methodology.

Since 2003, the Bank Group’s annual DB report and indicators have aimed at providing objective annual measures of business regulations and their enforcement across most of the world’s economies.

Motivation, Scope, and Objectives

DB indicators are considered a guide to the legal and regulatory framework for a country’s small and medium enterprises (SMEs). Their design covers the assumed key elements of a prototypical SME business life cycle, from start-up through operation to exit.

Reporting annually, DB has expanded over time from five initial indicators covering 133 countries in DB2004 to 12 indicators covering 190 countries in DB2020 (figure O.1).

An overall ease of doing business score and ranking aggregates the scores for 10 areas, which it claims assesses “the absolute level of regulatory performance and how it improves over time” (World Bank 2019a, 19) by “benchmarking 190 economies to the regulatory best practice” (iii).

DB’s stated aims are to (i) “motivate reforms through country benchmarking”; (ii) “inform the design of reforms”; (iii) “enrich international initiatives on development effectiveness”; and (iv) “inform theory” (World Bank 2003a, ix).

Figure O.1. Doing Business Indicator Expansion over Time

Image

Share Tweet Facebook WhatsApp Linkedin Email

Figure O.1. Doing Business Indicator Expansion over Time

Source: Independent Evaluation Group.

Note: Indicators in parentheses are not included in the composite ease of doing business index. DB = Doing Business.

Influence and Controversy

DB is highly influential: it is one of the Bank Group’s most widely read publications and most used set of indicators on business regulation.

Country governments use DB in their developmental strategies and programs. The Bank Group uses DB in (i) country strategy and policy dialogue; (ii) operations (both financing and advisory); and (iii) research and global knowledge sharing. Researchers use DB to assess the relationship of legal and regulatory conditions to various reform outcomes, to develop new indicators, and to test and critique the indicators.

DB is also one of the Bank Group’s highly controversial undertakings. It has been the subject of multiple reviews since its inception. Controversies have arisen surrounding its methodology, accuracy, and potential biases and the way DB indicators are used in shaping and assessing country policy reforms.

Methodology

This evaluation assesses the relevance of DB (“doing the right things”) and its effectiveness (“doing things right”) in motivating countries to reform their legal and regulatory environment for business and identifying areas for reform. It uses a combination of standard evaluation methods and newer ones to gather and triangulate evidence at multiple levels: global research, country experience, and project performance. Its scope, methods, and lines of inquiry were established in a prior Approach Paper (World Bank 2021b) and Issues Paper (World Bank 2022).

The Relevance of Doing Business: Is It “Doing the Right Things”?

DB is part of Bank Group efforts to produce reliable, relevant, and comparable data and analysis on private sector development; to promote job creation, economic productivity, and gender equality; and to encourage and guide social and economic reforms promoting an efficient and fair business environment.

Every year, since 2003, the DB team within the Global Indicators Group of the Development Economics Vice Presidency issues its flagship DB report. DB has actively promoted competition among countries, celebrating up through the 2020 edition “top reformers”: countries achieving the largest number of measured reforms.

Country Reforms

DB’s ranking encourages competition among countries and motivates governments to consider reforms. It is often a first point of engagement between country leaders and the Bank Group on addressing legal and regulatory constraints to businesses. Bank Group client countries have made substantial efforts to design and implement reforms measured by DB, often creating or empowering coordinating agencies to lead the reform agenda. Among the most common have been reforms to starting a business; paying taxes; getting credit; registering property; and trading across borders. DB-informed targets and reforms can feed into national development strategies and leadership initiatives.

In many countries, governments create or empower coordinating agencies explicitly focused on the DB reform agenda. In multiple countries, the agencies and capacities created or empowered to pursue DB reforms later turn to a broader or deeper agenda of business environment reform, creating a “spillover effect.”

Although countries are often motivated by DB, other analytic tools, indicators, and expertise are also mobilized to guide and deepen reforms. In many cases, reform momentum and capacity instigated by DB are applied to additional reform subjects. For example, “national”-level reforms, although often initially pursued in leading cities captured by DB, are frequently extended to subnational levels.

However, several country case studies reveal serious limitations to the DB indicators and agenda in capturing business environment reform priorities. In particular, the relevance of the DB agenda is weak in countries where structural or institutional factors act as binding constraints. In other countries, it is uncertain whether the domestic SME focus of the DB agenda is the real focus of the reform efforts.

Over time, the DB agenda can lose its relevance when (i) non-DB constraints become binding after early DB reforms; (ii) pending DB reforms prove less tractable; or (iii) a DB indicator does not adapt to changes in the underlying business process or technology.

Business Areas

The overall ease of doing business score serves as a general index of the regulatory environment. The ordering of reform priorities using DB indicators shows a relatively strong match to that of enterprise surveys within common areas of coverage.

However, DB indicators do not effectively capture the real conditions experienced by businesses within the business area they cover. This is generally attributed to shortcomings in the defined topical coverage of the indicators and in the representativeness of the base case scenario.

Lack of substantive coverage can limit the extent to which the DB indicators can (or should) guide country reforms or reflect reform progress. The characteristics describing the DB “base case scenario” are not always consistent with those experienced by the typical domestic SME.

World Bank Group Activities

DB indicators are used in Bank Group country strategies, assessments, and projects; in Bank Group client countries’ plans and reforms; and in academic research. DB is the most popular source of business environment information but not the only one, even among Bank Group products.

DB is used in country strategies to identify reform needs and to motivate future operations in priority areas. Many World Bank country strategies make substantial references to DB or propose a DB-related work program or both.

DB also informs a substantial share of the Bank Group’s projects that provide financing, advice, or technical assistance to client countries on the business environment. This DB-informed portfolio consisted of 676 projects representing $15.5 billion in commitments during fiscal years 2010–20.1 Informed by DB, the World Bank provided lending support to 97 countries and World Bank or International Finance Corporation advisory services and analytics to 126 countries during the same period.

Within the DB-informed portfolio, the most popular reform interventions addressed trading across borders (28 percent), ease of doing business (27 percent), and starting a business (25 percent). Project objectives focused most on improving a law or regulation (27 percent), reengineering a process (16 percent), building capacity and training (15 percent), and conducting a diagnostic (13 percent).

DB rankings are clearly motivational, facilitating Bank Group engagement with client countries on the business environment. However, DB indicators also have some notable inconsistencies with other Bank Group and global indicators. Many experts find the DB indicators more useful to initiate a country engagement than as explicit objectives and monitoring tools due to limited relevance and granularity.

Research and Evolution

Academics have found great utility in the DB indicators, producing thousands of articles examining the reform areas covered by the indicators and often using them to test hypotheses or construct new indexes. In several areas, there are no comparable annually updated data available for so many countries.

The evaluation team found an extensive body of research literature that uses or focuses on DB indicators. A review of DB’s own database from 100 top academic journals indicates that research has concentrated in a few areas, with disproportionate attention to starting a business, trading across borders, and protecting minority investors. IEG’s structured literature review points to similar clustering and gaps in methodologically rigorous research.

DB indicators have evolved, including the introduction of new indicators and revisions to the methodology of existing ones. Changes of indicators are a necessary way to reflect learning and evidence about their relevance. Yet even with such positive changes, additional modifications should be incorporated into future revisions of the indicators to enhance their relevance to country reforms. Each change bears a cost, so it matters how they are introduced. However, if such changes are infrequent and well communicated, they can serve to enhance the value of the DB indicator set.

The Effectiveness of Doing Business: Is It “Doing Things Right”?

Activities informed by DB vary markedly in terms of the degree to which they yield intended objectives and results—their immediate and intermediate outcomes. The evaluation examined evidence of effectiveness from several sources: research (academic and professional literature), DB report claims, country reform experience, and Bank Group activity (country strategies and projects informed by DB).

Research Evidence

IEG’s desk review of literature collected from leading journals shows limited replicated evidence of the relationship of areas covered by DB to outcomes; more evidence confirmed by a single finding; some areas of contradictory evidence in the literature; and some gaps. IEG’s structured literature review applied rigorous criteria limiting methodology and scope to confirm several relationships between laws or regulations tracked by DB and significant development outcomes, while raising some cautions and pointing to gaps. For example, the structured literature review found no rigorous empirical support for outcomes linked to the DB measures of dealing with construction permits, or getting electricity. It also raises some cautions about unintended consequences of reforms measured by DB.

Doing Business Report Claims

Doing Business reports make numerous claims linking reforms tracked by its indicators to outcomes, led by job creation and economic growth. The claims frequently associate the DB reform agenda with the Bank Group’s institutional twin goals of ending extreme poverty and promoting shared prosperity.

Although some of the claims contained in DB reports meet a high standard of evidence, most claims do not meet such standards, which can create a reputational risk to the Bank Group. Only a minority of those claims can be confirmed through DB’s literature database, and only a very small minority of those claims have been either replicated or confirmed using a rigorous methodology.

Country Reforms Experience

Although many countries, donors, and even Bank Group projects use movement in indicators as an outcome measure, such movements are inconsistently linked to both reform implementation and economic outcomes. Some countries show little increase in investment, employment, or productivity after a positive movement in indicators.

Overall, IEG finds a mixed picture of the links between indicators and outcomes. Deep dives carried out for five DB indicators (starting a business, getting credit, trading across borders, dealing with construction permits, paying taxes) found mixed effectiveness in specific indicator areas. IEG’s own econometric analysis suggests that it is difficult to find significant, systematic relationships between changes in DB indicators and measurable outcomes, such as gross domestic product growth, employment, foreign direct investment, trade, or labor productivity.

World Bank Group Activity Experience

Evidence from IEG-evaluated country strategies indicates that, of those that proposed a DB-related work program, 74 percent achieved or mostly achieved the DB corresponding objectives. However, only 45 percent also showed improvements in DB indicators. In some countries, limited impact is tied to failure to address binding constraints. IEG’s 10 case studies show both strong movement of DB indicators and tenuous links to measurable development outcomes.

The DB-informed project portfolio is generally successful in achieving stated project objectives. Success rates are lower in 3 of 12 specific indicator areas requiring deep institutional reforms: enforcing contracts, registering property, and resolving insolvency. Although client commitment and capacity are important external factors of project success, most success factors are internal and largely within the Bank Group’s own control; these include analytic work, client engagement and follow-up, effective coordination, and monitoring and evaluation.

Improving the Use of Doing Business in Country Reforms

DB indicators are most relevant to motivating countries to reform their legal and regulatory environment for business and pointing to areas for reform within each indicator’s coverage area. They are best used in conjunction with complementary analysis and indicators that assure limited development resources are focused on binding constraints. DB indicators are less relevant as project-level objectives or success metrics.

Given limited country reform bandwidth and resources, it is vital to contextualize the strongly motivating messages of DB rankings with complementary sources of information to guide country reform priorities.

Recommendation 1. In line with much existing practice, the Bank Group should continue to use DB to motivate client engagement and to assist in reform focus within its menu of regulatory areas–but only where the priority and nature of reforms are confirmed by complementary analytics.

Although the available evidence on the benefits of improving conditions measured by DB is mostly positive for development outcomes, strong evidence is limited by indicator area and type of reform.

Recommendation 2. Consistent with good practice, the Bank Group should avoid using DB indicators as explicit reform objectives or monitoring indicators in projects and country strategies and, where their use is unavoidable, should not use DB as primary indicators of reform progress.

DB indicators currently suffer from inadequate feedback loops between research and field experience, and between their design and application. Given its influence, it is desirable for the DB approach to capture a fuller range of regulatory, legal, and institutional conditions that influence the life cycle of enterprises.

Recommendation 3. The Bank Group should update DB indicator areas and definitions at regular and predictable intervals to reflect learning from research and field experience to improve links to important development outcomes, strengthen relevance to the experience of domestic SMEs and adapt to technological changes in the areas covered by the indicators.

DB reports have made many claims for the benefits of measured reforms that go beyond rigorous or replicated evidence.

Recommendation 4. The Bank Group should strengthen the accuracy and validity of DB claims in DB reports and related communications in line with robust evidence.

The ultimate outcome sought with this set of recommendations is to build on the many good practices observed in the course of this evaluation. In doing so, the Bank Group could ensure the DB indicators maintain their substantial power of motivating and engaging client countries in business environment reform in a manner that guides clients to prioritize the reforms with the greatest development benefits for their socioeconomic situation, based on a balanced and accurate consideration of evidence.