Back to cover

Structured Literature Reviews

Chapter 2 | Review of Evidence of the Effects of Doing Business Reforms: Case Study

This chapter presents an example of a structured literature review from an IEG evaluation of the Doing Business project for the Committee on Development Effectiveness of the World Bank Group’s Board of Executive Directors. The evaluation was conducted through the first half of 2021, and IEG shared its results with the World Bank’s management group for their factual review and comment on September 8 of that year. The review followed the release of the 2020 Doing Business report (World Bank 2020), which included a literature review of evidence on the effects of business regulation reforms. The analysis preceded Bank Group management’s announcement on September 16 that it had decided to discontinue the report (World Bank 2022).1 Despite the cancellation of the Doing Business project, the evaluation highlighted several generalizable lessons about the need to ensure standards of transparency in research. The evaluation’s structured literature review contributed to these lessons, providing a textbook example demonstrating the need to ensure transparency in standards of reporting and use of evidence to continue to build trust in the Bank Group’s research.

Background

Since its inception in 2003, the Bank Group’s annual Doing Business report and indicators had aimed to provide objective annual measures of business regulations and their enforcement across most of the world’s economies. The Doing Business indicators included an overall measure of the burden of business regulations, which was also disaggregated across 10 categories informing the main score: starting a business, dealing with construction permits, getting electricity, registering property, getting credit, protecting minority investors, paying taxes, trading across borders, enforcing contracts, and resolving insolvency. Each category also featured several subindicators disaggregating these themes further (World Bank 2020).2

To enable international comparisons, the indicators were based on the idea of a “standardized firm.” This required a set of assumptions about the characteristics of a firm and the market in which it operated. These assumptions elicited considerable attention and prompted debates regarding indicator methodology,3 comprehensiveness,4 representativeness,5 accuracy,6 and potential biases.7

Nevertheless, despite these issues, the Doing Business indicators were widely credited with influencing business regulations worldwide, dominating market share among business climate indicators, and directly motivating many regulatory changes as countries vied for improved status in the rankings the project produced (Besley 2015; Doshi, Kelley, and Simmons 2019). For example, after his election as India’s prime minister, Narendra Modi’s administration set a target of achieving a rank of at least 50th place (an improvement of more than 100 places) in the country rankings based on Doing Business indicator scores. Similarly, in 2012, Russian Federation president Vladimir Putin set a target of placing in the ranking’s top 20 by 2018 (Besley 2015; Fenton Villar 2021; Kelley and Simmons 2020). More than 70 countries established special policy committees and working groups dedicated to improving their respective Doing Business indicator scores, and the World Bank received hundreds of requests every year from officials seeking advice on reforms that could benefit their countries’ rankings (Kelley and Simmons 2020).

The Doing Business project’s popularity raised the question, was there evidence that the types of regulatory reforms the Doing Business project promoted had an impact on economic and other development outcomes? This issue became particularly salient among experts who were more skeptical about the popular narrative the project portrayed concerning the effects of reforms. For example, one study showed that the correlations between the Doing Business indicators and economic outcomes were unstable (Kraay and Tawara 2013); another showed that they suffered from omitted-variable bias (Pinotti 2008). Furthermore, changes in contextual factors that may have had nothing to do with the regulatory process could affect parts of the indicators, so it was sometimes impossible to attribute changes in the index solely to regulatory adjustments or the outcomes of reforms Doing Business promoted (Branstetter et al. 2014; Chemin 2009).

Review Inclusion Criteria

The structured literature review in the Doing Business evaluation sought to assemble an inventory of evidence on the effects of changes in regulatory arrangements Doing Business promoted. Rather than focus on evidence examining correlations between the indicators and social and economic outcomes, it concentrated on econometric studies exploiting natural experiments or variations in the implementation of specific reforms (such as the variation in the timing or region of the reform) to evaluate the reforms’ effects. In doing so, it adhered to prespecified criteria defining the characteristics of an includable study and the method of data (information) extraction and synthesis, and it also reported a specific search strategy. The PICOS(LY) framework (representing the population, intervention, comparison, outcome, study design, language, and year of the research endeavor) used in the review specified the characteristics of eligible studies, outlining includable populations, interventions, comparison groups, outcomes, study designs, study language, and study year (table 2.1).

Overall, the review included studies written in English on interventions identified in a framework based on the World Bank projects reform database (appendix A).8 The studies could include population groups in any country and be published in any format (journal, book chapter, working paper, institutional report, and so on), and studies were not excluded based on the outcome variable featured in the analysis. One exception concerned studies of the effects of infrastructure interventions (for example, in the “getting electricity” domain), which were limited to studies including business outcomes. This restriction reflected the consideration that such interventions might also have broader social impacts unrelated to changes in the business environment, which were beyond the scope of the review. The review also did not include studies of interventions that did not meet the assumptions underlying the stylized firm informing the indicators. For example, reforms to special investment or tax regimes and export processing zones were excluded. Studies were included if they used one of the following study designs:

Table 2.1. Inclusion and Exclusion Criteria for Empirical Studies: PICOS(LY) Summary Table

Criterion

Description

Population

There was no limitation on population covered; studies could focus on population groups in any country.

Intervention

The review included studies examining regulatory reforms related to the Doing Business project. The interventions and reforms included in the review were based on a review of the Doing Business reform database (see the framework presented in appendix A). The review excluded studies that did not examine the effects of a specific intervention or reform.

Comparison

Control groups could be subject to no intervention, on a waiting list for intervention, or part of an alternative intervention or condition.

Outcome

The review did not exclude studies based on their outcome except for when related infrastructure projects were considered. Studies examining the effects of infrastructure projects (for example, in the “getting electricity” domain) were limited to business outcomes.

Study design

The review included quantitative studies in which allocation to intervention and control groups was random, or selection bias had been addressed by design (see chapter text). It excluded simple before-and-after comparisons and simulation and forecast models.

Language

The review included only studies written in English.

Year

There was no limitation on the year the study was conducted, but the latest search date for studies was February 2021.
  • The structured literature review screened over 9,000 studies, but because the Doing Business topics covered such a broad area, unidentified evidence may still exist. The review did not use an exhaustive search strategy. Future research should focus on continuing to record and expand on the set of studies the review identified. However, even if the coverage of the studies included in the review was not comprehensive, the validity of the three key findings discussed in the chapter remains undiminished.
  • The analysis for the review focused on coding descriptive information about each study. Further analysis would benefit from examining the magnitude of reform effects and the determinants of effect size heterogeneity. Such analysis might consider using formal statistical methods, meta-analysis, or narrative approaches to synthesis. However, it should avoid making conclusions based purely on vote counting. Because it generally involves simply adding up (counting) the number of studies with positive, negative, and null findings, vote counting does not always account for the magnitude of studies’ estimated effect sizes or their precision (Waddington et al. 2012).
  • The analysis did not assess the overall strength of the evidence base or the risk of bias associated with study findings (see Moyer and Finney 2005; Sterne et al. 2016; and Waddington et al. 2012). Ideally, two independent assessors and an expert adjudicator would complete critical appraisals.

Source: Independent Evaluation Group.

  1. The announcement followed an external audit by the law firm WilmerHale, which highlighted data irregularities and ethical concerns about the conduct of former World Bank officials involved in the project.
  2. Indicators for two additional topics concerning employing workers and contracting with the government also featured in the 2020 Doing Business report (World Bank 2020), but they were not included in the indicators main score.
  3. For example, Cappiello (2014) discussed methodological issues with the way the indicators counted and aggregated the time associated with complying with regulations.
  4. The Independent Evaluation Group’s 2015 investment climate evaluation considered the indicators incomplete because they “do not cover all areas of regulation as identified in the best practice list” (World Bank 2015, xxxix). Specific areas identified as missing included accounting and auditing, contract laws, competition policy, consumer protection, environmental laws, intellectual property (including privacy, copyright, patent, trademark, and unfair business practices), investment policy and promotion, employment law, labor safety, and health and alternative dispute resolution.
  5. Besley (2015) observed that the standardized firm might be more relevant to the economies of some countries than others. He provided as an example a firm in the agricultural sector, which the stylized assumptions in the indicators methodology would represent very inadequately.
  6. Holden and Pekmezovic (2020) provided numerous examples from the Pacific islands demonstrating that changes in the indicators had little relationship with changes made to policies and regulations. Hallward-Driemeier and Pritchett (2015) also highlighted that the indicators did not summarize firms’ reported experience well.
  7. For example, some arguments highlighted the indicators’ potential bias toward deregulation (Besley 2015).
  8. The Doing Business reform framework presented in appendix A does not capture all possible types of interventions that may have been under implementation in each regulatory category. Interventions outside of the immediate focus of the Doing Business project were omitted from the structured literature review.
  9. World Bank (2022) offers additional information on the distribution of studies across Doing Business topics.
Previous chapter Next chapter