Highlights

Doing Business (DB) indicators are most relevant to motivating countries to reform their legal and regulatory environment for business and pointing to areas for reform within their coverage area. They are best used in conjunction with complementary analysis and indicators that ensure limited development resources are focused on binding constraints. DB indicators are less relevant as project-level objectives or success metrics.

Although DB has both influence and value, it is vital that the resource be as accurate and informative as possible and that it learn from evidence. Indicators suffer from inadequate feedback loops from research and field experience to their design and application.

DB reports have made many claims for the benefits of measured country reforms that go beyond rigorous or replicated evidence. By favoring supportive evidence and by not establishing strong criteria for filtering evidence, the reports open the door for critics of their objectivity and accuracy, posing a reputational risk to the World Bank Group and potentially misleading clients and stakeholders.

The evidence presented in this evaluation shows that DB indicators are powerful motivators of countries to reform their legal and regulatory environment for business. The high-profile DB report and indicators and the competition bred by country ranking are nearly uniformly recognized as features that motivate countries to engage in legal and regulatory reforms to enhance their business environments. Pragmatically, most experts judged this benefit to outweigh some of the methodological imperfections in the indicators and their indexation and ranking. In an institution that emphasizes the contribution of the private sector to development, being able to engage countries in enhancing conditions for doing business matters. The indicators have further shown value in pointing to problematic areas for reform within the areas they cover.

As a guide to binding constraints, DB is limited by its own coverage and methods. The stylization and economy of scope and data collection that make DB possible as an annual report on 190 countries’ regulatory conditions also necessitate careful application. This evaluation shows that in many country contexts, the following is true: (i) other factors—including policy, structural, and institutional ones—are essential for progress on the business environment; (ii) there is a broader or deeper reform agenda than that captured by the DB indicators; and (iii) the DB methodology and base case scenario may not necessarily reflect local conditions.

Given limited country reform bandwidth and resources, it is vital to contextualize the strongly motivating messages of DB rankings with complementary sources of information to guide country reform priorities. It is in the interest of the client countries and the Bank Group to focus on the reforms yielding the greatest benefit. The integration of DB with other sources of data and analytic guidance seems to work well. The more sophisticated country users of DB (for example, Colombia) and the most sophisticated Bank Group analyses (for example, recent CPSDs) do this as a matter of course. More generally, it is vital for all users of DB to bring other sources of evidence and analytic tools to bear in determining reform priorities. This would also resonate with earlier IEG recommendations (see World Bank 2015b; World Bank 2019b) about considering the policy objectives, such as protecting public health and safety, not just the compliance costs, of regulations. Such a broader set of sources has generally not been reflected in DB reform memos, but it is evident in many country cases over the broader reform dialogue. The Bank Group needs to provide such context in its guidance, not only in textual caveats in DB reports, but at all stages of engagement.

Recommendation 1. In line with much existing practice, the Bank Group should continue to use DB to motivate client engagement and to assist in reform focus within its menu of regulatory areas—but only where the priority and nature of reforms are confirmed by complementary analytics. The intended outcome of this recommendation is that when countries engage with the Bank Group on business environment reform, they are consistently offered guidance based on a balance of appropriate evidence sources and frameworks.

Although the available evidence on the benefits of improving conditions measured by DB is mostly positive for development outcomes, strong evidence is limited by indicator area and type of reform. It is important, therefore, to recognize the limited understanding of the extent to which improvements in DB indicators result in improved development outcomes. Further, the evidence gathered in this evaluation shows that substantive reforms can fail to move the indicators and that, even if they do, such reforms may not yield substantial development benefits. The same features that enable DB to be produced annually and globally limit its value as an indicator of progress and especially as an explicit objective of reform. By covering only one to two cities and an often-hypothetical specific company or transaction, DB is made possible but also constrained. A common complaint from client countries is that DB indicators are not granular enough to track their reform implementation. Given this lack of granularity, IEG found that IFC AS has largely stopped using DB indicators as primary monitoring indicators. In addition, establishing DB indicators as project objectives invites the kind of strategic behavior that narrows focus to just influencing the indicator, rather than addressing the full substantive area of reform. It can increase the kind of pressure on the Bank Group from clients to improve indicators that was identified in the Group Internal Audit Vice Presidency audit (World Bank Group 2020). Yet 42 percent of the identified portfolio used DB indicators as project objectives, project indicators, or both.

Recommendation 2. Consistent with good practice, the Bank Group should avoid using DB indicators as explicit reform objectives or monitoring indicators in projects and country strategies and, where avoidable, should not use DB as primary indicators of reform progress. The intended outcome is to reduce a practice that ties project success to movement in DB indicators, which has proven difficult and potentially misleading, eliminating the usage of DB as a sole indicator or objective and focusing instead on more tailored and granular indicators of success.

DB indicators currently suffer from inadequate feedback loops from research and field experience to their design and application. While it is important to recognize the influence and value of DB, it is also vital that the resource be made as accurate and informative as possible and that it learn from evidence. Complementary work by DEC’s expert committee will provide greater insights into potential improvements in the methodology of individual indicators. IEG found limitations in the extent to which indicators reflect evidence and experience. First, there are indicators for which there is little evidence regarding the economic relevance of what the indicators measure. DWCP and getting electricity are important areas for indicators, but there is no rigorous empirical support for outcomes linked to the aspects that DB indicators currently capture. Little rigorous evidence is available overall on getting electricity, registering property, and enforcing contracts. Three scholarly articles in the DB literature database question the ability of DWCP to capture the real difficulty of building regulations for domestic SMEs. This raises the question, why would the Bank Group continue to produce indicators and promote reforms based on them without firm evidence of their development benefits?

Standard assumptions about how compliance costs and time are influenced by “steps” and “documents” may be challenged by digitalization and e-government. It is vital, therefore, that indicators keep up with changes in technology. For example, use of big data and the growth of digital financial services may erode the relevance of traditional credit information systems.

Although this evaluation recognizes that changes to DB indicators have costs to client countries and to researchers, well-communicated and infrequent changes to indicators can improve their accuracy without imperiling their benefits. This is confirmed by the generally benign view many experts have of several past reforms to indicators. In spite of the acknowledged costs of discontinuities in indicators, underscored by the 2018 external audit (Morck and Shou 2018), there are ways to limit these costs.

Given its influence, it is desirable for the DB approach to capture a fuller range of regulatory, legal, and institutional conditions that influence the life cycle of enterprises. IEG has examined this issue before (World Bank 2015b), as have many others, highlighting areas ranging from consumer and environmental protection to competition and intellectual property regulation. Even in areas nominally covered, aspects of some key regulations—like sectoral business licensing and many areas of contract law—remain out of scope.

Recommendation 3. The Bank Group should update DB indicator areas and definitions at regular and predictable intervals to reflect learning from research and field experience. Doing so will improve links to important development outcomes, strengthen relevance to the experience of domestic SMEs, and adapt to technological changes in the areas covered by the indicators. The intended outcome is for the Bank Group to deliver the best possible information to country clients to inform their business environment reforms as guided by research and field experience. The indicator agenda and characteristics should reflect ongoing learning from evidence and experience, showing what matters and the need to adapt to technological changes in areas covered. Similarly, advice to client countries on appropriate reform models conveyed through the DB report, projects, and knowledge sharing should reflect such learning. Where review of evidence (like that undertaken for this evaluation) reveals knowledge gaps, this has direct implications for a strategic program of research to fill such knowledge gaps. Effective feedback loops require both a steady flow of information from which to learn—from research and monitoring and evaluation—and routines through which DB indicators can adapt to such feedback with the least disruption to users. Disruption caused by such updates can be managed by making such changes infrequently and predictably, by engaging in a transparent and consultative process, and by maintaining former indicator series for a period of years after changes are introduced.

DB reports have made many claims for the benefits of measured reforms that go beyond rigorous or replicated evidence. By favoring supportive evidence and by not establishing strong criteria for filtering evidence, the reports open the door for critics to question the objectivity and accuracy of this important resource. This also poses a reputational risk to the Bank Group and Development Economics Vice Presidency and may mislead clients and stakeholders. This evaluation shows how DB reports, alongside some robust claims, forward some claims on the relationship of changes in areas measured by its indicators to development outcomes that have no clear evidence, some with inadequate evidence, and some where evidence is mixed. The criteria for selecting and featuring such claims is not transparent or consistent, but the general tone is one of advocacy. This does not seem needed or advantageous given the traction DB has found worldwide and the considerable evidence in the literature cited in this evaluation validating important parts of its agenda. IEG’s SLR shows that rigorous methods and evidence are available and implementable. Where there are gaps, it establishes an excellent agenda for future research. The Bank Group need not set its criteria for selecting evidence at the level of the SLR, but it should establish, publicize, and apply its criteria.

Recommendation 4. The Bank Group should strengthen the accuracy and validity of DB claims in DB reports and related communications in line with robust evidence. The intended outcome is to produce DB reports that accurately inform their audience about the relationship of DB indicators to outcomes based on robust evidence. As a leading publication from the research arm of the World Bank, the Bank Group needs to ensure that claims made in DB reports are robustly substantiated by research and refrain from claims that are not. It should adopt clear standards of evidence and commission DEC or outside research if evidence is lacking.

The ultimate outcome sought with this set of recommendations is to build on the many good practices observed in the course of this evaluation. In doing so, the Bank Group could ensure that the DB indicators maintain their substantial power of motivating and engaging client countries in business environment reform, in a manner that guides clients to prioritize the reforms with the greatest development benefits for their socioeconomic situation, based on a balanced and accurate consideration of evidence.