We used a multitiered theory-based contribution analysis framework to assess the overall relevance of the World Bank Group’s program to Georgia’s development challenges and its contributions to three interlinked outcome areas: (i) development of the private sector as an engine for economic growth and development, (ii) infrastructure as an enabling sector for private sector development, and (iii) human capital as an enabling sector for private sector development and as the sector directly benefiting the Georgian population.

On the level of outcome areas, we traced the status of the analytic underpinnings at the start of the evaluation period and their evolution throughout. This includes reviewing (i) how analytics translated into strategy; (ii) how strategy informed project selection and design; (iii) whether reforms supported were in line with analytics and strategy; (iv) how successfully advisory, investment, and Program-for-Results projects were implemented and whether implementation appropriately adjusted to changing circumstances; (v) whether projects and reforms delivered results as intended; and (vi) whether delivery of results can be linked to higher-order results at the country level.

Evaluation Design

We used a mixed methods approach to triangulate findings. A trend analysis of the development context in Georgia, including relevant indicators, provides a background of the broader development in which the Bank Group program took place, with the focus on areas the Bank Group sought to influence. The theory-based approach is used to trace pathways of influence, focusing on the role of analytics and strategy in identifying areas of work and projects to ensure relevance and coherence. Contribution analysis explores the degree to which the Bank Group’s work helped shape policy dialogue, catalyze support from other stakeholders, and deliver its program in an effective and efficient manner.1

The evaluation design matrix summarizes methods and approaches used to answer the main evaluation questions (table A.1). Methods include document review, stakeholder mapping, and interviews with Bank Group staff, clients, and other partners.

Source: Independent Evaluation Group.

Note: IFC = International Finance Corporation.

Methods Used

We used a wide range of methods to collect and analyze evidence across a variety of data sources to answer the evaluation questions. Methods included the following:

• Review of analytic work focused on core documents produced by the Bank Group and other institutions. Sources include Country Economic Memorandums, Systematic Country Diagnostics, Financial System Stability Assessments, Public Expenditure Reviews, Public Expenditure and Financial Accountability Assessments, Debt Management Performance Assessments, and others. These were validated by reviewing relevant documents produced by other stakeholders, such as International Monetary Fund Country Reports and program documents, analytic reports by other multilateral development banks, and reports by academia and think tanks.
• Strategies reviewed for the evaluation include Bank Group corporate frameworks, such as Country Partnership Strategies and Country Partnership Frameworks, their evaluations and validations, relevant government strategies, and strategies produced by other partners. At the sector level, the evaluation reviewed strategies produced as part of Bank Group analytics and advisory products and government sector strategies supported through development policy operations.
• Portfolio review and analysis reviewed and analyzed aggregate portfolio data, identifying the relevant set of lending and nonlending projects active during the evaluation cycle. This included coding projects and prior actions by thematic areas they contribute to addressing, reviewing trends in portfolio allocation (considering the relative role of other development partners in the area), and analyzing the sequencing of products to assess programmatic effects. In-depth portfolio review consisted of detailed analysis of project preparation, supervision, and evaluation documents, as well as project outputs and monitoring data.
• Document review traced the evolution of analytics, including lessons learned, the effectiveness of preceding Bank Group and development partner support in achieving intended outcomes, and relevant reports by the Independent Evaluation Group.
• Analysis of country data included a wide range of sources, such as the World Bank Open Data portal, public statistics and custom data queries of the National Statistics Office of Georgia, International Monetary Fund data, the Logistics Performance Index, the Atlas of Economic Complexity, and the Organisation for Economic Co-operation and Development Creditor Reporting System and Programme for International Student Assessment.
• We used semistructured interviews to triangulate and validate findings. The focus of interviews with Bank Group staff was on validating the understanding of strategy, project selection and design in response to strategy, and learning and evaluative findings. Interviews with government counterparts and International Finance Corporation clients aimed at identifying the role of the Bank Group in providing relevant analytics to inform decision-making, responsiveness to development challenges and government demand in designing support, effectiveness of support, and the role of the Bank Group as a provider of expertise. Interviews with other counterparts, such as business associations, think tanks, academia, and others, complemented the picture in assessing the relevance and effectiveness of the Bank Group’s support and less tangible constraints.

Limitations

There are several limitations that potentially interfere with the validity of the conclusions of the report. Portfolio review and analysis relies on the information included and presented in the documents reviewed. Information presented may not be capturing all relevant considerations when preparing and supervising documents. Success ratings are based on judgment and targets that may be inconsistent across projects. Documented information is triangulated with information collected from key informant interviews. Availability of staff and counterparts involved throughout the evaluation period was limited because of interlocutors moving positions or retiring. Further, interviews inherently involve potential issues with recall. Given the complexity of the subject of the evaluation and involvement of various local, bilateral, and multilateral agencies, precise claims of attribution of outcomes to interventions are not possible. This applies both to the attribution of positive outcomes to complementary interventions and to the attribution of absence of outcomes that may result from complementary interventions not materializing. The evaluation assesses plausible claims of contribution and analyzes the relevance and operational effectiveness of interventions. Final intended outcomes in areas such as private sector development, connectivity infrastructure, and human capital may not fully emerge within the time frame of the evaluation. Where relevant, the evaluation reviews project outputs and intermediate outcomes.