The World Bank Group in Tanzania
Appendix A. Methodology
Table A.1. Evaluation Design Matrix
|
Evaluation Questions |
Methods and Approaches |
|||
|---|---|---|---|---|
|
Trend analysis |
Theory of change |
Relevance |
Effectiveness |
|
|
Evaluation question 1. How relevant to the development needs of Tanzania was the World Bank Group–supported strategy, and did it evolve appropriately over time to reflect changing priorities and changes in country context? |
Review of Bank Group and external diagnostics to understand the factors driving the trend in outcome data; analysis of secondary data to capture the evolution of key outcomes related to Bank Group support |
Content analysis of project documents to identify program description and program implementation to reconstruct the selection, design, and sequence of Bank Group support and to identify assumptions and intended contributions of Bank Group activities |
Content analysis of (self-)evaluation documents (and validation and program evaluation) to extract evidence on the relevance of the design and objectives of operations included in the portfolio; literature review of Bank Group and external analytics to establish coherence in the World Bank support |
Content analysis of (self-) evaluation documents, validation documents, and program evaluation to identify contribution narratives and capture evidence of Bank Group contribution to outcomes; identification of key performance indicators; recognition of factors of success or failure |
|
Evaluation question 2. To what extent has the Bank Group contributed to improving the business environment, especially through access to finance and energy, and by enhancing human capital through education and skills, especially in vulnerable groups, including women and girls? |
||||
|
Evaluation question 3. To what extent has the Bank Group helped Tanzania achieve pro-poor development through efficient and resilient spatial transformation and land use? |
Interviews with experts to identify, corroborate, or refine the team’s understanding of trends and explanatory factors and to gather information on additional stakeholders, data, and unintended consequences |
Interviews with Bank Group staff to gather additional information not available in documents to complete the theory of change; political economy analysis to identify constraints to key reforms and other risks and how the Bank Group mitigated them |
Interviews with clients and partners to gather opinions about the relevance and adaptation of Bank Group operations over the evaluation period |
Interviews with clients and partners to gauge opinions and collect additional evidence on the Bank Group contribution |
Source: Independent Evaluation Group.
The Country Program Evaluation used a mixed methods approach to triangulate evidence and answer the evaluation questions. Based on three design principles—theory, strength of evidence, and causal analysis—the evaluation followed four analytic steps in the evaluation of the World Bank Group support to Tanzania.
First, the team reviewed, categorized, and synthesized information from key diagnostics reports produced by the Bank Group, government, academia, and development partners to identify the main development challenges in each area of focus of the evaluation.
Second, the team reviewed, categorized, and synthetized Bank Group project documents to identify which developmental challenges and which specific constraints the Bank Group supported during the evaluation period, and what evidence of achievement of pursued outcomes could be documented.
Third, the team reconstructed the main pathways to country outcomes pursued by the Bank Group using the information collected in the previous steps and the information gathered through interviews.
Finally, by complementing the evidence gathered in the previous steps with statistical analysis, review of project ratings and performance indicators, and assessment of trends in high-level outcomes, the team assessed the relevance of the Bank Group support and determined the Bank Group’s contribution to the achievement of development outcomes along the causal pathways identified by the results chain.
In performing the aforementioned analytic steps, the evaluation team used the following methods:
- Content analysis of project documents and analytic literature. The team extracted relevant information from Bank Group projects, reports, and country engagement documents, as well as relevant documents from the government, academia, and development partners. The team codified and analyzed the gathered evidence to reconstruct a results chain and test its validity. This approach was used to answer evaluation questions 1, 2, and 3.
- Qualitative analysis of interviews. The team interviewed Bank Group management and staff, selected government officials, and private sector representatives. Additional interviews were conducted with civil society organizations and other development partners. Interviews were virtual or in person and were conducted in accordance with techniques that ensure objectivity and impartiality. Behavioral norms were followed to ensure the highest possible level of participation. This approach was used to answer evaluation questions 1, 2, and 3.
- Statistical analysis. The team researched, gathered, and analyzed official statistics from different governmental and nongovernmental agencies (for example, the World Bank, the International Monetary Fund, Tanzania’s National Bureau of Statistics, and the Tanzania Commission for Universities) and enterprise surveys data to detect trends in sectoral outcomes related to the Bank Group support. This approach was used to answer evaluation questions 1, 2, and 3.
- Geospatial analysis. The team used the geospatial analysis to assess whether climate resilience considerations were taken into consideration in the design of the Bus Rapid Transit phase 1. See the detailed description in appendix B. This approach was used to answer evaluation question 3.
- Results chain. The evaluation articulated a results chain to identify the different pathways of the Bank Group’s interventions to outcomes. The content analysis of project documents to identify program descriptions and program implementation was used to reconstruct the selection, design, and sequence of Bank Group support, identify the relevance of the Bank Group’s engagement and the contributions of Bank Group activities, and trace the causal logic of how and why the implemented program reached its intended outcomes. This approach was used to answer evaluation questions 2 and 3.
- Dynamic benchmarking tool. This is a tool developed by the Macroeconomics, Trade, and Investment Global Practice for benchmarking across countries, time, and variables. It allows the identification of structural and aspirational benchmarks for countries to compare their performance across various dimensions. More specifically, this tool helps users to (i) identify comparators, (ii) benchmark countries according to specific parameters, and (iii) plot all the desired variables on a single figure. The tool can benchmark a country with respect to structural peer countries (for example, countries with similar structural characteristics as the country of interest during a certain period); aspirational peers (for example, countries with similar structural characteristics as the country of interest during a certain period but that grew—in per capita terms—significantly faster over time); or a predefined set of comparator countries. It can even be used to conduct an event study analysis. In this analysis, the structural variables used to identify benchmarks were real GDP per capita growth (annual, percent); total investment (percent GDP); inflation (GDP deflator, percent); regulatory quality (estimate); terms-of-trade; and foreign direct investment inflows (percent GDP). This approach was used to identify comparator countries in chapter 1.