Provide guidance and actively encourage staff to develop and utilize sustainable forest management outcome indicators that can adequately track progress across the three pillars - including indicators that can track and mechanisms that can manage attendant tensions and tradeoffs in the forest landscape.
The monitoring and reporting systems of the World Bank forest sector operations are inadequate to verify whether its operations are supporting forest management in an environmentally and socially sustainable way, in line with the 2002 Strategy and the Bank Group's Operational Policies. Environmental indicators used in forest projects are mainly process or effort measures (such as number of hectares planted, or numbers of hectares under management plans).
Most poverty alleviation indicators were less direct indicators of poverty than is desirable both for accurately assessing project outcomes and for comparison across projects. Poverty reduction indicators like numbers of productive investments made are imperfect measures of whether programs are reaching the most vulnerable members of a community.
Several of IFC's downstream Forest Sector investments cannot be evaluated from a sustainable forest management perspective because information regarding chain-of- custody of the wood products sources is lacking.
Original Response: Agreed: Measuring and reporting on the impacts and structural changes from the WBG's engagement in the forest sector is essential and it needs to be strengthened. Bank Management agrees that this is an important area which requires attention, and is committed to working with specialists and development partners in clarifying how best these types of indicators could be developed and implemented in a cost effective manner. The use of quasiexperimental approaches will also be examined, and incorporated into planned work on the use of impact evaluation in natural resource management projects. In the coming months, we intend to launch an initiative with this objective in mind, which should complement the Bank's support for work on providing guidance for conservation monitoring and its work on Core Sector Indicators. The monitoring of changes at the project level poses enormous methodological challenges. Investments in improved forest management or in biodiversity conservation simply take a long time to show impact. Natural population variability makes short term biodiversity outcomes problematic to measure and the sustainability of forest management can be measured only over several rotations. Empirically robust methods to assess the impacts of forest investments on poverty outcomes are equally challenging because of the attribution problem. In developing more robust approaches to monitoring, these considerations will be central.
WB Action 4A: Prepare and distribute analytical products (AAA) providing guidance on practical methods to measure outcomes and that can be used both during and after project implementation and which will advise project design.
Indicator: number of AAA-products prepared and dissemination/training provided.
Baseline: zero
Target: One AAA-product (ESW or KP) produced annually two BBLs and one training event during a Learning Event (or through Global Distance Learning Network (GDLN) annually. AAA will address such issues as a) developing appropriate and practical proxy indicators to allow monitoring outcomes which have long incubation periods and for which attribution is unclear (e.g. changes in biodiversity and rural livelihoods) or which have a policy dimension to them (e.g. forest governance) b) providing guidance how existing poverty and biodiversity indicators could be best used to help monitor the performance of World Bank forest operations.
Timeline: FY14-FY16 ongoing training and dissemination after that.
WB Action 4B: Recruit technical skills to provide guidance for improving performance monitoring and assessment of impact of forest operations
Indicator: Number of technical specialists recruited.
Baseline: zero
Target: one
Timeline: FY13
Status: Completed
Action 4C: Strengthen collaboration with other development and technical agencies to build a critical mass of global knowledge on performance monitoring of forest conservation and management projects and programs.
Indicator: number of knowledge networks established
Baseline: zero
Target: one
Timeline: FY14
WB Action 4D: Strengthen the use of corporate level performance indicators (Cores sector indicators, GHG accounting) in forest interventions
Indicator: share of AT-coded projects using CSI and GHG accounting.
Baseline: CSI being introduced GHG accounting not used.
Target: 2/3 new AT-coded projects report against at least two CSI all forest projects use ex ante GHG accounting, if appropriate methodology and tools exist.
Timeline: on-going
This recommendation is rated substantial because only two out of the four actions have been implemented.
4A: Partially achieved
4B: Not achieved (the recruited technical skills have left the Bank and there is no replacement)
4C: Achieved
4D: Achieved (this is not linked to the recommendation)
IEG's 2013 Forest Evaluation found that the monitoring and reporting systems of WBG forest sector operations were inadequate to verify whether its operations are supporting forest management in an environmentally and socially sustainable way, in line with the 2002 Strategy and the Bank Group's Operational Policies. Environmental indicators used in forest projects had been mainly output indicators (number of hectares planted or put under management plans). In this year's MAR, WBG management points to the use of Core Sector Indicators in Forest Projects. IEG has repeatedly stressed that CSIs are "output" oriented.". Prxy indicators were developed by the PROFOR AAA, but these, with exception of a single project, are not being incorporated in the results frames of forest sector projects.
For the final MAR update, IEG conducted a review of all Forest Projects approved between since FY14 and FY17. While there have been noticeable improvements at the portfolio level in select projects that seek a more balanced and meaningful indicator mix, for the most part, the M&E systems of forest sector projects are still reporting at the output level.
Environmental Monitoring. Of the total forest portfolio (n=62), 35 projects had a project development objective that focused on environmental management outcomes. Many of these projects sought to achieve sustainable forest, land and water management. For these, none of the results frameworks included technical indicators capable of measuring land use change. Indicators remain limited to the number of hectares of forest restored, reforested, or put under enhanced protection. There are no water or soil quality metrics to support sustainable land management objectives and no forest quality or habitat indicators to support reporting on sustainable forest management.
Objectives that seek behavioral change are not accompanied by indicators to track behavior or capturing information about adoption or non-adoption rates and why. These projects heavily fund training but this is measured by the number of participants, or the number of trainings. Behavioral change features prominently in wildlife project where the sustained adoption of biodiversity friendly practices will be necessary to achieve project objectives over time. In biodiversity projects, the METT - or the GEF Tracking Tool - is being systematically applied. This is a useful tool, recognized by the international community, for systematically and temporally tracking the enforcement capacity of protected areas. However, it requires additional indicators to record the status and health of the flora and fauna internal to the system and pressures that exist without.
Poverty Tracking. Of the total forest portfolio (n=62), 18 projects (29%) had a poverty related aim. Of these, most used the core indicator: "people in targeted forest and adjacent communities with (increased) monetary or non-monetary benefits from forests" however, in several cases, there was no use of the word "increase." None of these indicators had a baseline (baseline =0) or set monetary targets. They all use "number of" people as the unit of analysis. Of these, half disaggregate this data either for gender or for vulnerable persons.
Forest projects (e.g. silviculture) that aim to achieve economic or sector growth have varying indicators, but generally do not have poverty related indicators. Some, like the Belarus Forestry Development Project, has good technical indicators related to thinning, seed growth and utilization, and carbon sequestration metrics, but neglect gender and poverty. Others, like in the Kyrgyz Republic, use weak indicators (# of ha) to measure achievements in the area of sustainable land management, including number of hectares reforested or number of of ha put under management plans.
Gender is fully integrated into forest project results matrices at the level of beneficiary participation and some (11 projects) include indicators to track the number of women that are projected to receive monetary and/or non-monetary benefits. However, data collection will be complicated by the aforementioned measurement challenges (lack of baseline, no monetary targets). In a few cases, women's participation in training and their adoption of new technology is tracked (number of). There are also some examples of indicators that track women's perceptions or satisfaction with project activities (4 projects) and their participation in consultation or decision-making activities (3 projects).
Indigenous People. 12 out of the 49 full size projects, or 24% include an aim to support Indigenous Peoples (DRC, Argentina, Albania, Brazil, Indonesia, Peru, Ghana, Zambia), of which only 5 include disaggregated IP (ethnic minority/indigenous) indicators, all of but which are designed to measure benefits, not just participation (e.g. monetary or non-monetary benefits). In these projectsm gender is all tracked, but only at the level of participation. In IP societies, including in many that are patriarchal, knowing how women are integrated into the benefit streams is important.
Trade-Off analysis
IEG's 2013 Forest Evaluation found that while sector trade-offs - between conservation, poverty alleviation, and growth aims - were identified in the portfolio, they were not being well managed during implementation to realize potential synergies or to reduce risks. A review of the results matrices and indicators of the post evaluation forest project portfolio reveals that, while exemplary project designs exist, this type of risk management is not well reflected in design.
In Africa, there is only one project in the portfolio examined that is engaging in forest concession development (Liberia). While the project engages at the landscape level, each landscape component is measured differently and inadequately. Protected areas are measured using the METT, but newly gazetted areas are not tracked from the point of view of poverty or social dimensions. Concession components include targets that track the number of new concessions put in place. There are no poverty or social indicators, nor growth, linked to the Bank's support for forest concessions. Community forest management components lack measures of environmental sustainability and the level of benefit sharing.
Forest projects that are seeking to create or strengthen protected areas, including in support of wildlife preservation, are not tending to the social and welfare related risks that are inherent to more restrictive land use schemes. These projects are gazetting or increase the level of protection of forest areas - in Sudan, Zimbabwe, Mozambique and Lao - but they do not include indicators to track welfare effects that occur as a result of the World bank supported land use change.
Action 4A: 6 Training Events for World Bank staff based in WDC and in the field on M&E for Forests, Landscapes and Sustainable Land Management Projects focusing on core sectors indicators and proxi indicators were carried out in the spring 2017. Going beyond the indicator on AAA and learning events and addressing the core of the action to provide guidance, a guidance note on core sector indicators was prepared by OPCS. http://go.worldbank.org/7KKD6TJZS0 Action 4B: N/A Action 4C: Since the evaluation was undertaken the World Bank has established partnerships and knowledge networks with many other many other organization on Forest and Landscape. In particular in relation to monitoring the action plan the World Bank is now a partner of Global Forest Watch and of the Global Partnership for Forest and Landscape Restoration (GPFLR). As a result of this partnership, new guidelines have just been developed by WRI on behalf of the partners, and these guidelines have been discussed in a Forest and Landscape Monitoring Workshop hosted by FAO in July 2017. See attachment. Action 4D: See above for training on core sector indicators. In addition projects clinics on results framework for specific new forests landscapes project were carried out. Forestry projects in the bank have progressively incorporated CSI and the new CRI indicators at the design stage (the new CRI include two indicators for the forestry sector: 1. Land area under sustainable landscape management practices and 2. Forest area brought under management plans). The table attached presents examples of Investment Operations in the forestry sector, approved in FY2016 and FY2017, that included Corporate Results Indicators in their designs.
**please see attachment for table of projects**
The progress in this area is substantial. However, IEGâs analysis of the FY14-FY16 portfolio reveals that many projects are not using a relevant or adequate mix of the core indicators to achieve the measurement aims (proxies) recommended by IEG and endorsed by CODE. While there are notable projects in this portfolio that utilize a mix of indicators that combine sustainable land and forest management with a measure of monetary and non-monetary benefits and sometimes with carbon benefits, these account for only 7 out of the 31 projects reviewed. Many projects continue to include indicators that are singular in focus, and therefore not able to track the attendant trade-offs referenced in the IEG review. For example, 6 projects track only land management and biodiversity outcomes, but not poverty and 4 projects track monetary benefits but not environmental sustainability. Some projects tracked environmental management and wildlife enforcement, but not livelihoods (2), and projects that tracked production (thinning, average utilizable volume of commercial timber harvested) did not track poverty (1).
While the language in the new core indicators is improved, the metrics being used are situated at the output level. (e.g. adoption of sustainable land management practices is measured by the # of hectares). A second core indicator that appears in 11 projects as a variation of âpeople in targeted forest and adjacent communities with increased monetary or non-monetary benefits from forestsâ measures the number of people that benefit. An examination of these projects (approved between FY14-FY16) reveals that in all cases the baseline was marked as 0 (which implies that analysis has not been conducted to understand how forest and adjacent communities presently benefit for forests). The value cannot be 0. Relatedly, the indicator does not include a value (by how much?). In the event of a regime change, certain groups that live in and around the area may gain or lose from the land use change. Knowing the percentage of persons that gain versus lose, and the depth of this gain or loss, is important to understand the projectâs aggregate effects, including its effect on poverty.
The review of the FY12-15 forest portfolio revealed significant weaknesses in reporting results and impacts of the forest interventions in particular an uneven use of indicators across the portfolio, and the challenge of capturing impacts that materialize beyond the lifetime of specific operations. CODE made the enhancement of the quality of the Monitoring and Evaluation of paramount importance. To do so, the Forest Action Plan FY16-20 identifies different activities that will complement and reinforce each other to enhance the quality of the Monitoring and Evaluation frameworks of upcoming operations: (i) systematic use of Core Sector Indicators, (ii) inclusion of Predictive Proxy Indicators and (iii) use of impact evaluations. Specifically, the endorsed FAP proposes the following enhancement on monitoring and reporting from FY 16 onwards:
Systematic Use of the Core Indicators across Sectors
- Several core sector indicators and related guidance notes for World Bank operations, including forests investments and other investments, were launched in July 2012 and updated in 2014. The systematic use of core sector indicators in projects and programs on forests will allow the aggregation of results at the portfolio level and a more realistic reporting of achievements and funding volumes. There is a renewed commitment to the rigorous application of these core sector indicators in the results framework of forestry and forest-relevant operations to overcome past inconsistencies. This will enable aggregation of results at the portfolio level in the future.
Key Actions: From FY16 on, operational teams will be required to systematically apply core sector indicators to operations in forestry and relevant sector interventions. Appendix B provides guidance for the use of relevant core sector indicators that should be included in each project/program as they correspond to the relevant investment areas of the Forest Action Plan.
Inclusion of Predictive Proxy Indicators
Most project indicators (even core sector indicators) used in the project results framework are mainly output-driven and tend to fall short in providing sufficient levels of information to predict longer-term forest sector outcomes. In response to the IEG review of the 2002 Forest Strategy, CODE urged the WB Management to develop a set of Predictive Proxy Indicators (PPIs) that could be included in project Results Framework and that would provide a robust enough predictive value of longer-term impacts.
The research on forest sector PPIs conducted by PROFOR identifies a set of PPIs with the potential not just to measure the impacts of forestry programs on poverty reduction and economic growth, but also on other important development outcomes, such as biodiversity conservation, climate change mitigation and adaptation, and good governance (see Box 3.2). Identified PPIs have been included in recently-approved operations on forests .
Key Actions: Teams preparing new forest-related operations will be encouraged to include PPIs in their Results Framework. More work will be done on this theme and particularly on the analysis of the causal chain so that PPIs can be constantly strengthened.
Impact Evaluations
The use of impact evaluations will be encouraged as part of project design. Impact evaluations, when used systematically, as in the health and education sectors, provide a strong evidence base of the results chain and conditions for success. Impact evaluations also provide opportunities for learning during project implementation and at completion and thus contribute to the continuous improvements of WBG interventions.
Key Actions: Task teams working on forestry operations will coordinate more systematically with Bank evaluation teams (for example, the Dev
The ESW on developing proxy indicators for project impact was delivered in FY15 and Forests Sector Proxy Indicators (P145206) are being mainstreamed in operations. More information on the training and dissemination of the ESW would be useful as part of the MAR process. IEG's evaluation found that existing indicators (including use of core) were insufficient for determining the likelihood of forest sector outcomes (SFM, biodiversity, poverty, etc.) so that while the Core Indicator target ( 2/3 of new AT-coded projects report against at least two CSI)) is useful for aggregating corporate aims, these indicators are insufficient to determine the likelihood of achieving forest sector outcomes. Even so, the MAR should report on whether this target was met. IEG recognizes the strong technical assistance provided by the interim staff position that was created to produce the Proxy indicators and regrets that a full time M&E specialist cannot be retained to support training and mainstreaming of this important piece of economic and sector work in the Bank. IEG recognizes the relevant partnerships being maintained with CIFOR, IUCN, and other actors in the sector, however these achievements only indirectly support better indicator development, monitoring, tracking and reporting within the Bank. It would be useful to know who will champion this mainstreaming going forward,
WB Action 4A: Ongoing
- ESW on developing proxy indicators for project impact delivered FY15: Forests Sector Proxy Indicators (P145206) and is being mainstreamed in operations.
- forest-related LSMS work continued by GPVDR. Findings to be presented e.g. in World Forest Congress (Durban, South Africa) in September, 2015.
WB Action 4B: Completed
- Senior Forestry Specialist with strong expertise in M&E and impact monitoring was recruited to [current] GENDR. After completing WB Action 4A, the specialist left the Bank and the findings are being institutionalized by relevant Bank staff and a separate position in no longer warranted.
WB Action 4C: Completed/ongoing
- formal forest knowledge network with IUCN and CIFOR (Knowfor) established
- active ongoing engagement within the network, Collaborative Partnership for Forests and with other partners
WB Action 4D: Ongoing
- forest projects (AT-code) report both their expected GHG impact and have included CSI in their results frameworks
IEG has rated the adoption of this recommendation - related to Forest Sector Indicators - as Medium and considers this recommendation to be Active. The ESW
on developing proxy indicators for project impact is ongoing with expected delivery FY15 and a proposal is being develpoed for a forest-related LSMS module. A
Senior Forestry Specialist with strong expertise in M&E and impact monitoring has been recruited to AES (currently GENDR) and there is an ongoing formal platform that has been established for knowledge exchange between the Bank, IUCN and CIFOR (Knowfor).
WB Action 4A: Ongoing
- ESW on developing proxy indicators for project impact ongoing with expected delivery FY15
- proposal for forest-related LSMS module developed
WB Action 4B: Completed
- Senior Forestry Specialist with strong expertise in M&E and impact monitoring recruited to AES (currently GENDR)
WB Action 4C: Completed/ongoing
- formal forest knowledge network with IUCN and CIFOR (Knowfor) established
- active ongoing engagement within the network, Collaborative Partnership for Forests and with other partners
WB Action 4D: Ongoing
- forest projects (AT-code) report both their expected GHG impact and have included CSI in their results frameworks