"Improving the results measurement system for knowledge services: The Bank should ensure and monitor high-quality results frameworks that respond to client concerns, and link more tightly knowledge services tasks with country partnership strategies milestones and outcome indicators. Use and continuously improve implementation results monitoring systems that track progress of knowledge activities towards achieving outcomes in the results framework of the country partnership strategies. Ensure that all knowledge service activities are timely, seek independent validation through systematic pointed client feedback, and use such feedback to assess results and improve results framework of knowledge services. Draw on appropriate evaluation resources and develop tools to evaluate knowledge-based programs."
"Bank knowledge services were not monitored and evaluated consistently in the sample of countries. In only 17 percent of the knowledge services assessed was there at least a partial results framework in the CPS, allowing a tracking of the contribution of the activity to the broader development outcomes sought by the CPS. Similarly, only 23 percent of the knowledge services included, at least partly, result indicators to track the achievement of the activity's outcome. Where monitoring and evaluation were better, knowledge services results were more likely to be achieved, most likely reflecting a link between M&E, knowledge services quality, and impact. Implementation and results monitoring systems are needed to track progress toward mutually agreed outcomes and mitigate the risk of fragmentation and loss of strategic focus intrinsic in reimbursable advisory services."
Original Response: Agreed: "WB: Management has already begun to address the issues raised by IEG. As noted above, knowledge services are now more closely linked to outcomes defined at the level of the CPS/CASs. The Bank has strengthened its results framework for knowledge services and is now putting in place processing steps and guidelines to support monitoring and quality. The self-assessment process has been strengthened to have more management oversight. A client feedback instrument is now in operation. The next step is for the Bank to learn from the exercise and use it to inform the automation of the process. This work is now ongoing. Going forward, the Bank will put in place an approach for ex-post assessment of knowledge work. In that regard, as plans for automation of the CFI proceed, we look forward to working with IEG on developing a framework for ex-post evaluation which covers the full group of client facing knowledge products and internal knowledge products, i.e., ESW/TA, IE, TE and KP. IEG has already initiated this work with a focus on ESW/TA and we expect, as an initial step, to have a framework along with advice on how to strengthen the information content of the Ops Portal to ensure adequate information for decision making, monitoring and ex-post evaluation in the first quarter of FY14."
Action 4A: See Action 1B
Action 4B: Develop a framework to evaluate knowledge activities and consistently apply it across the institutional portfolio. Link results from IEG evaluations, with operational portal to strengthen the information base for monitoring and evaluation of knowledge activities.
Indicator: Action completed.
Baseline: IEG is coming up with a standard methodology to assess knowledge activities and its evaluation database is not yet linked to the operational portal.
Target: The operational portal and GP knowledge platforms are connected to IEG assessments of knowledge.
Timeline: End FY16
Action 4C: Based on lessons learned from the FY13 pilot exercise on the client feedback instrument (CFI), design and operationalize a plan to systematically implement the CFI process for all KAS.
Indicator: CFI process systematically implemented for all KAS through Bank systems.
Baseline: CFI done manually and on pilot basis..
Target: CFI is fully implemented for ESW/TA/TE/IE.
Timeline: FY16
IEG notes the progress in developing a results framework for ASA. However, the new guidance on ASA results management will be issued in FY18, so consistently applying the evaluation framework for knowledge activities across the institutional portfolio is a work in progress.
All actions under recommendation 0226 have been completed. Management proposes the rating Complete, with a status of Active (to be made inactive). On Action 4A, the new Advisory Services and Analytics (ASA) product line was launched in July 2016 to improve governance, give teams more flexibility, and provide more user-friendly systems. It combines various knowledge services for external clients (i.e. ESW, TA, TE, IE, and PA) in a single product line. Consolidating these five discrete ASA product lines into a single product line reflects a fundamental shift from an output-based approach to Bank knowledge services to a more solutions-based or results-focused approach. A new and customizable results framework for ASA was introduced in the new Operations Portal (3.0). Accountability and Decision Making (ADM) roles for Regions and GPs were hardwired in the new Portal to ensure improved quality oversight.
New guidance on ASA results management will be issued in FY18, focusing on the most critical ASA results management issues. It presents a synthesis of lessons and good practices drawn from an array of Bank sources, including IEG resource materials and evaluations, as well as a review of the results management practices of other knowledge-based institutions such as the OECD and other published materials. Three key considerations are to help teams ensure that their ASA results management practices: (i) can provide a sound basis for managing implementation to achieve the ASA's development outcome (ii) will inform assessment of ASA effectiveness at completion, as well as lessons from the ASA experience and (iii) are consistent with the expectations of key stakeholders such as the TF donor community and the Board. The results management guidance provides an overview of results management throughout the ASA lifecycle. It lays out the fundamentals of results management, from setting out a theory of change to monitoring arrangements, and examines alternative approaches to the design of results frameworks and selection of indicators. The guidance offers advice for results monitoring and adaptive management during implementation, and explores the requirements for ASA completion, including how to assess outcomes and draw lessons. The ASA guidance on results management is complemented by a new guidance document on trust funds results management, which will also be issued in FY18. OPCS, IEG, and the Trade and Competitiveness Global Practice are also collaborating to determine evaluation criteria for Advisory Services and Analytics (ASA).
A new operational framework for RAS will be issued in FY18 to facilitate greater flexibility in engaging with different client countries and refine the quality assurance process, among other things. The new operational framework contains measures to refine the quality assurance process, which are being further elaborated in new guidance materials.
For Action 4B, the new Operations Portal includes a new section on Knowledge that pulls together a set of contextual information and knowledge for the benefit of task teams working on project design and implementation. This Knowledge section includes access to instrument specific policies, procedures, and guidelines, to country context such as CPF/SCDs and relevant ESW, to knowledge and learning curated by GPs and institutionally via the Open Learning Campus, and to lessons learned for similar projects extracted from ICRs and IEG assessments.
On Action 4C, the CFI was mainstreamed in the World Bank Satisfaction Survey in FY15. The Survey covers external and internal clients of both lending projects and ASA, and is linked to Bank's internal 2-Minute Feedback Survey. Most of the questions on the Satisfaction Survey come from the CFI. The survey is short, ongoing (triggered automatically at activity completion) and confidential (results are aggregated by Global Practice and Region). Management reconfirms that the new ASA instrument will keep applying the World Bank Group Satisfaction Survey. Further refinements and guidance for results monitoring and management are under development. Management and IEG have recently completed work on a WBG Evaluation Framework sets out the Group's vision for selecting, conducting, and using evaluations to strengthen accountability and learning for evidence-based program improvement and decision-making to enhance development results. This will be translated into the Bank's operational framework over the coming period. Management is also collaborating with IEG to develop a new evaluation approach for ASA, and appropriate follow-up action will be determined once this work is completed.
According to the Bank, the Operations Portal now includes a new section on Knowledge that pulls together a set of contextual information and knowledge for the benefit of task teams. Against the narrowly defined Action Plan, adoption is substantial. However, metrics for monitoring ASA relevance, technical quality, influence, etc are yet to be developed and implemented and ensuring a 'high-quality results frameworks' for knowledge activities are still a work in progress.
Progress in 2016 is summarized below. All actions under recommendation 0226 have been completed. Management proposes the rating Complete, with a status of Active (to be made inactive).
On Action 4A, the new Advisory Services and Analytics (ASA) product line was launched in July 2016 to improve governance, give teams more flexibility, and provide more user-friendly systems. It combines various knowledge services for external clients (i.e. ESW, TA, TE, IE, and PA) in a single product line. New and customizable results frameworks for ASA was introduced in the new Operations Portal (3.0). Accountability and Decision Making (ADM) roles for Regions and GPs were hardwired in the new Portal to ensure improved quality oversight. RAS procedures and systems have also been updated as part of the ASA product line launch. Currently work is underway on a RAS Issues and Options Paper to facilitate RAS growth, and which will further refine the quality assurance process.
For Action 4B, the new Operations Portal includes a new section on Knowledge that pulls together a set of contextual information and knowledge for the benefit of task teams working on project design and implementation. This Knowledge section includes access to instrument specific policies, procedures, and guidelines, to country context such as CPF/SCDs and relevant ESW, to knowledge and learning curated by GPs and institutionally via the Open Learning Campus, and to lessons learned for similar projects extracted from ICRs and IEG assessments.
On Action 4C, the CFI was mainstreamed in the World Bank Satisfaction Survey in FY15. The Survey covers external and internal clients of both lending projects and ASA, and is linked to Bank's internal 2-Minute Feedback Survey. Most of the questions on the Satisfaction Survey come from the CFI. The survey is short, ongoing (triggered automatically at activity completion) and confidential (results are aggregated by Global Practice and Region). Some 797 external clients of ASA activities were surveyed in FY16. Management reconfirms that the new ASA instrument will keep applying the World Bank Group Satisfaction Survey. Further refinements and guidance for results monitoring and management are under development. Management is also collaborating with IEG on the work to develop a new evaluation approach for knowledge services, and appropriate follow-up action will be taken once this is completed.
On 4B, the GP platform pulls content from Imagebank including all relevant IEG content. For example, a person searching the Knowledge base will be able to retrieve content from Imagebank (IEG included) as well as other GP content from the site. In terms of developing a framework to evaluate knowledge activities and consistently applyinmog it across the institutional portfolio, management expects to issue the new ASA--Advisory Services and Analytics--instrument in FY16, and IEG will continue to monitor if framework to evaluate knowledge activities is issued and applied consistently across the institutional portfolio. Perhaps one of the most significant weaknesses in this area is that there is very little done in terms of evaluation of knowledge (now ASA) activities, and thus little incentive to the providers in terms of achieving clear outcomes for the knowledge activities or applying lessons learned in similar past activities. Thus it will be essential that the self-evaluation framework for knowledge activities (now ASA) is developed and applied consistently across the institutional portfolio.
On 4C, management report indicates client feedback for AAA/ASA activities. It will be important to monitor in future comprehensive coverage of ASA activities.
As noted by IEG last year, the progress made in developing frameworks for 4B and 4C meant they were on track to be operational during FY16, as committed under MAR recommendations. During FY15, Action 4C was further implemented: CFI was systematically implemented and delivered to all AAA/ASA activities, upon their completion. CFI was also expanded to include lending clients, and rebranded to the World Bank Group Satisfaction Survey.
Additional Information from Management: On 4C, Management would like to clarify that CFI has been fully implemented for ESW/TA/TE/IE. This exercise has been so successful, that lending instruments are now applying too and the instrument has been rebranded to the World Bank Group Satisfaction Survey. Management can confirm that the new ASA instrument will keep applying the World Bank Group Satisfaction Survey.
To acknowledge the progress made under this recommendation, management suggests a rating of Substantial, one level above last year's rating of Moderate.
Progress made in developing frameworks for 4B and 4C. On track for frameworks to be operational during FY16, as committed under MAR recommendations.
Action 4A:
The following items were implemented in FY14:
1) ADM processes and instructions, including the ones for RAS, for AAAs ware revised to support new global practices and CCSAs. It was hardwired in the portal to improve the quality oversight.
2) Programmatic Approach was fully implemented in the Portal, including the accommodation of RAS and completions summary.
3) All AAA guidelines, including the RAS ones, have been updated to support the newly launched GPs and CCSAs.
Action 4B:
The following items were iomplemented in FY14:
1) IEG concluded feasibility study for the independent assessment/validation of AAA.
Action 4C:
The following items were iomplemented in FY14:
1) CFI14 has been recently launched as a regular annual exercise to focus on all AAA activities delivered to clients in FY14.
2) CFI a related processes including contact capturing has been implemented in Ops Portal. Relevant instructions have been created and disseminated.