Private Sector Advisory Projects
Chapter 2 | How IFC’s Self-Evaluation of Advisory Services and IEG’s Independent Validation Work
With the growth of its advisory services portfolio, IFC needed to develop an internal governance system that ensured the portfolio’s accountability and transparency throughout the project cycle. In creating such a system, IFC took a systematic approach covering all client-facing activities.1 It developed standard mandatory documents for the project life cycle, starting with an approval document, complemented by biannual supervision reports.2 IFC’s self-evaluation document, the Project Completion Report (PCR), launched in 2008, completes the project life cycle. That year also marked the start of the independent validation process IEG uses for this subset of advisory services projects.
IFC and IEG collaborated to develop a results-based project cycle that met minimum standards for evaluation, especially during conceptualization of PCRs, and that could be implemented at scale. Underlying the design of templates for approval and PCRs was the objective-based evaluation approach, which fulfills IFC’s need to hold project teams accountable for the development objectives they formulated at project approval while generating lessons that could be used to improve future interventions. IFC and IEG jointly developed the rating guidelines for PCRs, conducting several pilot tests to refine and validate the guidelines, which IFC staff members and IEG evaluators follow when preparing and validating, respectively, PCRs. PCRs and the associated rating guidelines are the keystones of the advisory services self-evaluation system, which concludes with IEG’s independent validation of PCRs through its Evaluative Notes. These client-facing advisory services projects follow an approved project cycle, illustrated in figure 2.1, and the teams working on the projects must prepare PCRs using the agreed methodology. IEG validates a random stratified sample of these PCRs every year. Since 2008, it has validated the PCRs for more than a thousand projects completed between FY 2008 and FY25.
Figure 2.1. Advisory Project Cycle
Figure 2.1. Advisory Project Cycle
Source: Independent Evaluation Group.
Note: IEG = Independent Evaluation Group.
PCRs serve a dual purpose. First, they are an integral part of advisory services governance because they mark the official closure of the project in IFC’s books. Second, they are self-evaluation documents that fulfill the institution’s needs for accountability and learning. Consequently, they are expected to tell the project’s story. PCRs not only report the degree to which a project achieves its development objective, as supported by evidence, but also reflect on factors that influenced the project’s effectiveness (drivers of success or failure) and how it addressed challenges encountered during implementation. According to the aforementioned PCR rating guidelines, “the project completion report should flow logically from the analysis of the data, showing a clear line of evidence (theory of change) to support the conclusions. Conclusions should be substantiated by evidence and appropriate analysis needed to establish a plausible attribution of said results to the project as well as clear context in which the results have been achieved” (IFC 2020, 4).
The PCR rating guidelines provide project teams, IFC staff members, and IEG evaluators with a common methodology for assessing and rating projects’ achievement of development objectives and identifying and extracting lessons related to internal or external factors that may have affected projects’ implementation and results.3 Projects are assessed and rated across three dimensions (figure 2.2):
- Development effectiveness reflects the extent to which a project achieved its intended objectives. It is a synthesis (not an average) of five underlying subdimensions: strategic relevance, output, outcome, impact (that is, results), and efficiency.4
- IFC role and contribution reflects the value added by IFC versus alternative providers of advice or lack thereof.
- IFC work quality assesses the performance of the IFC project team and has two subdimensions: project preparation and design, and project implementation and supervision.
Source: Independent Evaluation Group.
Note: IFC = International Finance Corporation.a. IFC project teams are not required to rate this dimension.
As noted earlier, IEG independently validates a sample of IFC PCRs each year for accountability and learning purposes. Projects are selected by applying stratified random sampling to the population of projects completed in the previous fiscal year. The stratified random sample has two main strata (region and primary business area) so that, within statistical limits, IEG can make observations about both the performance of IFC projects overall and key subareas important to IFC on a three-year rolling basis.
When conducting the PCR validation, IEG reviews a project’s primary sources of evidence (for example, project documentation, client reports, and evidence files), conducts interviews and discussions with IFC teams, and searches for postcompletion evidence and secondary data from external and investor sources. IEG records its evaluative findings in Evaluative Notes, in which it assigns projects ratings and provides its rationale for the ratings. For quality assurance purposes and for consistency across Evaluative Notes, IEG advisory services sector leads—senior IEG staff members familiar with the advisory services sector—peer-review Evaluative Notes before they are finalized.5
The validation process is deeply cooperative as well: IEG and project teams discuss the findings of Evaluative Notes at the draft stage. Discussions held during the validation process are key to maintaining the system’s integrity, ensuring ratings’ transparency, and giving IFC teams the chance to improve or qualify IEG’s assessments. The process also provides an opportunity for IEG and IFC to discuss how to improve results frameworks and quality of evidence in future projects, important feedback in cases of new types of projects undergoing the validation process for the first time.6
As described earlier, IEG evaluators follow an objective and rigorous approach in reviewing self-evaluations by project teams. Yet they face myriad challenges during the validation process, from lack of clarity of project objectives to weak theories of change, lack of reliable and available evidence, changes to project objectives and outcomes or impacts during project implementation, attribution issues, and difficulties in assessing expected and unexpected project effects. Over the years, IFC and IEG teams have worked together to address some of these issues and develop solutions. Some other issues, however, remain as tension points or differences in opinion between IFC project teams and IEG evaluators.
In the next two chapters, we discuss key challenges that may arise during PCR validation and how they have been dealt with, either at the corporate level or by IEG evaluators. We first focus on the assessment of project effectiveness, given its critical role in evaluating advisory services. We then zoom in on challenges and perspectives that go beyond assessing performance and effectiveness, focusing on issues related to objectives, changes in outcomes over time, theories of change, and work quality. Notably, we do not discuss IFC’s role and contribution directly, even though it is rated in PCRs; because there is typically little divergence between IFC’s and IEG’s assessments in this area, we do not focus on it in our discussion of evaluation challenges.
- The system covers solely single project-level work. The self-evaluation system was designed to assess effectiveness of individual projects; hence, it does not assess results of programmatic interventions that pursue an overall goal through a sequence of complementary projects implemented over time. Such programs are best covered by other IEG evaluation products such as Country Program Evaluations or thematic evaluations (https://ieg.worldbankgroup.org), which use the information from individual project-level validations (if available). Self-evaluations do not cover the following: (i) client-facing work that is primarily output oriented (for example, delivering a feasibility report) where there is no expectation by the IFC team that tangible behavior change (outcomes) by the client will materialize by project completion. Such projects are classified as client preparation because the advisory services are delivered with expectation to prepare the client for more comprehensive advisory services for which outcomes are expected or for possible IFC investment transaction; (ii) non-client-facing work that is geared toward developing IFC’s understanding on a certain topic, business development activities to develop new project ideas or develop or enhance a standardized IFC product or approach, or development of thought leadership papers or conferences to share knowledge and advocate for the value of IFC offering; and (iii) upstream activities, specifically collaborations and codevelopments, in which IFC acts as a potential investor. Where there are coordinated investment and advisory interventions, in which the advisory services project may be undertaken with the objective of improving the client’s operations or making the investment project more developmental (for example, mitigation of environmental and social effects or improvement to a supplier chain), the self-evaluation systems for advisory services and investment projects operate independently of one another, and the completion report for an advisory services project may be selected for IEG validation but not the report for the corresponding investment services project (and vice versa). The only way to address this shortcoming, in the case of an evaluation of an advisory services project, is for the evaluator to identify the related investment project and consider relevant investment information (and if available, the group’s validation of the completion report for the investment project) when preparing the advisory services Evaluative Note.
- At approval, advisory services projects are expected to present an outcome-oriented statement of the project’s development objective and develop a results framework, including relevant key performance indicators, baselines, and targets. IFC guidelines require development objectives for advisory services projects to focus on outcomes and impacts rather than outputs (the activities or tasks completed by IFC). Outcomes refer to changes in clients’ behaviors, knowledge, and practices, whereas impacts relate to the effects of those changes on clients or other stakeholders (such as suppliers, borrowers, and the public at large). Approval documents also include results frameworks, which reflect projects’ causal chains by listing key outputs, outcomes, and impact indicators with baselines (as relevant) and targets. IFC management and evaluation officers, who are part of project teams, provide advice and quality assurance related to projects’ results framework. Project supervision reports are prepared biannually and discuss implementation progress during the period they cover, issues, risks ahead, and changes in project plans, along with lessons learned. They also include analysis of results to date and self-rated progress in achieving
- The guidelines follow the evaluation criteria of the Development Assistance Committee of the Organisation for Economic Co-operation and Development.
- A project’s outcome is the dimension that has the most weight when its development effectiveness is assessed, and it is key to unlocking the project’s effects (impacts) on beneficiaries and stakeholders at large. Outcomes must be achieved within the typical duration of an advisory services project; in fact, a PCR is prepared only when sufficient time has passed for a project to have achieved its outcomes. Impacts, conversely, are difficult to achieve fully within a project’s life cycle and usually start becoming apparent only after project completion. A project might not have completely achieved its impacts at closure, but it can still be considered successful if its accomplishment of its outcomes is strong and it achieves other subdimensions of development effectiveness, such as strategic relevance, output, and efficiency.
- The sector leads also oversee onboarding and training of new evaluators and most importantly contribute to keeping the system updated and relevant by identifying gaps in the evaluation methodology and proposing changes in methodology and guidelines to IFC’s counterparts.
- IFC has a centralized group of monitoring and evaluation officers (known in IFC as results measurement specialists), who are distributed across work sectors. They are embedded in project teams but report to a central department that is responsible for (among other things) maintaining the integrity of the advisory services self-evaluation system. These monitoring and evaluation officers are expected to advise operational teams on their results frameworks and management and evaluation plans and activities and to make sure that reported results are evidenced and the IFC teams adhere to the rating guidelines for project completion reports. The monitoring and evaluation officers play an important role in the IEG validation process by supporting the IFC team in their engagement with IEG, including guiding teams on providing additional evidence supporting the drafting of comments provided by IFC project team to the IEG’s draft Evaluative Note.