Challenges Related to Statements of Objectives

The methodology IFC uses to evaluate advisory services projects assesses achievements against predefined objectives and goals. Therefore, the clarity and accuracy of the objectives in project approval documents are crucial for effective validation of PCRs. The rating guidelines for PCRs establish that the statement of a project’s development objective sets the purpose toward which an intervention is directed based on the identified problem, market failure, or opportunity that the project intends to address. A statement of objectives must include the changes the project is expected to generate (that is, an outcome-level objective) and the expected effects of those changes (that is, an impact-level objective) and be accompanied by specific indicators, baselines, and targets. Box 4.1 illustrates a clearly stated objective, one that describes the overall aim of the project alongside specific objectives; defines specific, measurable, achievable, results-oriented, and time-bound (SMART) metrics at the outcome level; and finally includes the effects of outcomes or the level of impacts.

IEG evaluators sometimes encounter statements of objectives that are unclear, are convoluted, or focus more on outputs or indicators than outcomes, making the basis for evaluation unclear. Box 4.2 illustrates a problematic objective statement in which the outcome-level objective does not accurately reflect the overall aim of the project; the indicators are standard ones and focus mainly on the delivery of outputs and early outcomes, failing to reflect achievement of the project’s development objective; and the impact-level objective is missing (the one appearing in the box was added by IEG).

In such cases, IEG evaluators, before assessing the project, must first clarify the objective further or reconstruct it based on the project’s broad objective, gaps to be addressed (market failure), results chain, key performance indicators, and other information included in official project documents (such as the description of expected benefits and beneficiaries of the project, details of project components, and other elements of project design) and consultations with project teams (IFC 2020). In addition, the IFC and IEG teams must agree on the refined or reconstructed statement of the project’s objective before the evaluation can proceed.

Projects’ objectives may be revised during the project life cycle, further complicating assessment of project performance. Such revisions may change evaluators’ approach to validation compared with how they would have proceeded based on the original objectives. Project objectives may need to be adjusted in two instances. The first is new or updated information becoming available. Because of the nature of advisory services work, fully fleshed-out assessments (for example, client or market assessments) that inform objective setting and project design might be conducted in the early stages of project implementation if the project budget permits. These assessments’ findings might shed new light (for example, on the project’s assumptions, understanding of the problem, and stakeholders’ views) that might require adjusting the project’s objectives and design to better reflect the actual situation. The second instance relates to changes in external factors, from natural disasters to changes in clients’ priorities, that might also require a rethinking of the intervention.

In the past, the methodology for evaluating advisory services projects still held the project accountable for the original objectives in such cases, regardless of the circumstances, potentially creating a bias toward a negative rating and, most important, risking discouraging operational teams from applying adaptive management and supervision. To deal with this issue, IFC and IEG agreed on two solutions: introducing a grace period for making changes and giving teams the option to restructure projects. The grace period allows teams to adjust project objectives, scope, and design in the early stages of implementation. Changes to the objective, if made within the grace period (following the 25 percent threshold criteria),1 are accepted as final and hence replace, for the purposes of evaluation, the original objectives set at approval.

Beyond the grace period, changes can be made through formal restructuring of the project, which triggers a special approach to evaluation, the “split rating” approach, in which project results are assessed against both the original and revised project objectives. The split rating weighs pre- and post-restructuring performance using the share of actual implementation expenditure before and after the restructuring, favoring project management that identifies issues and makes decisions early in the process. The split rating approach also aims to provide a disincentive for “lowering the bar” late in project implementation—that is, curtailing project ambitions to match achievement in the field rather than to account for changes in context.

Box 4.3 illustrates the original objectives in just such a case, in which a project’s design changed significantly during implementation. The project was revised several times, dropping its main impact indicators (that is, loan recovery rates), removing work on the regulatory framework (component 1), and adding a new component, a revision of the bankruptcy draft law. The project also eliminated several activities, reducing the scope from a market-level intervention to a more focused approach that dealt only with financial institutions.

Depending on when these changes were made, the project’s development effectiveness could be considered successful or unsuccessful:

• If the project made the changes within the grace period, the evaluation would hold the project accountable for the revised components and indicators, without considering the original ones.
• However, in this actual case, changes were made after the grace period, and therefore the split rating approach was applied. The project was still held partly accountable for the original objectives, though achievement of the revised objectives was also recognized. The project did not achieve its original objectives because it did not work with a regulatory framework for insolvency practitioners. However, the newly added component, which centered on bankruptcy law, achieved its objective. The project was restructured when the project’s implementation expenditures were 40 percent of total implementation expenditures; therefore, based on the weighted approach, the final assessment was positive.

Another common challenge is that teams might not factor in the context of project implementation while setting project objectives. At the approval stage, a project’s potential to generate development results is often presented in its best light, leading to overly optimistic objectives. The root cause is IFC’s underestimating or underplaying well-known risks tied to country conditions (for example, political and social risks) or client conditions (for example, commitment, capacity, and resources) to get the project approved by management or funded by donors. At the time of validation, IEG follows rating guidelines for PCRs and holds the project accountable for accomplishing its stated objectives. When it does so, IFC might claim that IEG did not consider the difficult environment in which the project was being implemented; in turn, IEG would argue that the level of ambition of projects’ objectives set at approval should be commensurate with the risks. These differing perspectives often result in divergent opinions.

For example, a project to be implemented in a country that experiences political instability should set more modest objectives and targets than if it were being implemented in a politically stable country. Consider the case of an advisory project in an African country that aimed to reduce costs of the country’s agricultural inspections by 15 percent, attract $20 million in private agribusiness investment, create 400 jobs, and improve the country’s tax administration (internal IFC document). The project fell short of achieving its goals, despite two extensions, a lengthy implementation period of six years, and a budget increase from $2.5 million to $5.4 million, because of a variety of challenges, including election disruptions, budget constraints, high staff turnover, low staff capacity, and a health crisis. The project team indicated that it took twice as much effort to deliver half as much in outcomes in the country where the project was implemented than in other countries. However, the original project objectives set and approved by IFC should have considered these potential limitations.

Challenges Related to Weak Theories of Change

To fairly assess a project’s contribution to desired development outcomes, evaluators must first understand the logical steps required to achieve the outcomes, or in other words, the intervention’s theory of change. A good theory of change illustrates both what a project aims to achieve and how it plans to do so within the specific project context.

Although IFC documents related to project approval include results frameworks, which reflect projects’ causal chains by outlining key indicators regarding output, outcome, and impact with baselines and targets, some projects rely primarily on a few standard key performance indicators to establish a results chain. The resulting theories of change are weak or incomplete and fail to capture fully the rationale behind a project’s interventions.

For example, an IFC project developed training to enable a client to improve capacity and performance among its distributors, aiming to ultimately increase rates of cell phone penetration in rural areas. The results chain presented in the approval document did not fully capture the project’s rationale or the logical steps needed to achieve its ultimate goal. To address these shortcomings, IEG evaluators thoroughly reviewed project approval documents, including details on market failures the project aimed to address, the project description, and IFC’s planned support. IEG identified project activities and outputs, defined expected outcomes based on these outputs, and assessed critical and underlying assumptions, revising the original theory of change as reflected in table 4.1. Considering the overall goal and type of training, IEG added new outcomes related to the enhancement of distributors and their staff’s skills and increased business performance of the distributors, which the project’s original theory of change did not reflect. IEG also included additional impacts—increased business performance of the client and continued performance of distributors—both results of outcomes identified. This revision strengthened the evaluation process, providing a clearer picture of the project’s impact and effectiveness.

Source: Internal International Finance Corporation document.

Note: IEG = Independent Evaluation Group; IFC = International Finance Corporation.

Assessing Work Quality

IFC’s self-evaluation system includes ratings of work quality, which assess the extent to which IFC followed its prescribed operational procedures.2 The rating guidelines for PCRs include detailed descriptions of categories of work quality that evaluators also consider during validation of PCRs. These categories include appropriate market or needs assessment, risk appraisal, project design, client commitment and involvement in project preparation and design, and timeliness and quality of output delivery.

Ratings of work quality enable projects’ performance to be distinguished from teams’ performance, reflecting the reality that a project may fail for reasons unrelated to the project team’s performance. Assessment of work quality is quite appropriate, since advisory services projects are implemented in-house by an IFC team comprising a mix of staff members and consultants. In some cases, consulting firms are contracted to deliver advisory services, but they are actively managed and supervised by the IFC team. However, before IEG began rating work quality separately, IFC teams perceived projects’ performance ratings as evaluations of team performance. In some cases, this led to disagreements and difficulty in reaching consensus with IEG during the project validation process. The separate assessment of work quality offers a constructive avenue to evaluate team performance independently of project outcomes.

A key challenge when assessing project work quality is the hindsight trap. Evaluators are expected to assess work quality considering the information available to the team at the time decisions were made and work was conducted. Evaluators must determine whether those decisions or actions were informed, that is, whether they were made taking into account all the information available at the time, as opposed to judging whether decisions were right or wrong considering later events and project results—information not available to the project team when it made its decisions. Evaluating whether decisions and actions were informed could include assessments of risks and their mitigants, with a full understanding of pros and cons and with adequate internal discussions at the right levels.

Assessments of work quality usually start with the assumption that IFC’s work is in line with standards of good project management—namely, corresponding to a satisfactory rating according to rating guidelines. Any issues identified by the evaluators that point out potential shortcomings in work quality should be fully unpacked and considered with respect to whether the shortcomings resulted from faulty work quality (for example, inadequate client assessment) or from something IFC could not know or control at the time. In making such assessments, evaluators are cautioned against suggesting alternative courses of action that are not rooted in lessons learned from similar projects or best international practices.

To assess work quality, evaluators rely on evidence embedded in IFC’s internal systems, which generate and systematically record a significant amount of information on project management throughout implementation. Biannual supervision reports, comments from approvers, internal memorandums, back-to-office reports, midterm evaluations, and similar documents usually provide a good picture of how a project was designed and managed. Internal governance of advisory services projects also provides benchmarks for comparison.