Improve monitoring and evaluation of ECD interventions during and after project closure to strengthen evidence on their medium- and long-term impact. Common ECD indicators should be developed and tracked across Bank operations to permit aggregation of results across Bank projects. In addition, follow-up studies should be undertaken to better understand the long-term impact of ECD interventions.
This evaluation is not able to provide any aggregation of changes in outputs or outcomes, as there is no consistency in the Bank's monitoring and evaluation. Within the Results Frameworks analyzed outputs rather than outcomes were typically tracked. When outcomes were noted, they were health measures such as infant, child or maternal mortality rates, which are not attributed solely to the interventions and do not capture impacts on children's development. Twenty percent of Results Frameworks from investment lending planned to measure changes in at least one child development domain. There is a huge need for more harmonized monitoring and evaluation of ECD interventions across the Bank, as well as need for tracer studies to be employed more frequently in projects, and follow-up studies to better understand the long-term impact of ECD interventions.
WB: Agree. The Bank has been a leader in the area of impact evaluations for ECD intervention, in part through the Strategic Impact Evaluation Fund, and this work will continue to be supported. In addition the Bank will continue to work with external partners on ways to better monitor ECD outcomes, including through the Measuring Early Learning Quality and Outcomes Initiative that aims to construct a global set of indicators for ECD. Given the wide variety of ECD interventions and outcomes (e.g., education, health, nutrition), such indicators could be developed by broad type of intervention.
Action 4a: The GSG will promote increased consistency in the use of specific result indicators across similar types of ECD projects (e.g. preschool interventions early stimulation and/or nutrition etc.)
Indicator: consistency in indicators used across similar types of ECD projects
Baseline: ECD projects typically include relevant result indicators, but there is limited consistency in the results indicators that are used across ECD projects.
Target: (a) A selected number of key indicators are identified for the most prevalent types of ECD interventions and (b) at least half of new ECD projects (or projects with an ECD component) under preparation incorporate these new indicators in their results frameworks
Timeline: FY17 for Target (a) and FY19 for Target (b).
Action 4b: Promote longitudinal studies in ECD (to track impact over the medium to longer term) and consolidate the findings.
Indicator: Availability of longitudinal data from Bank-led ECD impact evaluations.
Baseline: Most Bank-led ECD impact evaluations focus on measuring the short-term impacts of the intervention(s), but a few follow-up surveys are currently underway or being considered.
Target: A review of the evidence gathered through Bank-led longitudinal studies in ECD will be carried out.
Timeline: FY19
IEG Response: The ECD GSG has made substantial progress to improve both longitudinal data and monitoring and evaluation data. A common results framework has been adopted for the first wave of countries participating in the Early Years Initiative. Baseline data is being collected to provide common indicators among these countries. Other action items are on track and likely to be completed by next fiscal year. The Bank is also involved at the global level to promote more coordinated global measurement of ECD. Next year the Bank will be launching a global, longitudinal study in 7-10 countries. The Strategic Impact Evaluation Fund is reviewing findings from its portfolio that provide longitudinal evidence from 33 impact evaluation from countries such as Colombia, Malawi, Jamaica, and Nepal. This review is expected to be completed in FY19. SEIF has developed a new tool kit with a section devoted to helping researchers improve the measurement of ECD. All of these initiative align well with targets in the action plan.
As part of the Investing in the Early Years initiative, a common results framework has been adopted for all "first wave" countries to track key cross-sectoral indicators at country and project level. A baseline is being determined and will be tracked annually. This fulfills Target (a) for Action 4a and we are on track to fulfill Target (b) for Action 4a.
The Bank has continued to engage in promotion of more coordinated global measurement in ECD, through the Measuring Early Learning Quality and Outcomes (MELQO) initiative. Through this work, comparable measurements of child development and preschool quality have been field tested in eight countries. In FY18, the Bank will be launching a global, longitudinal study, to measure child development and quality of preschool settings, at population level in 7-10 countries. The Bank is working with key foundation and other agency partners to ensure this effort will yield high-quality and globally comparable data to measure children's cognitive and socioemotional development throughout the early years and beyond.
The Strategic Impact Evaluation Fund (SIEF) has continued to review findings from 33 evaluations within the portfolio which focus on ECD. There are several evaluations in the portfolio with longitudinal follow-up (e.g. Colombia, Malawi, Jamaica, and Nepal). In FY19, the proposed review of longitudinal studies will be completed (Target (a) for Action 4b). This year, SIEF has produced an updated version of the ECD Toolkit which provides guidance to researchers and task teams on tools to be used to measure ECD. Through SIEF-commissioned research, researchers from New York University's Global TIES for Children analyzed select psychometric properties of the IDELA instrument to assess its suitability for use in program evaluation and making global comparisons of children. This coming year, SIEF will invest in more general psychometric research of all the most frequently used instruments in the SIEF portfolio to learn the limits of interpreting evaluation results of ECD outcomes.
IEG Response: The ECD GSG provided a peer review service of all operations under preparation this fiscal. This review provided feedback which created some consistency across results frameworks, which is a step forward. Additional efforts have been initiated to develop indicators for particular interventions in thinking ahead to the SDGs. This work will be shared with staff across HD next fiscal year, which should bring more consistency across project result framework, as well as improved indicators. If there is greater consistency across results frameworks, aggregation of results may be possible in later years. Progress is consistent with the target established for next fiscal year. In relation to tracking of longer-term impact, 31 existing impact evaluations plan to collect longitudinal data. These data will bring additional evidence than found in IEG's ECD Systematic Review and contribute to the limited body of knowledge. The Bank's action plan indicates that review of this evidence will be done in FY19.
For Action 4 (a): The ECD GSG helped ensure (through peer review of all ECD projects under preparation in FY16) a certain degree of consistency in results frameworks. Efforts are underway to further define a consistent results framework for all investments in the early years (in line also with the relevant SDG targets), and this will be systematically shared with HD teams in FY17.
For action 4 (b): A number of impact evaluations in ECD (out of a total of 31 in FY16) include plans to collect longitudinal data, which would provide a significant body of evidence to inform the debate on medium to longer-term impacts of ECD interventions in LICs and MICs by FY19.