The authors identify three dimensions that are critical for FMIS reform to contribute to improved budget management: (1) the diagnostic, (2) the system’s lifecycle, and (3) the system’s coverage and utilization...Deficiencies in any one of these affect the ability of the investment to deliver results, and the authors argue that a programmatically coherent approach across all dimensions is necessary for the reform to approximate the FMIS production frontier.
There is a wealth of experience in the World Bank in implementing projects and supporting reforms. But how do we draw meaningful, and practical lessons from this experience? After all, reforms are context specific and what may work in one country does not necessarily apply to others.
A recent World Bank working paper that draws lessons from evaluating financial management information systems (FMIS) may provide some insight. The authors took the following steps:
Developed a conceptual framework of how change is expected to happen in FMIS reforms. What are key dimensions in the reform process that would lead to improved outcomes?
Drew on purposeful field-based FMIS project evaluations to populate the conceptual framework with specific lessons.
Reflected on the external validity of these lessons.
The FMIS conceptual framework. The authors identify three dimensions that are critical for FMIS reform to contribute to improved budget management: (1) the diagnostic, (2) the system’s lifecycle, and (3) the system’s coverage and utilization. The diagnostic phase covers a review of relevant public financial management (PFM) deficiencies, the legal and institutional framework, control protocols, and business processes. The system lifecycle lays out what it takes to make an FMIS operational. Lastly, the coverage and utilization of an FMIS will determine its scope and effectiveness. The framework is shown in figure 1 below.
Deficiencies in any one of these affect the ability of the investment to deliver results, and the authors argue that a programmatically coherent approach across all dimensions is necessary for the reform to approximate the FMIS production frontier. This may appear to be an obvious feat. Clearly, if procurement fails, there will be no system and no results to show. Similarly, it is axiomatic that a system that is not effectively used will not deliver the desired results. In practice, however, we often think linearly about the task at hand and a ‘failure to notice’ can be a key binding constraint to learning and an effective engagement (see Hanna et al (2012)). In terms of FMIS reform, a focus on one dimension alone (e.g. implementing a system) may not be sufficient to achieve better PFM results.
Using project level evaluations to develop lessons. The authors drew on 5 project performance assessment reports (PPARs) that act as field based case studies, and a review of the history of World Bank FMIS experience as documented in implementation completion reports (ICRs) and IEG reviews thereof (ICRRs). Lessons were synthesized and mapped to the various stages of the stages of the framework.
Among the many issues the authors address, are: what factors are important to consider in a diagnostic? What is an enabling environment, and what may be the implications of setting up an FMIS in the absence thereof? Can expediting the speed of transactions mean ‘doing the wrong things faster’? Getting a system up and running or replacing an existing system is no easy task. What stumbling blocks should we look out for, what has worked, where and why? Procurement challenges are endemic and keep slowing down the process. Where has procurement worked well and why? Once the system is operational, how do we actually get people to use it and ensure good coverage? What consequences does it have for budget management if large transactions happen outside the system? How reliable is transaction processing and what are the implications for the integrity of FMIS reporting?
Reflecting on external validity. Drawing lessons from five case studies requires careful consideration of the relevance of these for other contexts. Five cases cannot be representative for all regions and countries across the development spectrum. To address this, an effort was made to triangulate information from the literature, World Bank project documentation, and the PPARs to assess convergence of findings and provide important contextual factors that may inform policy makers on the relevance of the findings to their specific situation. While context plays an important role, focusing on mechanisms (as this report does) and then judging whether a mechanism is likely to apply in a new setting has a number of practical advantages for policy making. Bates and Glennerster (2017) provide a four-step schematic on how to assess whether particular findings can be applied to a specific setting.
On some issues, the report was innovative and identified lessons that were previously not explored. For example, the authors find a striking pattern across countries in the transactions profile: few transactions tend to make up the bulk of the budget (see figure 2). This information could be used strategically for a differentiated control strategy with high value transactions being subjected to rigorous ex-ante control and low value transactions to less rigid ex-post control. This could expedite results in terms of expenditure control, whist enabling access to smaller amounts of money and the flexibility necessary for service delivery. This may not apply to all countries, but practitioners are encouraged to reflect on this for their setting.
Figure 2 Sample transactions profile by number of transactions and size of budget
The nature of other arguments is axiomatic and external validity needs no justification. For example, the system will only benefit the transactions that are routed through the system. If a large share of the budget is executed outside the system, this will undermine the system’s ability to contribute to improved budget management. Similarly, if the transactions a system captures are suspicious, any reports the system generates from these data will also be suspicious. These arguments hold regardless of context. They may be obvious, but this does not diminish their importance. The review for example found that frequently budget coverage is a problem and the use of FMIS outputs / reports with insufficient attention to the underlying quality of transaction data can produce misleading conclusions. Since a lot of economists and sector experts frequently draw on FMIS outputs, the integrity of transaction processing concerns them and they too should therefore be interested in joining such a conversation.
We sincerely hope that visualizing programmatic coherence and outlining lessons alongside it will help practitioners during the reform process, and pave the way for better attribution of FMIS investments to larger PFM outcomes.
 In a separate IEG working paper the authors developed a diagnostic framework that can be used to assess FMIS deficiencies. It can be accessed here.
 Hanna, Rema, Sendhil Mullainathan, and Joshua Schwartzstein. 2012. Learning Through Noticing: Theory
and Experimental Evidence in Farming. No. w18401 National Bureau of Economic Research.
 Bates, Mary Ann, and Rachel Glennerster. 2017. The Generalizability Puzzle. Stanford Social Innovation Review.