#6. Focus resources on high risk contracts for prior review, reducing emphasis on value thresholds. Improve the use of risk monitoring tools and risk mitigation methods, through the analysis of risk data from PRAMs and post-procurement reviews, for example by clustering to draw out patterns and trends and through streamlining details of the PRAMs instrument.abc
The Bank puts considerable emphasis, in terms of its present risk management framework, on ex ante risk controls through mechanisms such as prior review and clearance thresholds, which require contracts above certain values to be prior reviewed, by procurement staff at increasing levels of seniority, depending on contract value. Yet findings show that prior review instruments at best partially reflect country or project risk, and as such, could be relied on less as risk control mechanisms. There is likely to be scope to reduce the risk efficiency trade off by reducing the share of contracts that are prior reviewed and focusing prior review on the highest risk contracts. Bank procurement staff acknowledge that current review procedures are helpful for controlling fiduciary risk. Nevertheless, on balance, there was a view that current prior review threshold could be raised with beneficial impacts on workload and clearance time, and only limited impact on risk or competition.
Additional risk that this may imply could be mitigated by better use of Post Procurement Reviews (PPRs) and Independent Procurement Reviews (IPRs). This need not imply an increase in their numbers but rather, a more strategic use of their findings. There is limited analysis of the effectiveness or outcomes of pivotal risk management tools such as the procurement threshold system, the content of post procurement reviews, or data collected through the most recent tool, the P-RAMS instrument, in terms of analyzing content, tracing trends, or correlating risk management and procurement outcomes.
The P-RAMS template and process can be cumbersome and time consuming, especially with multiple implementing agencies, and exacerbated by multiple sequencing being conducted in far too short a time span. Its template, which limits discretionary response, may contribute to fostering a rigid "check the box" approach which limits added value. Risks identified are not weighted or prioritized and can lead to misleading 'averages' or risk in situations where a small number of high risks dominate overall project risk. While the P-RAMS system has a facility for procurement staff to override automatic ratings, it is not transparent and there is little evidence that it is exercised. The P-RAMS template could also further sharpen its focus on fraud and corruption (F&C) by featuring F&C risk as an additional Risk Factor.
WB: Partially agree. # 6. The prior reviews are not based on value thresholds only. The approach is a combination of risk and value. However, Management agrees that more focus needs to be placed on higher risk/higher value activities, reducing the review of lower value and/or lower risk contracts. In practice, this will mean reducing the number of prior reviews of lower value and/or lower risk contracts, as such, increasing the Banks risk exposure in these areas. However, Management agrees with IEG that this will have "only limited impact on risk and competition" - in FY13, 80% of prior reviews undertaken (by volume), account for 6% of the total value of these contracts. Management proposes to update its approach to prior-reviews, reflecting IEG's findings, releasing resources so they can focus more effort on the highest risk and/or highest value contracts.
Management agrees with IEG that the institutional system to manage procurement risks can be sharpened in focus, better integrated with other systems, and made better use of in terms of data input and analysis of findings.
PRAMS can be simplified and more guidance to staff on how to use the tool can be provided. Reporting tools can be built to address analytical needs. F&C dimensions of risks can also be more explicit within the PRAMS tool.
Action 6 (a) : Develop strategy/guidance to determine country/sector/agency thresholds based on risk and market profile define high risk/complex procurement and develop the appropriate level of prior review required
Indicators: Bankwide strategy/guidance developed on risk thresholds
Baselines: Thresholds are low and have limited correlation with risk and principles for setting thresholds vary across Bank regions
Targets: Strategy/guidance developed and revised risk thresholds implemented
Timeline: FY 15.
Action 6 (b) : Improve the strategic use of post procurement reviews
Indicators: Guidance note prepared on better use of post procurement reviews for risk management.
Baselines: Post procurement reviews are not systematically used.
Targets: Guidance prepared on strategic use of Post procurement reviews
Timeline: FY 15.
0328-01 - Strategic guidance on thresholds. In the absence of a further update from Management IEG refers to the Management Update to the Board, discussed in July 2017:
The World Bank has begun to focus its prior review activity to optimize performance.
Prior review is being streamlined, with an overall reduction of 46% in numbers in FY16, as
compared to the simulated FY14 baseline - from 11,550 to 6,200 prior reviews (see Figure 1).13
Prior reviews performed below the thresholds set are decreasing. From the FY16 data entered so
far, 37% of prior reviews below the thresholds were undertaken by Procurement Staff - with the
balance undertaken by Task Leaders (TLs). Management is implementing country-specific
approaches to ensure minimum exceptions in the future, with appropriate justification. Staff
resources released from prior reviews have been refocused on the most complex activities, including
new approaches and methods under the Procurement Framework, and capacity building.
While IEG notes that the core issue is the clarity of threshold setting rather than prior review reduction, IEG accepts that the reductions have been based on new guidance on threshold setting, which is in the direction of the recommendation.
0328-02 - post procurement reviews. IEG has already noted the booklet prepared on post-procurement reviews in 2016.
On balance IEG deems this condition to be met.
Unchanged from 2016. All actions identified in the Action Plan have been completed.
Latest progress update on the agreed actions on this recommendation are as under:
(i) This recommendation requires development of strategy/guidance to determine country/sector/agency thresholds based on risk and market profile and define high risk/complex procurement and develop appropriate level of "prior review" required. The New Procurement Framework (NPF) has streamlined prior review to free up resources to support other features of the framework and could save significant procurement resources. Under the NPF, the Bank's procurement prior review is focused on high-risk and/or high-value contracts and is applied through mandatory thresholds. Annex 1 of the Procurement Procedure sets out the new risk-based thresholds for different kinds of contracts that apply both to new projects following the NPF and to existing projects. Specific targets will be set to streamline prior reviews. The approach to streamline prior review is complemented with the strategic use of additional oversight instruments to be applied following a fit-for-purpose approach, for example, integrated fiduciary reviews, HEIS, increased oversight and monitoring on contract management, use of third-party assurance, and risk-based post reviews.
(ii) Improve the strategic use of post procurement reviews: Management has issued a methodology for procurement post reviews, as explained in the Guidance for Procurement Post Review. This methodology, which was informed by IAD's advisory review, is more targeted and risk-focused than the former post review. The guidance outlines the agreed risk-based methodology for sampling contracts to be subject to post review, with an annual target of post-reviewing 10% of contracts in the Bank wide portfolio that have not been subjected to prior review. The strategy also provides opportunity for relying on countries' own supreme audit institutions and other third parties to conduct post review.
In addition, IEG is happy to note that Project Procurement Strategy for Development (PPSD) is intended to be a "living document" prepared by the borrower taking into account risks (operational context, client capability, procurement complexity, and market risks) which together with the Bank's analysis of the risk of the project (Procurement Risk Assessment) in turn should inform decisions on Bank's prior reviews and ex-post. It is also noted that in addition to project-level risk assessment, the NPF introduces the concept of activity level risk assessment to drive procurement and fiduciary decisions. Management has reviewed the PRAMS risk assessment methodology and issued updated Guidance to staff. Management has also integrated PRAMS with the STEP system to enhance the ability of staff to make adjustments to the procurement risk ratings and to further ensure transparency and reporting capability of the system.
IEG shall await outcomes of the use of new risk-based prior and post review methodologies.
The NPF has streamlined prior review to free up resources to support other features of the NPF, including country capacity building. Analysis of the framework shows that initiatives to focus prior-review activities can save significant procurement resources (up to the equivalent of 60 Bank staff) resources that can be used, for example, to support HEIS at key stages of the procurement process for operations covered under paragraph 12 of OP 10.00. Specific targets will be set to streamline prior reviews. Under the NPF, the Bank's procurement prior review is focused on high-risk and/or high-value contracts and is applied through mandatory thresholds (Annex 1 of the Procurement Procedure sets out the new risk-based thresholds for different kinds of contracts) that apply both to new projects following the new procurement framework and to existing projects. The approach to streamline prior review is complemented with the strategic use of additional oversight instruments to be applied following a fit-for-purpose approach, for example, integrated fiduciary reviews, HEIS, increased oversight and monitoring on contract management, use of third-party assurance, and risk-based post reviews (discussed below). Management has issued a methodology for procurement post reviews, as explained in the Guidance for Procurement Post Review. This methodology, which was informed by IAD's advisory review, is more targeted and risk-focused than the former post review. The guidance outlines the agreed risk-based methodology for sampling contracts to be subject to post review, with an annual target of post-reviewing 10% of contracts in the Bankwide portfolio that have not been subjected to prior review. The strategy also provides opportunity for relying on countries' own supreme audit institutions and other third parties to conduct post review. Procurement Risk Assessment: the Project Procurement Strategy for Development (PPSD) is intended to be a "living document" prepared by the borrower taking into account risks (operational context, client capability, procurement complexity, and market risks) which together with the Bank's analysis of the risk of the project (Procurement Risk Assessment) in turn should inform decisions on Bank's prior reviews and ex-post. In addition to project-level risk assessment (which is currently captured in Project Risk Management System, P-RAM, by risk rating multiple aspects/dimensions relating to the project and/or implementing agency), the new framework introduces the concept of activity level risk assessment to drive procurement and fiduciary decisions. Management has reviewed the PRAMS risk assessment methodology and issued updated Guidance to staff. Management has also integrated PRAMS with the STEP system to enhance the ability of staff to make adjustments to the procurement risk ratings and to further ensure transparency and reporting capability of the system.
IEG notes the global consultation on prior reviews and concerns about funding issues. IEG has requested and received information on the global consultations including a summary of the second consultation (attached). The summary finds that the proposed new procurement framework would require a culture change among World Bank staff in order to be effectively implemented. World Bank staff would need to understand the intricacies of country contexts, legal systems, and sectoral issues. Consultation participants also pointed out that the World Bankâs procurement function would need to be adequately funded. They agreed that focusing prior reviews on high risk/high value procurements would help free up some of the necessary resources. Stakeholders pointed out that some risk was inherent in an increase in staff discretion, which would need to be mitigated through adequate training and oversight.â
IEG notes that specific conditions for the use of the Borrowersâ bidding documents and processes below ICB are set out in the new procurement framework, Regulations for Borrowers paragraph 7.20 of the Section VII.
IEG would also like to review the sampling process suggested for procurement that is below the threshold and an update of the agreed final method as agreed with IAD. IEG would also welcome some evidence of instances where countries' national authorities are being invited to participate in post-reviews,
Reflecting IEG feedback, Management has consulted globally on proposals to optimize prior review. Proposals to streamline prior review are currently awaiting Board approval, if agreed a program of prior review optimization will be implemented by the GGP with reporting to OPCS.
The Bank will focus its prior review activity to optimize performance. The new Procurement Framework allows the Bank to focus most of its prior-review resources on high-risk and/or high-value procurements, reducing reviews of lower-value and lower-risk procurements to free up resources.
Management has set an absolute value and risk threshold that determines mandatory prior review, below which prior review should not be carried out (except in fragile and conflict-affected situations and other specified circumstances). For procurements that are not prior-reviewed, Management has outlined a sampling method for post review and will obtain assistance from IAD to refine it. Consultations also showed broad support for the appropriate use of a country's own supreme audit institution and other third parties/civil society organizations to undertake post reviews of procurements.
Management is currently reviewing PRAMS to streamline the approach and to join it up with other risk tools in the proposed new Procurement Framework. Management has already implemented SORT as a mechanism to integrate FM and procurements risks to create a fiduciary profile for a project.