Typical performance monitoring mechanisms

There is a range of performance monitoring mechanisms that can be used during both the construction and operations phases of a PPP, which are typically associated with payment (e.g. payment deductions to penalise underperformance) to incentivise the Project Company’s better performance. These mechanisms typically include:

• construction milestones reliant on completion of agreed sections of work, subject to quality requirements
  
  
• construction look-forward or look-ahead tests
  
  
• review and analysis of compliance with KPIs
  
  
• review of quality control and quality assurance procedures to ensure that quality systems are in place and effective
  
  
• informal audit processes, or conducting surveys and interviews to gauge user satisfaction
  
  
• independent monitoring by the Procuring Authority to verify the accuracy of the reporting delivered to the Procuring Authority
  
  
• independent calibration of measurement equipment used in the delivery of the service to verify its accuracy

Monitoring of performance will have different characteristics during the construction and operations phases. Construction milestones are the primary performance monitoring tool available to the Procuring Authority during the construction phase, while KPIs are the primary performance monitoring tool during the operations phase. Where the operation of existing assets takes place at the same time as construction and expansion activities (e.g. on a brownfield project or one with phased commencement of operations), operational KPIs and construction milestones may be combined.

Self-monitoring

The Project Company typically reports on its own performance following the quality management plans. These documents are developed by the Project Company and reviewed by the Procuring Authority. They set out activities, standards, tools and processes to be followed in order to achieve quality in the delivery of the project. The onus is on the Procuring Authority to ensure compliance of the Project Company with the prescribed quality plans, undertake spot checks, testing and physical inspections, and provide any sign offs and certifications required by the PPP contract. The ability to make enquiries against performance monitoring is an important right for the Procuring Authority. It is a right that needs to be exercised properly to ensure the Project Company is complying with the PPP contract and the Procuring Authority is receiving value for money.

The level of monitoring input from the Procuring Authority may depend on the quality of the self-monitoring procedures and systems deployed by the Project Company. It is common for these tasks to be completed in conjunction with an independent certifier during construction (as is discussed below) or with other technical consultants during operations.

Key Performance Indicators

KPIs are designed to allow the Procuring Authority to measure the level and quality of service that is being provided. They are a collection of measurable indicators of performance chosen to reflect how well the Project Company is providing the service that the project was designed to deliver.

Monitoring KPIs is the primary way the Procuring Authority ensures it is receiving the level of service prescribed in the PPP contract during operations. Service delivery should demonstrate soon after commencement whether the KPIs and the payment mechanism are working as intended, and if the associated contract drafting is clear in the recording of performance levels and application of the payment deductions. Where the KPIs and associated payment mechanism have been adequately designed, their application by the Procuring Authority should provide the Project Company with a real incentive to perform.

There is a conceptual difference in how performance monitoring should be implemented during operations depending on whether the PPP is based broadly on an availability payment model or on a user-fee model.

• Where the PPP uses an availability payment model, the Procuring Authority is the ultimate customer of the service being provided, and therefore has a pressing need to ensure that the quality targets are being met. For these types of projects, use of detailed KPIs is common.
  
  
• Where the Project Company earns its revenue from user fees or tariffs, it is holding some of the quality risk; a reduction in service levels and user dissatisfaction can lead directly to a reduction in revenue. As the Project Company is incentivised to provide a quality service to maximise its revenue, detailed KPIs become less crucial. However, the Procuring Authority cannot take an entirely hands-off approach. It is likely that there will be aspects where poor performance of the Project Company (e.g. failing to perform long-term maintenance or failing to provide a safe working environment) may not adversely affect the Project Company’s short-term revenue, but may have an adverse impact on the Procuring Authority. The KPIs in these projects should also consider aspects that do not lead to reduced revenue but nevertheless constitute reduced performance. For example, it can take a long time after the appearance of potholes on the shoulder of a road for road users or safety concerns appearing for road users or airport passengers to look for alternatives that may be less convenient to them.

KPIs will depend on the asset type. For example, KPIs may be punctuality indicators for rail projects, time of unavailability for power supply or transmission projects, lane availability for road projects, water leakage rates, social acceptance surveys, response times, etc.

The level at which to set KPIs when negotiating the PPP contract is an ongoing challenge. The study found examples where KPIs were considered very difficult to meet as well as examples where KPIs were too vague. The over-specification of KPIs can mean that payment deductions associated with the KPIs are so minor that monitoring does not provide value for money. The Project Company may also make a decision that the cost of compliance is higher than the relevant payment deduction, incentivising it to simply ignore the KPI. The correct level of KPIs and their flexibility is a complex topic in itself and not covered in detail in this reference tool.

Example – Different levels of KPIs

On the Segarra Garrigues Irrigation System project in Spain, there are requirements to repair all damage to certain irrigation facilities within 48 hours, regardless of the scale of damage. This has the potential to sour relationships between parties, with the Project Company and its contractors feeling that they are being held to an unobtainable standard, and the Procuring Authority being put under pressure to waive deduction rights. In another example from the study, the Project Company felt that it barely had to consider the KPIs, as they were set so low, which suggests that the KPIs were not designed to incentivise good performance or penalise failures in performance standards.

For more information, see the Segarra Garrigues Irrigation System Case Study.

Payment mechanisms

Payment mechanisms come in the form of a range of financial incentives including increased unitary payments to the Project Company, lump sum payments (e.g. attached to a construction milestone), deductions, agreed compensation, adjustment of rights to receive revenue from the project, changes to the required level of investment by the Project Company, etc. The key is that there will be either a positive or a negative impact on the Project Company’s revenue depending on the level of performance.

Failure to meet KPIs will typically result in a corresponding payment deduction or agreed compensation payable to the Procuring Authority

Payment mechanisms during construction are typically tied to construction milestones, subject to quality requirements, subject to quality requirements. Many PPP projects will involve no payment to the Project Company during construction, as payment is linked to availability of services or user fees during operations. This structure incentivises the Project Company to complete work within the agreed timeframe, as availability of revenue is dependent on completion of the construction works, and so late delivery will erode potential Project Company profit (in addition to any agreed compensation payable because of the late delivery). This structure still provides the Project Company with the autonomy it needs to properly manage its contractors.

The payment mechanisms during construction may be one-off lump sum payments for the completion of milestones implemented through an increase in the unitary payment as milestones are reached, or as a deduction from the unitary payment until the milestones are reached.

Example – Milestone payments

Milestone payments were used successfully in a number of case study projects. On the Zaragoza Tramway project in Spain, 10% of the subsidy from the Procuring Authority was dependent on the achievement of certain milestones, while a substantial payment was available on completion of the Port of Miami Tunnel project in the USA. These were seen as useful incentives for on-time delivery.

For more information, see the Zaragoza Tramway and Port of Miami Tunnel Case Studies.

Independent monitoring

It is common to employ independent certification/verification, in particular during the construction period. This involves an independent certifier that is commonly appointed during construction under a tripartite agreement between the Project Company, the Procuring Authority and the independent certifier to monitor compliance with the output specifications, overall progress and quality control. In addition, a third party technical advisor is typically appointed by the lenders to monitor construction progress and approve Project Company loan drawdowns when payments are due to the construction contractor.

For complex projects, the independent certifier appointed by the Procuring Authority and the Project Company may continue its role during the operations phase in a more limited or ad hoc capacity.

An additional safeguard is the lenders’ oversight of the Project Company’s compliance with the PPP contract. The interests of the lenders and the Procuring Authority are aligned on this issue of compliance, as any material underperformance by the Project Company will ultimately affect the Project Company’s cash flows (through payment deductions) and the Project Company’s consequent ability to service its debt. However, it should be noted that this scrutiny may be limited. The lenders exercise this scrutiny through their independent technical advisor’s reporting on the project’s operational performance, which is typically not shared with the Procuring Authority.

Go to 3.2.2 Guidance