The objective of the data collection on 250 projects was to ensure:

• Robust identification of issues faced on the projects as part of the Procuring Authority team set up and stakeholder management, routine contract management and non-routine contract management.
  
  
• A global representation of sector and region-specific trends and issues faced during PPP contract management.
  
  
• The prevalence and timing of issues faced during contract management.
  
  
• An insight into key events that had a notable impact on the project and underlying causes of these events.
  
  
• An insight into overall performance of the projects.

To meet the objectives of the data collection, 250 projects had to be randomly selected from an overall database containing all relevant PPP projects. This process adopted for the data collection exercise is set out below.

1.   PPPs were downloaded from online databases

The online sources used were the World Bank Private Participation in Infrastructure Database [1], as well as proprietary databases from Inframation News [2], IJGlobal [3] and InfraPPP [4].  

2.   The projects were cleansed and compiled into a Master Database

a)   All PPPs were combined into a single database (the ‘Master Database’).

b)   Each project was assigned a unique ID. This was done by removing duplicates, removing additional sections of the same PPP (e.g. project extensions), and removing secondary market financial transactions associated with the project. Where clear, projects from the databases which didn’t fit our definition of a PPP were also removed. This cleansing was necessary to ensure the sample wasn’t skewed when the Master Database was used to select the random sample, as each project had an equal chance of forming part of the study.

c)   Projects with a transaction value of less than USD 20 million were removed. These were agreed to be too small for the purpose of the study.

d)   The projects were sorted by region, sector and financial close period. The breakdown categories were as follows:

Region: UK and Europe, North America, Latin America and the Caribbean, East Asia (including China), South and Central Asia, South-East Asia and the Pacific, Australia and New Zealand, Middle East and North Africa, and Sub-Saharan Africa)

Sector: Transport (including rail, roads, airports and ports), energy (including renewable and non-renewable generation, and distribution), water (including supply and distribution) and waste (including solid waste, waste to energy and waste water treatment)

Financial close (be period): Period 1 (January 2005 to September 2007), Period 2 (October 2007 to June 2010), Period 3 (July 2010 to March 2013) and Period 4 (April 2013 to December 2015)

The breakdowns of the overall population of relevant PPPs by Region, Sector, and Financial Close period are displayed in Appendix A (Data analysis).

3.   A random sample of 250 PPPs was chosen, as representative of the Master Database

a)   It was decided to select a sample of 275 projects. This allowed for some leeway (i.e. 10% surplus) when collecting the data in case it proved difficult to gather information on some projects.

b)   The percentage breakdown in the Master Database for each region, sector and financial close period was recorded.

c)   A script in Excel was created which carried out the following:

i. Randomly selected 275 projects from the Master Database, creating a ‘Target Database’.

ii. The percentage breakdown for that Target Database was calculated for each region, sector and financial close period.

iii. The differences between the Master Database percentage breakdowns and the Target Database percentage breakdowns were calculated and the differences added together. For example, the Master Database had 17.2% of eligible projects in Europe and 3.4% of eligible projects in North America. A sample with 15% and 3% of projects in those regions respectively would have a difference of 2.2 + 0.4 + … = 2.6 + … for these characteristics.

iv. The process was repeated 10,000 times, and the Target Database with the smallest difference to the Master Database was selected.

Once this process was completed (including removing the additional 25 projects, as described below), the result comprised the ‘Sample Database’. The composition of the Sample Database is shown below.

4.   A template for data collection developed

Once the Sample Database was selected, a data collection template was developed to capture the topics of interest for the data collection exercise. The template was structured to collect information on key features related to the Procuring Authority team set-up, main challenges associated with routine contract management (e.g. claims, changes, performance monitoring) and major, non-routine contract management events faced on the project in the Sample Database. The data collection template was also designed to capture basic project information, such as location, value, key parties, basic financing structure, revenue source, etc.

The key sections of the data collection templates are set out below.

Project ID
As well as being used to identify each project, this section included information such as the location, key parties, value, revenue source, etc.

Major Events
This section investigated events such as insolvency, termination and force majeure. The prevalence of these events informed the development of the reference tool itself.

Renegotiation
The prevalence and impact of renegotiations is a key theme of the reference tool. For this reason, this section of the template went into more detail than simply whether the renegotiation occurred, and included questions including why it occurred and what the outcome was.

Disputes
The prevalence, management and outcome of disputes is also an important factor in project success, and so this section also went into further detail. Additionally, the process for handling disputes is referred to across the literature as a particular success factor.

Contract Management
How to set up the Procuring Authority contract management team is another key theme of the reference tool. Many documents in the literature referred to examples of leading practice in contract management, such as the use of a contract management manual.

Project Success
The ultimate aim of the reference tool is to provide guidance that helps to improve the delivery of PPP projects. It is therefore important to investigate elements of project success, including cost and time overruns.

Ownership and Financing
Changes in ownership and other secondary market transactions can give additional information.

5.   Desktop research was conducted using publicly available information on projects in the Sample Database

The desktop research was conducted by Turner & Townsend offices around the world, using publicly available sources as well as local knowledge. The research was conducted to populate the data collection templates with as much information as possible.

6.   Stakeholder interviews were conducted where possible to complete the data collection

As much of the data was difficult to gather from publicly available sources, stakeholders on the projects were contacted and interviewed. The stakeholders came from either the Procuring Authority, the Project Company or in certain instances the central PPP unit or lenders and Procuring Authority’s advisors.

7.   The Sample Database was reduced in size to 250 by removing 25 projects

25 projects for which it had proven difficult to gather data were selected to be discarded from the Sample Database. This was done carefully to ensure the proportion of projects in each region and sector did not change after these projects were removed.

[1] https://ppi.worldbank.org/

[2] https://inframationgroup.com/

[3] https://ijglobal.com/

[4] http://www.infrapppworld.com/

The objective for developing Case Studies on 25 projects was to demonstrate real life examples of lessons learned from PPP contract management as well as leading practices. As Case Studies were seeking a deeper insight into how the main challenges and key events faced on the project were managed, the approach adopted focused on conducting face-to-face or phone structured interviews with key stakeholders from the Procuring Authority and Project Company. In some instances, the structured interviews were also been conducted with PPP units, lenders and Procuring Authority advisors.

The majority of the 25 Case Studies are shared in Appendix B (Case Studies). Because of the sensitivities of ongoing projects (e.g. some may be experiencing disputes), not all Case Studies are currently available for publishing in full detail. For this reason, a selection of the Case Studies has been anonymised or omitted entirely. However, the fundamental lessons learned from all 25 Case Studies have been incorporated into the reference tool.

The task of developing the Case Studies was undertaken as detailed below.

1.   A selection of 25 projects were selected for the Case Studies

The selection process was dependant on two factors: the nature of the challenges identified in the data gathering exercise on the Sample Database, and the willingness of the stakeholders to participate in the research.

A list of projects of interest was compiled and assessed for possibility of gaining access to the relevant people in Procuring Authorities, Project Companies, and other key stakeholders. The initial list contained more than 25 projects in order to mitigate the risk of lack of willingness to participate, or lack of access to the relevant stakeholders.

Contacts within the Global Infrastructure Hub’s and Turner & Townsend’s global networks were identified and engaged to introduce the research and request input into the selected Case Studies. Some stakeholders responded by suggesting other projects, or were not willing to participate in the research, and the Case Study selection therefore evolved and changed throughout the research. It became clear that the greatest challenge was securing participation of the stakeholders. Most of the Case Studies were drafted and developed with the help of one or two stakeholders. Only a small number of Case Studies had the full support of the Procuring Authority, the Project Company, and the lenders.

2.   A questionnaire was created to be answered by the stakeholders

A questionnaire was drafted based on the challenges identified through the data gathering exercise on the Sample Database. The questionnaire, directed to the stakeholders, was structured to facilitate the understanding of challenges found during the data collection, the existence of other challenges not initially identified and to go into detail on how those challenges were managed in practice.

Different versions of the questionnaire were tailored to stakeholders to draw comparisons between their experiences on the same project. The questionnaire was translated where the stakeholders preferred communication in their native language.

The questionnaire was used both as a guide through the interview and a preparation document for the stakeholders. Once an interview was scheduled with a stakeholder, a copy of the questionnaire was sent to them to have the information ready by the time of the interview. This proactive approach significantly reduced the time needed for each interview as the interviewees were well prepared.

3.   Structured interviews were carried out with the Procuring Authority, the Project Company and other stakeholders, as appropriate

Interviewing both parties to the PPP contract was important, as it provided the balance and range of views required for the Case Studies. Where possible, lenders and other key stakeholders were also interviewed.

The interviews were conducted mostly through conference calls. Where possible, some interviews were conducted through physical meetings in either Turner & Townsend’s regional offices or the relevant stakeholder’s own facilities. It was evident that face to face interaction was essential in some regions, such as South America, India, and China. The majority of interviews were conducted in English, with interviews also conducted in Mandarin, Arabic, Spanish and Portuguese as appropriate.

In some cases, where needed, the interviews were undertaken in two or more sittings due to the level of information the stakeholders were providing, or to accommodate the availability of the participants.

The interviews for the Case Studies provided real life experience on many of the challenges commonly faced on PPP projects. However, some events are rare by nature and are not faced commonly on PPP projects. Consequently, regardless of the large sample, real life examples of some challenges were not found (e.g. lender step-in).

To help address the lack of examples and make the guidance in the reference tool as broadly applicable as possible, a range of interviews were conducted with experts in the industry. When engaged, the majority of the industry experts around the world responded to our request with enthusiasm. The interviewed experts included lawyers, project managers, insurers and lenders, experienced dispute and dispute resolution consultants, as well as financial advisors. A substantial legal review of a draft version of the reference tool was also undertaken to pick up nuances in different legal jurisdictions. The legal review included input from legal practitioners with experience in Europe (the UK, a common law country, and civil law countries), Asia, Latin America, Africa and Australia.

Once a substantial number of Case Studies had been completed and a draft version of the reference tool had been developed, three regional workshops were held, to share the preliminary findings and to gain further insight from PPP practitioners into their challenges during PPP contract management. The first workshop was in Bogota, Colombia, the second in Singapore, and the third in Rome, Italy, with attendees from regional Procuring Authorities, private sector organisations as well as multilateral development banks. Feedback and additional lessons learned from the workshops were then incorporated into the final reference tool.

Limitations to the data collection process

There were a number of limitations to the data collection process, which will have affected the data analytics results. These were mitigated to the greatest extent possible, however could not be removed entirely. The limitations to the data collection process are set out below.

• Desktop research – publicly available information. The availability of information varied significantly between regions. Procuring Authorities in some regions make project information easily available online, including the PPP contracts themselves, while multi-national bodies such as development banks publish information on the projects they are involved with. However, this is often not the case, especially in developing markets such as India or China, and more so in earlier years. This challenge was to some extent overcome by contacting project stakeholders, but this was also not always possible. The results for each metric are therefore based only on the projects for which firm data was available.
  
  
• Desktop research – accuracy of information. In some cases, the accuracy of the information collected was clear. For example, because it came from the original PPP contract. In other cases it was less clear, with news articles referring to events on a project but limited further information available. This was addressed as far as possible by cross-checking data against other sources as well as with project stakeholders.
  
  
• Availability of project stakeholders. It was not always possible to contact stakeholders on the projects for which data was being collected. Where it was possible to make contact with stakeholders, not all of them were willing to participate in the research. In several instances, key stakeholders with the relevant knowledge had left the project, which added to the challenges of data collection.
  
  
• Commercial sensitivity. Certain information was commercially sensitive, in particular relating to cost overruns and variations, as well as contract renegotiation and disputes. For this reason, even where access to project stakeholders was available, they were often unwilling to share certain information. In addition, a number of stakeholders (in particular on the Project Company side) were not willing to engage in any interviews due to confidentiality restrictions in the relevant PPP contract. In general, lenders were not willing to communicate any project specific information due to the confidentiality restrictions in their agreements with the Project Company.
  
  
• Transparency and availability of data presents a challenge in some regions. In regions where no reliable project data could be collected and particular challenges were faced when identifying the relevant stakeholders and engaging with them, the overall data collection was reduced to a smaller number of projects than identified in the Sample Database.

Statistical analysis

Given the number of projects in the Sample Database, it is possible to draw conclusions to a certain level of confidence. Confidence intervals are used to understand how well a sample represents the whole population, in this case how well the Sample Database of 250 projects represents the Master Database. A confidence level describes how likely it is that a characteristic falls within a particular confidence interval. For example, a 95% confidence level indicates that the characteristic being investigated will fall within that interval in 19 out of 20 instances. For some specific examples:

• With the entire Sample Database of 250 projects, the confidence interval will be approximately ±5% at a confidence level of 95% (calculation not shown here). For example, this means that if 30% of projects are found to have experienced renegotiation in the Sample Database, it can be said with 95% certainty that between 25% and 35% experienced renegotiation for the entire population of 3736 PPP projects.
  
  
• For smaller subsectors of the Sample Database, this confidence interval increases. There are 126 transport projects in the sample, which increases the interval to approximately ±8%. For example, if 30% of transport projects are found to have experienced renegotiation, it can be said with 95% certainty that between 22% and 38% experienced renegotiation out of the entire population of 3736 PPP projects. The size of this interval shows that it is not reasonable to draw strong conclusions from small sample sizes.