The reference tool has been designed to be based on real-life experiece of PPP contract management from around the world, based on both extensive data collection as well as interviews with key stakeholders on the selected case studies. It should be used in combination with the selected case studies and data analytics from global desktop research.
In order to capture data and lessons learned representing the full array of PPP projects across different sectors and regions, the following high-level approach was adopted for developing the reference tool:
The sections below provide greater detail on how the reference tool was developed using the steps summarised above.
The study has some limitations due to the approach adopted and challenges faced during data collection and stakeholder interviews. The limitations are detailed in Section E (Limitations).
The objective of the data collection on 250 projects was to ensure:
To meet the objectives of the data collection, 250 projects had to be randomly selected from an overall database containing all relevant PPP projects. This process adopted for the data collection exercise is set out below.
a) All PPPs were combined into a single database (the ‘Master Database’).
b) Each project was assigned a unique ID. This was done by removing duplicates, removing additional sections of the same PPP (e.g. project extensions), and removing secondary market financial transactions associated with the project. Where clear, projects from the databases which didn’t fit our definition of a PPP were also removed. This cleansing was necessary to ensure the sample wasn’t skewed when the Master Database was used to select the random sample, as each project had an equal chance of forming part of the study.
c) Projects with a transaction value of less than USD 20 million were removed. These were agreed to be too small for the purpose of the study.
d) The projects were sorted by region, sector and financial close period. The breakdown categories were as follows:
Region: UK and Europe, North America, Latin America and the Caribbean, East Asia (including China), South and Central Asia, South-East Asia and the Pacific, Australia and New Zealand, Middle East and North Africa, and Sub-Saharan Africa)
Sector: Transport (including rail, roads, airports and ports), energy (including renewable and non-renewable generation, and distribution), water (including supply and distribution) and waste (including solid waste, waste to energy and waste water treatment)
Financial close (be period): Period 1 (January 2005 to September 2007), Period 2 (October 2007 to June 2010), Period 3 (July 2010 to March 2013) and Period 4 (April 2013 to December 2015)
The breakdowns of the overall population of relevant PPPs by Region, Sector, and Financial Close period are displayed in Appendix A (Data analysis).
a) It was decided to select a sample of 275 projects. This allowed for some leeway (i.e. 10% surplus) when collecting the data in case it proved difficult to gather information on some projects.
b) The percentage breakdown in the Master Database for each region, sector and financial close period was recorded.
c) A script in Excel was created which carried out the following:
i. Randomly selected 275 projects from the Master Database, creating a ‘Target Database’.
ii. The percentage breakdown for that Target Database was calculated for each region, sector and financial close period.
iii. The differences between the Master Database percentage breakdowns and the Target Database percentage breakdowns were calculated and the differences added together. For example, the Master Database had 17.2% of eligible projects in Europe and 3.4% of eligible projects in North America. A sample with 15% and 3% of projects in those regions respectively would have a difference of 2.2 + 0.4 + … = 2.6 + … for these characteristics.
iv. The process was repeated 10,000 times, and the Target Database with the smallest difference to the Master Database was selected.
Once this process was completed (including removing the additional 25 projects, as described below), the result comprised the ‘Sample Database’. The composition of the Sample Database is shown below.
Once the Sample Database was selected, a data collection template was developed to capture the topics of interest for the data collection exercise. The template was structured to collect information on key features related to the Procuring Authority team set-up, main challenges associated with routine contract management (e.g. claims, changes, performance monitoring) and major, non-routine contract management events faced on the project in the Sample Database. The data collection template was also designed to capture basic project information, such as location, value, key parties, basic financing structure, revenue source, etc.
The key sections of the data collection templates are set out below.
As well as being used to identify each project, this section included information such as the location, key parties, value, revenue source, etc.
This section investigated events such as insolvency, termination and force majeure. The prevalence of these events informed the development of the reference tool itself.
The prevalence and impact of renegotiations is a key theme of the reference tool. For this reason, this section of the template went into more detail than simply whether the renegotiation occurred, and included questions including why it occurred and what the outcome was.
The prevalence, management and outcome of disputes is also an important factor in project success, and so this section also went into further detail. Additionally, the process for handling disputes is referred to across the literature as a particular success factor.
How to set up the Procuring Authority contract management team is another key theme of the reference tool. Many documents in the literature referred to examples of leading practice in contract management, such as the use of a contract management manual.
The ultimate aim of the reference tool is to provide guidance that helps to improve the delivery of PPP projects. It is therefore important to investigate elements of project success, including cost and time overruns.
Ownership and Financing
Changes in ownership and other secondary market transactions can give additional information.
The desktop research was conducted by Turner & Townsend offices around the world, using publicly available sources as well as local knowledge. The research was conducted to populate the data collection templates with as much information as possible.
As much of the data was difficult to gather from publicly available sources, stakeholders on the projects were contacted and interviewed. The stakeholders came from either the Procuring Authority, the Project Company or in certain instances the central PPP unit or lenders and Procuring Authority’s advisors.
25 projects for which it had proven difficult to gather data were selected to be discarded from the Sample Database. This was done carefully to ensure the proportion of projects in each region and sector did not change after these projects were removed.
The objective for developing Case Studies on 25 projects was to demonstrate real life examples of lessons learned from PPP contract management as well as leading practices. As Case Studies were seeking a deeper insight into how the main challenges and key events faced on the project were managed, the approach adopted focused on conducting face-to-face or phone structured interviews with key stakeholders from the Procuring Authority and Project Company. In some instances, the structured interviews were also been conducted with PPP units, lenders and Procuring Authority advisors.
The majority of the 25 Case Studies are shared in Appendix B (Case Studies). Because of the sensitivities of ongoing projects (e.g. some may be experiencing disputes), not all Case Studies are currently available for publishing in full detail. For this reason, a selection of the Case Studies has been anonymised or omitted entirely. However, the fundamental lessons learned from all 25 Case Studies have been incorporated into the reference tool.
The task of developing the Case Studies was undertaken as detailed below.
The selection process was dependant on two factors: the nature of the challenges identified in the data gathering exercise on the Sample Database, and the willingness of the stakeholders to participate in the research.
A list of projects of interest was compiled and assessed for possibility of gaining access to the relevant people in Procuring Authorities, Project Companies, and other key stakeholders. The initial list contained more than 25 projects in order to mitigate the risk of lack of willingness to participate, or lack of access to the relevant stakeholders.
Contacts within the Global Infrastructure Hub’s and Turner & Townsend’s global networks were identified and engaged to introduce the research and request input into the selected Case Studies. Some stakeholders responded by suggesting other projects, or were not willing to participate in the research, and the Case Study selection therefore evolved and changed throughout the research. It became clear that the greatest challenge was securing participation of the stakeholders. Most of the Case Studies were drafted and developed with the help of one or two stakeholders. Only a small number of Case Studies had the full support of the Procuring Authority, the Project Company, and the lenders.
A questionnaire was drafted based on the challenges identified through the data gathering exercise on the Sample Database. The questionnaire, directed to the stakeholders, was structured to facilitate the understanding of challenges found during the data collection, the existence of other challenges not initially identified and to go into detail on how those challenges were managed in practice.
Different versions of the questionnaire were tailored to stakeholders to draw comparisons between their experiences on the same project. The questionnaire was translated where the stakeholders preferred communication in their native language.
The questionnaire was used both as a guide through the interview and a preparation document for the stakeholders. Once an interview was scheduled with a stakeholder, a copy of the questionnaire was sent to them to have the information ready by the time of the interview. This proactive approach significantly reduced the time needed for each interview as the interviewees were well prepared.
Interviewing both parties to the PPP contract was important, as it provided the balance and range of views required for the Case Studies. Where possible, lenders and other key stakeholders were also interviewed.
The interviews were conducted mostly through conference calls. Where possible, some interviews were conducted through physical meetings in either Turner & Townsend’s regional offices or the relevant stakeholder’s own facilities. It was evident that face to face interaction was essential in some regions, such as South America, India, and China. The majority of interviews were conducted in English, with interviews also conducted in Mandarin, Arabic, Spanish and Portuguese as appropriate.
In some cases, where needed, the interviews were undertaken in two or more sittings due to the level of information the stakeholders were providing, or to accommodate the availability of the participants.
The interviews for the Case Studies provided real life experience on many of the challenges commonly faced on PPP projects. However, some events are rare by nature and are not faced commonly on PPP projects. Consequently, regardless of the large sample, real life examples of some challenges were not found (e.g. lender step-in).
To help address the lack of examples and make the guidance in the reference tool as broadly applicable as possible, a range of interviews were conducted with experts in the industry. When engaged, the majority of the industry experts around the world responded to our request with enthusiasm. The interviewed experts included lawyers, project managers, insurers and lenders, experienced dispute and dispute resolution consultants, as well as financial advisors. A substantial legal review of a draft version of the reference tool was also undertaken to pick up nuances in different legal jurisdictions. The legal review included input from legal practitioners with experience in Europe (the UK, a common law country, and civil law countries), Asia, Latin America, Africa and Australia.
Once a substantial number of Case Studies had been completed and a draft version of the reference tool had been developed, three regional workshops were held, to share the preliminary findings and to gain further insight from PPP practitioners into their challenges during PPP contract management. The first workshop was in Bogota, Colombia, the second in Singapore, and the third in Rome, Italy, with attendees from regional Procuring Authorities, private sector organisations as well as multilateral development banks. Feedback and additional lessons learned from the workshops were then incorporated into the final reference tool.
There were a number of limitations to the data collection process, which will have affected the data analytics results. These were mitigated to the greatest extent possible, however could not be removed entirely. The limitations to the data collection process are set out below.
Given the number of projects in the Sample Database, it is possible to draw conclusions to a certain level of confidence. Confidence intervals are used to understand how well a sample represents the whole population, in this case how well the Sample Database of 250 projects represents the Master Database. A confidence level describes how likely it is that a characteristic falls within a particular confidence interval. For example, a 95% confidence level indicates that the characteristic being investigated will fall within that interval in 19 out of 20 instances. For some specific examples: