Best-Value Process Implementation at the City of Peoria : Five Years of Research Testing

The City of Peoria is Arizona’s ninth largest city. It covers nearly 178 square miles and is home to over 153,000 residents. The projected 10 year growth of the City is estimated to be over 204,000, which has prompted a significant expansion of the City’s municipal services and facilities. In an attempt to efficiently meet the demands of the projected growth, the City of Peoria partnered with the Performance Based Studies Research Group out of Arizona State University in 2004. The objective of the partnership was to test and implement a best value structure within the City’s construction program, specifically as applicable to capital projects. This paper presents a five year summary of the test implementation results, the evolution of the best value structure within the city, project performance, service expansion to include non-construction projects, and lessons learned from the research. The research effort has included $385,691,802 in total projects awarded and implemented under the best value structure, with documented performance increases in cost metrics, schedule metrics, and customer satisfaction.


Introduction
The City of Peoria is Arizona's ninth largest City.It covers nearly 178 square miles and is home to over 153,000 residents.The projected 10 year growth of the City is estimated to be over 204,000 which will prompt the significant expansion of the City's municipal services and facilities.The City of Peoria has taken steps to enhance development through revitalization plans to provide for further development of the central downtown area to support new and existing businesses as well as economic growth (City of Peoria, 2008).
As with many public organizations, the City of Peoria was required by law to award all competed projects to the lowest priced vendors.The results of these price-based projects fell far below the City's expectations (Sullivan et al, 2006).In May 2006, approximately 75 percent of the price-based projects were not completed on time, 75 percent had cost generated change orders, and the overall customer satisfaction was only 20 percent (Sullivan et al, 2006).Several projects had also gone to litigation over quality issues.The City was struggling to increase performance in a price-based environment.
In 2000, the State of Arizona passed H.B. 2340, which allowed agencies to use alternative project delivery methods to select construction firms (including design-build and constructionmanager-at-risk).This allowed agencies to select vendors based on performance factors, rather than just price alone.The City of Peoria began to consider the value of their existing procurement system and investigating further options.
In 2004, the City of Peoria partnered with the Performance Based Studies Research Group (PBSRG) at Arizona State University (ASU) in order to implement a best-value procurement process.Through the system, the City hoped to identify and select higher performing vendors to increase the performance of their outsourced services.

The Performance Information Procurement System (PIPS)
The Performance Information Procurement System (PIPS) is a best-value selection, project delivery, and contract management tool that can be used to purchase any type of product or service.Since 1994, the PBSRG has researched and tested the PIPS process on hundreds of projects, resulting in extremely high customer satisfaction (98%) (Kashiwagi 2009).
The PIPS process provides clients with tools to assist them in making an informed decision based on performance information (and not solely on price or marketing information).The process not only assists in selecting a high performing vendor, but also has a mechanism to document and manage the vendor/service during the project.This tool has proven to be just as valuable as the selection process itself (Sullivan et. al 2007).
The PIPS process has seven major steps that are used to procure and manage high performing vendors, which are outlined below: 1. Vendors and their critical individuals are required to collect performance ratings on projects that they have completed in the past.The surveys (from their past clients) are averaged together to obtain a "past performance information" score.These scores are used later in the selection process.2. The second step requires that the contractors submit a two-page Risk Assessment and Value Added (RAVA) Plan on the project being procured.This document provides the vendors with the opportunity to differentiate themselves from their competitors in terms of value and expertise, justifying to the client that they are not a commodity.A key aspect of this step is the "blind" evaluation of the RAVA Plans by a committee.During the evaluation process, all company names are withheld in order to eliminate bias and rate solely on performance.3. The third step consists of identifying the top three firms and interviewing the critical individuals that would actually be involved in the project.This allows each option to be realistically compared, potentially disqualifying a vendor if the critical individuals are changed.A standard set of questions are asked to each firm and their responses were documented.4. The fourth step in the process includes using decision making models to analyze and compare all of the data and evaluation scores.5.The fifth step involves identifying the best value option and providing time for the potential vendor to carefully pre-plan the entire project to ensure that they have not missed anything.Both the owner and vendor must be satisfied with the plan before the contract is signed.This period of time is called the pre-award phase.
6.After award is made, the vendor is then required to submit a weekly risk report that outlines any issues that develop during a project with a probability of affecting time, money, or quality (owner satisfaction).These reports are standardized and are due every week (Friday) until the project is complete.These reports provide the client with a documented overview of the development of all project deviations.7. Once the project is complete, the last step involves evaluating the performance of the vendor on the project.This rating accounts for 50 percent of their "past performance information" score (which places them at risk to perform on this project or jeopardize their competiveness on future best-value projects).
The PIPS process has been implemented by many different clients in both the private and public sectors (by Local, State, and Federal agencies).Due to the rules and regulations of each client, the PIPS process is easily adapted and tailored to meet the constraints of each user.Clients have used the system on construction projects ranging from under $10,000 to over $100 Million, as well as non-construction service contracts that have exceeded over one billion dollars.Under the State of Arizona procurement codes, the City of Peoria was allowed to use the PIPS process under the Design-Build (DB) and Construction Manager at Risk (CMAR) delivery processes.However, State law prohibited the City from asking for or considering project costs.This meant that the City could only review performance information, without any consideration for price or fee.The impact of this law will be discussed later in this paper.A summary of the City's proposal process is shown in Table 1.

Best Value Research Results
The City of Peoria has implemented the PIPS best-value process on 55 projects, over $389 Million in contract value.The City initially implemented the process on construction projects, but after documenting the performance results, they expanded the use to Architectural and Engineering services, and service-type projects (such as emergency radio services, landscaping, and software services).Table 2 illustrates the City's distribution of the best-value projects.Due to the large size of the projects, many of the projects are still in progress or on-going services.Ten projects have been completed and the performance of these projects are documented in Table 3.The table also provides a baseline performance measurement of 38 traditional price-based projects that were also completed at the City (Zenko, 2009).
The overall change order rate was reduced by 99% by going from the traditional price-based award to the best-value award.The overall project delay rate was reduced by 77%.The overall customer satisfaction rates were increased by 395%.These numbers are impressive due to the large size of the projects (over $193 Million).

Case Studies
The first PIPS Best-Value Project at the City of Peoria was the Rio Vista Project.This project was procured under the design-build delivery method, and consisted of designing and building a recreational park with seven baseball fields and a 51,736SF recreational facility.Seven teams submitted proposals for the project.The City identified the best value selection as the firm who performed the highest based on the owner's selection criteria.This project was awarded for $20 Million but was completed for $19.4 Million (the contractor was able to return money back to the City).The project met the City's expectation and was awarded the Construction Owners of America Top Gold Award for Project Leadership in 2007.
The Fire Station #7 project was an excellent case study of the impact of using the PIPS process for both the design and construction services.The selected design team and the construction firm scored the highest in interviews and RAVA Plans.The vendors were able to successfully partner to make the $3 Million project a success.Similar to the Rio Vista Project, the contractor was able to complete the project under the awarded fee, and was able to return money back to the City.This project won several awards including the Gold Medal Design Excellence Award from Fire Chief Magazine in 2007, and the Design Excellence Merit Award from Fire Rescue Magazine in 2007.

Summary of Selection Factors
Due to the Arizona Procurement code, the City of Peoria was prohibited from requesting project costs/fees from vendors on best-value projects.The selection process was based strictly on performance measurements.To assist the vendors in improving their competitiveness, the project information available was analyzed to determine the factors that contributed to the bestvalue vendor or team being selected.
The City shortlisted vendors prior to interviewing to minimize the resources required from both the City and from the non-competitive vendors.This was done on the majority of projects where the City received more than three proposals.An analysis of 15 shortlisted projects indicates that the vendors that were shortlisted had 18% higher RAVA Plan ratings and also had 6% higher PPI scores (See Table 4).An analysis of 28 final selection models indicates that the awarded best-valued vendor had a 15% higher interview rating than the non-awarded vendors.A comparison of the RAVA Plan scores, and PPI scores indicates that there is very little differential between the firms.From this, it can be concluded that while the RAVA Plan and PPI are important factors in being shortlisted, the interview phase is the most important in determining the best-value award.

Impact of Not Reviewing Cost Proposals
Due to the Arizona Procurement code (ARS, 2000), the City of Peoria was prohibited from asking for project costs/fees from vendors on best-value projects.The lack of this critical information made it difficult to justify that the City was truly receiving the best value for their money.This is illustrated by the comparison of the originally estimated budget of the project, to the final awarded cost of each project.Out of the 10 completed projects, the original estimated budget was $140 Million.The awarded cost of these projects was $193 Million, which is 38% higher than the estimated budget.Once again, it is unclear whether the original budget was inaccurate or not (since the City was not allowed to ask for costs from any of the competitors).However, it is clear that there was a significant increase in the performance of these projects with regards to cost increases, project delays, and overall customer satisfaction.It is also important to note that on 50% of the completed projects, the vendors actually returned money back to the City (since they were able to complete the project under their originally estimated fee).

Important Research Developments
Throughout the five-year research program, the City has tested and developed alternate procedures to steps within the PIPS best-value program.The key developments are discussed below: 1. RAVA Plan Template (2005) When the City first began implementing the PIPS process, there was no standard risk assessment and value added (RAVA) Plan template or format.Vendors were allowed to create and format their own RAVA Plans.This resulted in Plans with colors, pictures, alternate font sizes, and alternate page margins.After procuring several projects, the City realized that they could begin to identify which RAVA Plans belonged to which vendors (even though the plans had no names in them).The City requested that a RAVA Plan template be created so that all of the RAVA Plans looked visually the same (to minimize evaluator bias).

RAVA Plan Page Limit (2005)
Due to the large size of the City's projects, the City requested that vendors be allowed to submit up to 5 pages on their RAVA Plans (instead of the 2-page limit).This modification was made and tested on 7 projects.The results were compared to 4 projects that only allowed 2 pages.The research showed that by increasing the page limit from 2pages to 5-pages, no additional value was generated (Table 6).The additional pages resulted in additional marketing material which had no value to the individual projects.Since then, the City has returned to the 2-page RAVA Plan limit on all of their projects.

Past Performance Information Modification (2009)
The collection and use of Past Performance Information has undergone significant changes.When the City first began the process, Vendors were required to send surveys to their past clients, and the past clients were required to return the surveys directly back to the City of Peoria.Over 7,000 surveys had been collected, which required a significant amount of resources from the City to monitor and track.This also made it difficult for the vendors to know which past clients had returned a survey (in order to increase their response rate), and made it difficult for the vendors to know their actual scores.In 2009, the City modified the PPI process to allow vendors the opportunity to collect their own surveys.This made the process easier for the vendors, and minimized the amount of resources required from the City to manage the process.

Lessons Learned
Throughout the City's five-year best-value program research, several key lessons learned have been documented.
Continuous Education of Core Group.The difference between this organization and many that have failed to implement the Best Value PIPS approach is the continual proactive education of the core group.The procurement department has received numerous education and training sessions, attended yearly best-value conferences, and also enrolled in several of the best-value Master's Degree courses at ASU.
Go Slow and Document the Results.One of the greatest challenges with implementing the PIPS best-value process is the speed of implementation.The PBSRG recommends testing pilot projects that are small, simple, and can be completed within a short time frame (less than one year).This allows the client time to learn the basic steps of the process and document whether the process can work for the user or not.The City of Peoria was unable to run their pilot projects in this manner.Their capital improvement projects were all large in scope and took over 1-year to complete.This made it difficult to provide dominant information that the program was improving the City's processes.Although the program has been a significant success, the City did run into some issues that jeopardized the future success of the program (discussed in next section).
Avoid Highly Political Areas.Due to the success of the construction implementations, the City expanded the use of the PIPS program to Architectural and Engineering services.The first few A/E projects were successful; however, a small group of engineers began challenging the process.These vendors claimed that the process was unfair since they were not winning any projects.Through political pressure, the vendors were able to force the procurement department to stop using the PIPS process (on A/E services).Without dominant information of documented successes, the City was unable to continue the use of A/E services.In 2008, the procurement department was able to provide the documentation performance results to the newly elected City Mayor.The information provided enough dominant information to reinstate the use of PIPS on A/E services.The lesson learned from this instance, is that in order to effectively use time and resources, highly political areas should be approached with caution.They should only be attempted after great consideration has been made and careful documentation is available to provide proper support.
Interview of Key Personnel.In the past, the City traditionally had an interview period, but it was treated as more of a presentation period, where the vendor would provide a presentation of past projects they had completed.This resulted in a marketing presentation, in many cases presented by marketing personnel.When the City implemented the PIPS best-value process, a major differential was that the vendors key personnel were actually interviewed.No marketing or upper management personnel were permitted to sit in the interviews.The City realized the importance of conducting interviews with key personnel after several firms were represented by individuals stating that they had, "no idea what the project they were interviewing for consisted of," or, "why their firm had sent them," to the interview.The City took this concept one step further by interviewing additional personnel besides the Site Superintendent and Project Manager.The City interviewed estimators and pre-construction personnel, which provided valuable information on CMAR and DB projects.
Importance of the Weekly Risk Reporting System.When the City of Peoria first began using the PIPS process, there was no standardized reporting system.In 2007, a standardized Weekly Risk Reporting System was created to assist clients in monitoring and tracking the performance of their projects.This system allowed the vendor to document all risks that developed throughout a project that impacted project duration and project cost.This information is critical to have, as it provides a standardized method of documentation that identifies who or what generated the risk and what actions were taken to minimize, or exacerbate, the risk.
The advantages of this tool are clear when considering previous Table 3.While the table shows that the completed projects have an overall change order rate of 0.1% and an overall project delay rate of 8.2%, it is not clear who or what caused the schedule and budget deviations.
Without the weekly risk report, the cause of the changes would be unclear (owner, designer, or contractor generated), as well as the type of risk (simple scope change, contractor mistake, design error, etc).It allows all parties to be held accountable for their actions or lack of action.The Weekly Risk Reporting System template provides the client with a simple report to monitor and track their projects.This report has since been included in all recently award projects at the City.

Conclusion
The implementation and testing of a best value procurement, project delivery, and contract management structure at the City of Peoria has yielded many positive and useful results: 1.The research has shown, that despite price not been a factor in the procurement process, a high performing firm can still be selected, though the value of that firm is unknowable.
2. Despite not fully and correctly implementing all aspects of the best value system, specifically, failure to completely incorporate the pre-planning and weekly risk reporting processes, project performance has still increased.
3. Inefficiency is a business process and structural issue, not a technical issue.This has been shown via the successful application of the best value structure to construction (under three different delivery systems), A/E services, and several non-construction services all being improved over prior performance.The only modification was a move from a technical focus for risk mitigation, to a process and outsourcing focus for risk mitigation.
4. Education is critical to sustained change and successful adoption of a new process.
The research effort is still underway at the City of Peoria with more tests, new methods, and different types of projects be run under the best value process.