Arizona Public Service is the largest electric utility in Arizona, with 6,300 MW of generating capacity and serving 1M customers in over 11 counties. In 2016, they decided to enter the California ISO (CAISO) market, which allows them to sell power into the California market. One of their key assets was Sundance, a 420 MW unmanned peaker plant located 50 miles outside Phoenix. The entry into the CA energy market meant that starts tripled and run hours doubled almost immediately at the plant. Southern California has a lot of solar generation, and APS wanted to use Sundance for rapid ramping in the shoulder hours (morning and evening) to add energy to the grid in hours when solar isn’t generating.

They started looking for wireless Predictive Maintenance (PdM) system because the running hours were typically when no one was on site, which meant that traditional forms of PdM were not possible. Typically, a specialist would collect vibration and other condition data on equipment, but it had to be taken during operation, and it was difficult to get personnel out to the site.

“Reliability was foremost on our minds,” commented Don Lamontagne, Supervisor of Equipment Reliability Engineering. “We faced huge loss of potential revenue, as well as fines if we weren’t able to generate power when it’s needed.”

The corporate APS team started looking into options that would help get into a more proactive mode and provide visibility into asset performance. They had four main criteria:

1. Frequent, cost-effective monitoring
2. Security compliant
3. Environmentally robust
4. Scalable system

Cost-effective monitoring

When choosing a system, APS was not looking only at the price for the sensors, but also the installation and personnel costs. They wanted to minimize conduit and it was important that data be collected autonomously, and frequently enough to enable early detection of problems.

Security compliant

Utility plant control systems are subject to a set of security requirements (NERC Critical Infrastructure Protection), including electronic security. In order to stay compliant with these requirements, it was important for APS to find a system that would be used only by maintenance personnel so that control operators would not see alarms from the sensors. The IT department needed to carefully vet any vendor’s security protocols, but APS wanted minimum IT resources for implementation and ongoing maintenance.

Environmentally robust

APS operates in some of the harshest climates, with average daytime summer temperatures of between 103-107 degrees Fahrenheit. They needed equipment that could stand up to the heat and sand, and that was able to reliably transmit data wirelessly from remote areas.

Scalable system

APS wanted to start with a pilot, and if it proved successful, increase monitoring to other equipment at Sundance and other generation facilities.

Implementing wireless predictive maintenance

APS selected Petasense, a leading IIoT startup, because it met these four criteria and more. They liked that the machine learning algorithms automatically analyzed data coming from the sensors, saving time that was previously spent inspecting healthy machines. Given the remote nature of Sundance, the ability to access the reliability data from anywhere and allowing remote personnel to interact with the raw data and document maintenance tasks, was very important. APS liked both the Vibration Motes and Transmitters, which could bring data from multiple sensor types (pressure, temperature, current), providing for expansion to different types of assets.

The central APS team wanted to work collaboratively with the plant to help shift away from a reactive maintenance culture. They selected more than 100 rotating assets to monitor, including critical pumps, motors, and fans. In most cases, they were able to install the Motes in approximately 30 minutes, without shutting down the equipment.

Getting the network right

Setting up the wireless network was more complicated than it initially seemed, due to the distance and obstacles that attenuated the wireless signal. They started with a site survey to determine locations to maximize WiFi coverage. In order to cover the 500,000+ square foot plant, they decided to use two cellular gateways and 12 Mesh repeaters which extended the network coverage across the entire plant. They mounted fan-cooled, metal enclosures 45 feet in the air on the exhaust stacks for added signal strength, in most cases using existing power and minimal conduit. APS was able to achieve nearly 5-bar signal strength throughout the plant.

Results

Within the first week of operation, an alarm was generated on a Combined Water Pump, and an analyst verified the defect with a portable vibration analyzer. Both readings indicated the high speed shaft of the pump had rotating looseness. The pump had latter stages of journal bearing wear, evident by the presence of running speed harmonics. The rotating looseness was caused by excessive clearance between the rotating and stationary parts of the machine. An action was taken to change the bearings, and vibration levels were checked after replacement. Overall there was a 70% reduction in 1x running speed amplitude and a reduction of 1x harmonics.

One of the combined water pumps was showing persistent increases in the overall vibration levels, which continued to trend upward at a concerning rate. The plant was notified, and they reported that the mechanical seal had failed and was allowing water to enter the gearbox. A spike in surface temperature indicated that the asset was deteriorating faster than anticipated, and it was shut down in order to troubleshoot the underlying issue. They found that the discharge control valve had failed open and the mechanical seal was failing, allowing water to enter the gearbox. The plant was going into outage at the end of the October and wanted to know what they could do to run until the planned outage. By using an ultrasonic grease caddy to grease the bearings, they were able to reduce vibration and keep it in operation until the outage. An autopsy of the pump showed pitting on the impeller and damage to the high speed journal bearing.

Within the first six months, APS collected over 1.4 million readings and caught 13 defects with their wireless predictive maintenance system. In comparison, their old walkaround program would have only collected 5,000 readings, and most of these wouldn’t have corresponded with the key operating hours. The defects caught range from bearing issues, misalignment, soft foot, cavitation, and a broken cooling fan. They also had several occasions where they were able to instruct the plant to grease bearings in order to prevent failures.

Even at this accelerated rate of data collection, APS wasn’t overwhelmed by the amount of data because the machine learning system generates email alerts when there’s an issue with a particular piece of equipment and helps to prioritize maintenance tasks. They don’t have to analyze assets that are in normal condition.

Next steps

APS wants to continue to shift to a proactive maintenance approach, and they are planning to install Motes on remaining assets at Sundance, as well as install transmitters for hard-to-reach assets. They plan to integrate the Petasense data with PI Historian and CMMS for work order generation. The success of the program has made it easy to justify roll-out to two new sites that are critical facilities for their load.