By BECKY KISER
Hays Post
The SCADA (Supervisory Control and Data Acquisition) system which operates the Hays water plant and 25 ancillary sites (production wells, booster stations, and two water towers) was installed 28 years ago.
The outdated system is unreliable and requires a substantial upgrade in computer hardware and software, as well as connectivity, according to Jeff Crispin, director of water resources.
The water processing/distribution system in Hays is quality tested every two hours by employees.
Of the approximately 25 water asset sites, there are 12 with which the current SCADA system cannot communicate. Those sites are in the Big Creek and Smoky Hill River well fields.
The Smoky water well field is 12 miles south of Hays and the water treatment plant, located at 1000 Vine.
"If you're the operator and it's 11 p.m. and you're working by yourself and you need to shut off a well and you can't, that's a pretty big deal.
"That creates some issues with operating things remotely," Crispin said. "Sometimes we have to call somebody out."
The SCADA system is used to monitor and gather real-time data, and control equipment at the water treatment plant and 25 its ancillary sites.
"Once water is used in the towers, there are set points that basically kick on pumps, sending more water into our system, filling the water towers back up," Crispin explained.
"Water towers are basically how we maintain pressure in our system, and they're also used for extra water for firefighting."
If everything is working properly in the aging computer system, water department employees can "click" on an individual asset, such as an individual water well which can be turned on and off to bring water into the city. Operators can see how any asset it operating and running.
"They can look at pressures, gallons-per-minute, water levels, many things to be able to operate those things."
But not everything can be controlled because the system is old, Crispin told city commissioners Thursday night.
Two filter controls at the water plant still use a computers from 1993 with a 3.5 inch floppy disk drive and a CD disk drive.
"It reminds me of high school," Crispin said, who noted he graduated in 1993, the same year the current SCADA system was installed.
Some of the older Hays water wells use remote telemetry units (RTUs) using radio frequencies, which will be replaced by programmable logic controllers (PLCs), allowing monitoring and control of the SCADA system.
At 18 of 25 water asset sites, a fiber optic system from Hays vendor Nex-Tech terminates at those site, but has never been used. Crispin said Nex-Tech is willing to make the connections at no cost to the city.
"This is a complicated project and a lot of things need to be done," Crispin said.
"Fiber will ensure that we have reliable communication and connection to our assets in all weather conditions. No longer will be have to rely on antennas, radio frequencies and interference of any kind.
"This is the way we really should be communicating with our water system to make sure that we are completely reliable at all times."
The SCADA improvements request for proposal created by Crispin and other staff members specified the same hardware and software that was installed in 2019 in the upgraded wastewater treatment plant.
"This would provide us the ability to actually switch parts back and forth if we needed to."
The updated system would allow operators to make adjustments at the ' lick of a computer mouse and make our system much beter
The city's current SCADA vendor, R.E. Pedrotti, based in Mission, is the low bidder for the updated SCADA project at $801,225, including an alternate three-year warranty extension. It is below the $850,000 budget and would be funded from the water capital budget.
Although the water and wastewater systems are computer-operated, they are both separate from the city's IT (information technology) department. IT Department Director Chad Ruder has lent his expertise to this project, Crispin said.
The system is not connected to the city's internet system.
Commissioner Michael Berges asked about cyber security.
"Utilities are very attractive targets right now — and you know this -" Berges said, "and are being hacked into."
In early February, a computer hacker tried to poison the water supply of Oldsmar, Florida's water treatment system by changing a setting that would have drastically increased the amount of sodium hydroxide — known commonly as lye — in that city's water supply. It can be deadly if ingested in large amounts.
A city employee saw the change and quickly reversed it.
Crispin said that Florida system did not have proper security systems in place.
"This is another conversation I've had with Chad [Ruder] many times. We have three or four firewalls within our system to protect our city [water] system," Crispin told Berges.
Even though the current system is outdated, Crispin said the city is "very well protected with our firewalls. There is no connection between the city system.
"When we want to get on the internet, we have to make some adjustments, and what we're doing as part of this project is beefing up those firewalls.
"I feel very, very sure that right now we are as secure as we can be. Pedrotti would be able to 'dial in' through a firewall if we need to do any upgrades or security updates, and actually access that site."
According to Crispin, the managed option within the low bid includes a firewall for all 25 site assets.
"You're talking 25 firewalls. It's going to be very secure."
City commissioners will consider approving the low bid at their May 27 meeting.