“If it ain't broke why fix it” is usually a good rule of thumb; however, when applied to automation systems and SCADA applications this can be a costly mistake. Typically, when a control system or SCADA is put into place it is during initial construction or installation and a group of engineers look at what is needed by the systems and specify appropriate controls to accomplish that function. Leaving that system in place long term without evaluating it periodically can cause a significant loss in either revenue from downtime or costly repairs that are not planned for. I will discuss why ignoring your control system could cost both time and money and how these issues contain increased risks and hidden costs that you do not want to be surprised by.
Risks are present when you leave your system on autopilot and do not get a professional evaluation:
Security Vulnerabilities
Having your system off the internet does not protect you from hacking or computer malware. Unless done correctly with proper updates it can leave your system more exposed than if you had your system connected to the internet.
Extended Downtime
A failure can result in a lengthy recovery time that can have a significant impact on your operations.
Age of the Current Operating System
It might not be possible to replace your current operating system or not advisable due to the same issues that caused the initial failure. Replacement of older operating systems may require you to make other changes due to compatibility.
Age of the PLC, Control, and Communication Components
Components are routinely retired by the manufacturer and direct replacements are not available or can only be purchased through 3rd party sites. If direct replacements are not available there could be extensive cost and time involved in getting a suitable replacement that will work, which will be even more expensive if you need to get the system running.
Diversity of Components
The risk here is that a failure and replacement in one component can require a domino effect of changes due to compatibility issues well beyond the scope of the component that failed.
Loss of Tribal Knowledge
Things operators know but aren't located in any documentation. As systems age and especially if the environment they are used in changes they typically end up having several aspects that require tribal knowledge to operate and/or maintain. What happens to this knowledge when you have employee turnover, or a tragic event can pose a lot of risk to your operation.
Hidden Costs Leaving Your Control System As Is
- Spare Parts. As your system ages maintenance costs can increase as components are retired and replacement parts get harder to find and more expensive.
- Hidden Personnel costs.
- Recruitment of qualified staff – it can be harder to find and more costly to hire people with expertise in older systems.
- Historical knowledge- Either because the system is older, or staff is unfamiliar to older standards, the loss of experienced personnel is common. Training newer staff on older systems is costly.
- Human error – Mistakes happen but having systems that require some tribal knowledge, extensive training or are built on interfaces that are completely foreign to interfaces seen outside the facility can lead to costly errors.
- Decreased Productivity – as personnel use the system there are typically many things that could be done by the system automatically, which would increase productivity and allow staff to be more innovative and strategic. A simple example would be calculating the additive needed for a liquid system. The operator reads the flow rate and volume and then opens a spreadsheet and calculates the amount of additive needed. Alternatively, the system could tell the operator that value.
- Facility additions or upgrades could require additional modifications or components to be compatible with your existing system leaving either a separate control system for addition or an increase in costs to integrate with the existing system.
Key Things to Consider When Evaluating Control Systems
Computers / Operating Systems
Any computers should be kept up to date with an operating system that is still supported with security updates. The older your system is the more likely the operating system is no longer supported. Replacing a computer in a control system in most cases is not as simple as replacing the computer and re-installing the “app.” It could require a major update, licensing changes, or replacement of the SCADA software, which will require a significant amount of time. It is better to make that change during a planned downtime than when your system is down for failure, and you need it back up urgently.
PLCs (Programmable Logic Controllers) and Control Components
PLCs and components should be evaluated to ensure you are using components still supported by the manufacturer. This helps ensure that any failures can be replaced with the known good component from the manufacturer and will not require additional work to make the replacement. Having components that are past the end of life is sometimes offset by keeping spares on hand, but these spares can often sit on the shelf for an extended period and end up not being serviceable when installed. Having spares on hand is advisable to make quick replacements to keep the facility operational but should not be the go-to method if those spares cannot be replaced by the manufacturer if the spares on hand prove to be faulty
Spreadsheets / Tribal Knowledge
Most control systems are designed by engineers and programmed by programmers. What is often not considered is how the operators and supervisors use them once installed. Here are some examples of things that are frequently missed or organically grow as the system is used.
- Sticky notes to pass information to other operators or keep track of information
- Spreadsheets where information is copied from SCADA to do calculations
- Reports generated by SCADA and then manually placed into Spreadsheet
- Reports handwritten by operators looking up key data in the SCADA system
Expansion Creep
Your facility might not have been built all at one time or by the same vendors, which leads to a mess of control systems with a wide range of controllers and a diversity of HMI’s and even SCADA systems. This can cause you increase in costs because each system is different therefore every operator needs to be trained separately on each system. Systems that are infrequently used can end up being forgotten about or require consultation with a manual to figure out how to accomplish something. Divergence in how systems operate can also cause mistakes when an operator does not realize that on one system, they need to do a particular function differently. Combining systems that have organically grown in your facility can lead to less training time for new employees, fewer mistakes that could lead to failures, and one stop to check system status and generate reports so that you can respond to potential issues before they occur.
You realize some of these issues, but you put these off for “next year” and then “next year” again. As your system ages it can get more difficult and more expensive to update or upgrade. And the number of people that are experts in your equipment diminishes as they retire, and new people are less familiar with older systems. Hiring people with experience and knowledge to work on your aging system can be more expensive due to rarity and lead to increased costs for system maintenance, emergency support, and updates. At some point every system will no longer be able to simply be updated and upgraded as a quick process but will need to be completely rebuilt from the ground up.
Do not wait till your system is broken to get it evaluated and fixed. Doing so can result in expenses at an unplanned time when you do not have the budget. Get a professional to evaluate your system, develop a plan to minimize your risks, and plan for your control system to work for you.
About the Author
Ralph is a Senior Controls Engineer working out of our Austin, TX office. He joined Hallam-ICS in Feb 2023 and prior to that worked in the Water and Wastewater, Oil and Gas, and Semiconductor industries.
Read My Hallam Story
About Hallam-ICS
Hallam-ICS is an engineering and automation company that designs MEP systems for facilities and plants, engineers control and automation solutions, and ensures safety and regulatory compliance through arc flash studies, commissioning, and validation. Our offices are located in Massachusetts, Connecticut, New York, Vermont, North Carolina, Texas and Florida and our projects take us world-wide.
