The Project: Address the Possibility of Obsolescence Ahead of Time
A large food and beverage client reached out to us, as they were concerned about the state of both their ethernet and non-ethernet connected Industrial Control System Networks at all thirty-six facilities. Understanding there was an inherent risk of obsolescence as the state of their network, physical asset health and security was of an indeterminable nature, they wanted to have an IT/OT Vulnerability Assessment done to identify any potential threats, what was nearing end of life, and areas where they could push for optimization. They needed an in-depth assessment and a targeted plan for remediation.
Having an existing working relationship with this client, E Tech Group was immediately one of the control system integrators considered for this project, and was eventually contacted for a bid. If awarded this contract, E Tech Group would be responsible for walking all thirty-six of their facilities to perform the same type of assessment and provide a plan for remediation.
The Process: Assess, Diagnose & Remediate Control System Concerns
After reviewing proposals, the client selected E Tech Group and another control system integrator to provide the initial assessment and develop their individual plans for remediation based on their findings. We knew our IT/OT assessment and subsequent project design needed to be competitive and cutting-edge while remaining attainable.
This assessment would cover both a network driven and physical on-site inventory and evaluation process. Once completed, the findings would be rolled up into reporting that would be consumed at the corporate level down to the operations team.
For the network portion of the assessment, the client’s IT provided our team documentation around program files for the PLCs, HMIs, Drives, and were granted network vizios to view network topology. They were then given limited network access, and using different protocols were able to show where their PLCs, HMIs and Drives were connected on their IT network. This allowed them to dig a bit deeper into their topology rather than just where their IT switches were.
Lastly, the E Tech Group team worked to compile an asset inventory to help the client understand what equipment was currently living on their network and where in its lifecycle it was. This was a two-prong process that required close collaboration between our automation experts and the client’s operations personnel to ensure that all bases were covered.
Network Assets to be inventoried and assessed:
- Network switches (Not including Plant Network IT devices, For example Cisco)
- Routers
- Firewalls
- Servers
- PCs
- NATs
- HMIs
- PLCs
- VFDs
- IO Racks
- Any other Devices on a communication network
With it now time to walk the facility floor, our E Tech Group team members would be joined by facility engineers, technicians and other staff to assist in locating the critical components that needed to be identified, documented and inventoried. Working closely with their on-site team allowed this process to move more efficiently and ensure nothing was overlooked.
Our automation and IT engineers utilized the existing controls network logical infrastructure drawings to work swiftly, documenting and inventorying each panel and its contents, making sure to note if they were reflected accurately in the provided drawings.
Physical Assets to be inventoried and assessed:
- Physical Network Infrastructure, for both ethernet and non-ethernet.
- Physical Topology and Active Device Inventory
- Switch Selection (Not to include IT Infrastructure Cisco Switches)
- Router Selection (Not to include IT Infrastructure Cisco Routers)
- Communication configurations for both ethernet and non-ethernet (Configuration, Lost Packets, Collisions, etc)
- Environmental Conditions
- Enclosures
- Cable Selection
- Cable Management
- Conduit and Routing
- Cable Labeling
- Power Redundancy
- Grounding
Once all digital and physical assets were documented, the E Tech Group team worked diligently for two weeks to take this massive aggregate of information and transform it into a usable database for the client to digest.
The Solution: Tailor Remediation Plans to Each Facility’s Needs
Upon initial review, the client was painfully unaware so many components were near end of life or patchworked together to keep them running. This targeted view into their IT and OT helped them prioritize their next steps that would allow them to start working more proactively to improve and maintain their network and asset health.
Because the E Tech Group assessment best aligned with the client’s standards, we were awarded the contract. Using this initial work as a benchmark, the templated assessment would now be scaled across all thirty-six sites, allowing for tailored plans of remediation.
The Results: Avoided Complications of Large-Scale Obsolescence
Long term, the client intends to set themselves up to implement a 3D model to monitor their network health and asset lifecycle. This system would allow them to utilize predictive maintenance for planned downtime that has the least impact on production, improving efficiency and uptime as upgrades are performed. Understanding the client’s goals, E Tech Group outlined their first plan for remediation:
Site 1 Phases of Remediation:
- Consolidate network servers across all facilities
- Identify isolated assets not living on the network
- Work to migrate non-network connected assets
- Implement connected shop floor cloud-based monitoring to track asset efficiency and network health
The E Tech Group team will continue to work through the remaining sites, utilizing the templated assessment to help build customized plans for remediation. Since these sites were acquired individually and over time, there is a wide range of equipment manufacturers and assets in various stages of their lifecycle.
Each facility will require a different type and extent of control system upgrade, which is why the assessment design was key in this process. Being able to re-use our assessment strategy across all sites will help in quickly identifying vulnerabilities and being able to move into the remediation process as quickly as possible.
Begin Your Journey to Industry 4.0
Download our Beginner’s Guide to IT/OT Assessments to gain an understanding of what an IT/OT Assessment is, its benefits, and how to get started today.
