We’re your source for automation news. Keep up with the latest industry updates and E Tech employee spotlights, as well as tips and guidance from our manufacturing experts.
Dealing with Obsolescence Can Become a Bit of a Pickle….
A pickle plant faced obsolete control system hardware but couldn’t afford downtime. E Tech Group’s automation engineers were able to upgrade their decades’ old obsolete system to a current high-speed system in 2 days with minimal downtime.
The Project: A Massive Control Systems Upgrade with No Downtime
E Tech Group engineers were contracted to update and replace obsolete hardware in a pickle plant. This project was time sensitive in two ways: the upgrade is being performed to mitigate impending risk of equipment failure, and the new building automation system must be implemented without halting production.
We knew that if the one centralized PLC5 that controlled five systems responsible for 50+% of pickle production failed, they would have to shut the entire plant down to identify and fix the issue, losing valuable time and revenue during peak production periods.
Our automation team knew that in order to upgrade and separate the equipment, the five systems controlled by the central PLC would need to be migrated to either ControlLogix or CompactLogix and all communication transitioned to ethernet.
The Challenges: Time Constraints & High Stakes
Before the systems were separated
Because there were a number of hurdles to take into consideration when beginning this automation system upgrade, it took careful planning to ensure this project could be completed without disrupting production or extending the deadline. First and foremost, we had to untangle and map out the three decades of piecemeal, undocumented control upgrades that were currently keeping the facility operational.
We also had to keep in mind that this facility uses only fresh produce, and at certain times ran at close to 110% production capacity. This meant systems upgrades had to be completed over the weekend and often tested early Monday before they began production, requiring our team to pivot quickly and mitigate any potential hiccups that might occur.
Lastly, the plant itself was located in Michigan, and some of the panels in need of upgrade were outside. If operations were shut down in winter, it could potentially damage the pumps responsible for moving hundreds of gallons of water and pickles from frigid conditions into the plant. We had a small window between cold conditions and production ramp-up to execute, test and ensure operation of the modernized systems.
At this point, clear communication and setting realistic expectations with the client was now critical and of the utmost importance. While we could tailor a solution to their control systems integration that didn’t hurt their bottom line, it would take careful coordination and cooperation to pull it off with minimal disruption to production.
The Solution: Taking Things Apart to Put Them Back Together
Our relationship and open communication with the onsite operations team was invaluable, as it made it much easier to directly identify those five systems in need of audit, and to document the existing setup. This would allow our team to lay out the best solution for such an involved automation and integration install required over a very truncated timetable.
APPLIED SOLUTION:
System 1:
The decision was made to tackle the conversion of the largest system first. It consisted of a 17-slot PLC5 rack and 12 hardwired drives, with a mix of 120V digital inputs/outputs and analog inputs. All of this was converted to ControlLogix and PowerFlex 525s over an ethernet connection. NAT modules were then used to connect the PLCs to the network. This would allow the new ethernet control system to remain isolated while still enabling the controller to communicate with the network.
While the hardware installation was simple, programming proved to be a bit more challenging. Since the original PLC5 had to stay in place, code for the newly-independent system had to be disabled in the PLC5 once the new controller came online. Numerous messages between the old and new PLCs had to be mapped and tested, including some with other controllers. All the IO was then remapped and as much of the process functionality as possible was tested and confirmed operational.
Systems 2 & 3:
While systems 2 and 3 were smaller, they were no less complicated. Due to spatial constraints, CompactLogix controllers were used instead of ControlLogix. Communication with the original PLC5 still had to be set up and tested, while messaging between the PLC5 and each of the new controllers had to be configured.
Control was then disabled in the original PLC so the new ones could be tested. We were then able to remove the disabled logic in the original PLC5 for system 1 as it had been running for several weeks with no issue and established confidence in the new controller.
After system separation and upgrade
Systems 4 & 5:
The conversion for systems 4 and 5 were similar to system 1. PLC5 IO racks utilized hard-wired drives, and both were converted to ControlLogix and PowerFlex 525 drives over ethernet. As in system 1, NAT modules were used to keep the systems on the plant network but remain isolated. The same cutover approach was repeated as with the first three systems where logic was disabled in the original PLC as systems 4 and 5 came online.
Messaging between the original PLC5 and the new controllers was configured, along with communication to several other controllers. Disabled logic in the original controller for systems 2 and 3 was removed at this point. We were then able to remotely remove disabled logic for systems 4 and 5 when it was determined safe to do so.
The Results: This Control System Integration was Kind of a Big DILL
As with any large installation to be completed quickly, the need for troubleshooting issues can increase. Because we upgraded a decades-old obsolete system to a current high-speed system over a weekend, there was more testing and calibrating necessary than if it were a simpler, incremental upgrade done over time.
Learning the client wanted to walk in Monday morning and resume production at 100%, our team had to manage those expectations by communicating the need to ensure functionality at all stages of production, and that they would need to begin at 50% and increase gradually to 100%. This would allow for an easier shut down to solve any potential bugs or hiccups that may arise before reaching full production.
Our team’s clear communication and ability to manage the client’s expectations, coupled with the expert work of our engineers, has opened the door to a multi-year project with the potential for future work. Except now we know that next time, they won’t be in nearly as much of a pickle.
This article originally appeared in Automation World and features Melissa Bruno, Group Engineering Manager at E Tech Group. Click here to read the full article. During any technology assessment to determine the need for an upgrade, a direct line of communication from integrators to decision makers and maintenance and engineering teams can help them better understand how to evaluate potential solutions. When you’re working in a facility that has equipment reaching end of life or components on the verge of obsolescence, it would be nice to simply wave a wand and have everything be instantly updated. Unfortunately, because multiple installations and upgrades have been done over decades, this process tends to be a bit more involved. Before you can address obsolescence, breakdown, and security issues, you need to have a full picture of where your facility stands. To do this, we perform what we call an IT/OT (operations technology) assessment. While companies like ours perform this work, the client’s presence and cooperation plays a crucial role. Here’s how to get the most out of your assessment. Engage and understand Compounded fixes over time make it difficult to identify the main issues or where to start resolving them. Having an open line of communication with the decision makers and maintenance team, we can help them better understand what we’re assessing and what it will take to correct it. The goal is to take a large amount of material and truncate it into digestible pieces that will allow all parties to evaluate potential solutions. In many cases, maintenance technicians or other employees may not understand what we’re doing. They see an engineer walking the floor and assume we’re there to complete a particular task. In reality we’re performing an in-depth assessment that will be used to give them control over the optimizations … Continued
From decreased productivity to system downtime, legacy equipment can pose many challenges to operational efficiency. This was the case for a leading consumer packaged goods company who was relying on PLC-5 hardware to control most of their facility. The plant recently experienced a major disruption during peak production because of a network switch failure, and it was later discovered that the disruption could have been minimized if the facility had been separated into multiple virtual local area networks (VLANs). To support the creation of VLANs, the customer’s existing PLC-5 control system would need to be upgraded to the ControlLogix system which could handle multiple independent Ethernet connections. E Tech Group was called to support the project given their previous assistance with consolidating the company’s DH+ and ControlNet network communications in support of their industrial automation applications.
Solution
E Tech Group performed a comprehensive investigation to identify all PLC types and determine each PLC’s connection to other systems. After completing their analysis, E Tech Group presented which PLCs were deemed the most critical to upgrade and designed a VLAN separation plan to reduce the likelihood of connection loss given the vast amount of device communications. The customer’s personnel were initially surprised by the number of PLCs that needed to be upgraded to fulfill the VLAN implementation, which was about 21 PLCs. For the best implementation process, it was determined to perform the PLC upgrades in three phases with the first phase switching out the most critical PLCs.
Six PLC-5s needed to be upgraded in the first phase. E Tech Group considered several factors to include local and other server racks, a labor-saving conversion kit, the maintenance of other communications protocols, and ways to thoroughly test the success of the conversion. The customer also requested the migration and testing be completed within 72 hours. With these considerations in mind, E Tech Group worked closely with the customer to determine the best conversion process within the short timeframe. This included the use of only replacing local racks, using DH+/RIO and ControlNet communication modules to support existing networks, complete I/O list development, and spot testing plans. A PLC-5 to ControlLogix conversion tool was utilized to migrate logic from the old hardware to the new hardware, and all drivers were updated in the Wonderware system to allow for proper communication to the new PLCs. DH+/RIO ControlLogix cards were also used to connect into existing DH+ networks and allow ControlLogix PLCs to talk to existing PLC-5 Remote I/O racks.
Results
The project was commissioned on a Friday of a holiday weekend and was completed that Monday. Committed to the success of the project, E Tech Group had two teams supporting the project throughout the entire weekend in 12-hour shifts. In a few short days, E Tech Group was able to have the six PLCs successfully migrated by the customer’s requested timeframe and resolved numerous pre-existing issues within the logic by the time the PLC’s remote monitoring was completed. Using the knowledge gained during the first phase, the next two phases were completed with much smaller teams and the customer had the ability to improve their operational efficiency with a control system platform.
Providing Expertise for a Control System Evaluation
E Tech Group, being a trusted system integrator for this biotech client, was asked to consult on the analysis of their current control system and evaluate potential new platforms. Our automation team helped them bring an aging control system up to current needs with future growth in mind.
The Project: Determine the Scope & Execution of a Control System Upgrade
An American biotechnology company was dealing with an aging control system. The vintage installation of GE iFix, iBatch and CompactLogix were approaching end-of-life, making an upgrade essential for multiple locations in California. Understanding intervention by an automation company was necessary.
The client grappled with key questions such as: What is the best control system platform to meet their business needs? Do they upgrade their current system or install a new system altogether? The company approached E Tech Group with this problem, interested in learning about the different platform options and working with a control system integrator to make the right choice based on their needs.
For a project of this magnitude, creating a strong partnership between client and integrator was crucial. Fortunately, the E Tech Group automation team included seasoned engineers who have been through large-scale control system platform selection and conversion projects before and supported similar decisions for other clients.
The Solution: Evaluate, Collaborate, Integrate
E Tech Group went through a process of user requirements definition and user group meetings, assessing the company’s site and global needs. E Tech Group control system engineers participated in workshops with all stakeholders within the company:
Automation
Environmental Health &Safety
Facilities/Maintenance
Instrumentation
IT
Production/Manufacturing
Process Engineering/Development
Procurement/Finance
Quality
System Integration Partners
Based on experience in providing countless custom controls upgrades, the E Tech Group team challenged the client to think beyond basic requirements from a technical and business perspective. Engaging the team together in workshops enabled each group to learn about how others in the company would be using the system.
At the conclusion of the process, the control platform vendors considered were evaluated against the newly-defined requirements. E Tech Group worked with all of the internal stakeholders at the company to evaluate the options and assess how the different platforms could best support the business’ current and future needs.
Using a scoring system and working with the decision analysis team at the biotechnology company, E Tech Group supported efforts to develop financial and statistical models to help the client determine the best option from a financial and risk perspective.
After deep analysis and evaluation, it was unanimously decided that the proposed DeltaV control system would best help reduce the risk that obsolescence and instability will disrupt the supply of critical patient therapies, help enable agile expansion of future capacity through a thoughtfully-designed automation system, provide flexibility and self-sufficiency for production needs, and ensure delivery of quality product in a compliant manner.
The client and all stakeholders, having been provided with control system software options and educated on the pros and cons of each, reached a unanimous decision on the future platform for the site, landing with DeltaV.
The Result: Long-Term Plans for Long-Term Growth
Understanding what a large financial undertaking this would be, the client, the platform vendor, and E Tech Group continue to collaborate on a long-term roadmap to support the upgrade/conversion of the site to the new platform.
