The Project: A Massive Control Systems Upgrade with No Downtime

E Tech Group engineers were contracted to update and replace obsolete hardware in a pickle plant. This project was time sensitive in two ways: the upgrade is being performed to mitigate impending risk of equipment failure, and the new building automation system must be implemented without halting production.

We knew that if the one centralized PLC5 that controlled five systems responsible for 50+% of pickle production failed, they would have to shut the entire plant down to identify and fix the issue, losing valuable time and revenue during peak production periods. 

Our automation team knew that in order to upgrade and separate the equipment, the five systems controlled by the central PLC would need to be migrated to either ControlLogix or CompactLogix and all communication transitioned to ethernet.  

The Challenges: Time Constraints & High Stakes

Because there were a number of hurdles to take into consideration when beginning this automation system upgrade, it took careful planning to ensure this project could be completed without disrupting production or extending the deadline. First and foremost, we had to untangle and map out the three decades of piecemeal, undocumented control upgrades that were currently keeping the facility operational. 

We also had to keep in mind that this facility uses only fresh produce, and at certain times ran at close to 110% production capacity. This meant systems upgrades had to be completed over the weekend and often tested early Monday before they began production, requiring our team to pivot quickly and mitigate any potential hiccups that might occur. 

Lastly, the plant itself was located in Michigan, and some of the panels in need of upgrade were outside. If operations were shut down in winter, it could potentially damage the pumps responsible for moving hundreds of gallons of water and pickles from frigid conditions into the plant. We had a small window between cold conditions and production ramp-up to execute, test and ensure operation of the modernized systems. 

At this point, clear communication and setting realistic expectations with the client was now critical and of the utmost importance. While we could tailor a solution to their control systems integration that didn’t hurt their bottom line, it would take careful coordination and cooperation to pull it off with minimal disruption to production.

The Solution: Taking Things Apart to Put Them Back Together

Our relationship and open communication with the onsite operations team was invaluable, as it made it much easier to directly identify those five systems in need of audit, and to document the existing setup. This would allow our team to lay out the best solution for such an involved automation and integration install required over a very truncated timetable. 

APPLIED SOLUTION:  

System 1: 

The decision was made to tackle the conversion of the largest system first. It consisted of a 17-slot PLC5 rack and 12 hardwired drives, with a mix of 120V digital inputs/outputs and analog inputs. All of this was converted to ControlLogix and PowerFlex 525s over an ethernet connection. NAT modules were then used to connect the PLCs to the network. This would allow the new ethernet control system to remain isolated while still enabling the controller to communicate with the network. 

While the hardware installation was simple, programming proved to be a bit more challenging. Since the original PLC5 had to stay in place, code for the newly-independent system had to be disabled in the PLC5 once the new controller came online. Numerous messages between the old and new PLCs had to be mapped and tested, including some with other controllers. All the IO was then remapped and as much of the process functionality as possible was tested and confirmed operational.  

Systems 2 & 3: 

While systems 2 and 3 were smaller, they were no less complicated. Due to spatial constraints, CompactLogix controllers were used instead of ControlLogix. Communication with the original PLC5 still had to be set up and tested, while messaging between the PLC5 and each of the new controllers had to be configured. 

Control was then disabled in the original PLC so the new ones could be tested. We were then able to remove the disabled logic in the original PLC5 for system 1 as it had been running for several weeks with no issue and established confidence in the new controller. 

Systems 4 & 5: 

The conversion for systems 4 and 5 were similar to system 1.  PLC5 IO racks utilized hard-wired drives, and both were converted to ControlLogix and PowerFlex 525 drives over ethernet.  As in system 1, NAT modules were used to keep the systems on the plant network but remain isolated.  The same cutover approach was repeated as with the first three systems where logic was disabled in the original PLC as systems 4 and 5 came online. 

Messaging between the original PLC5 and the new controllers was configured, along with communication to several other controllers. Disabled logic in the original controller for systems 2 and 3 was removed at this point.  We were then able to remotely remove disabled logic for systems 4 and 5 when it was determined safe to do so. 

The Results: This Control System Integration was Kind of a Big DILL 

As with any large installation to be completed quickly, the need for troubleshooting issues can increase. Because we upgraded a decades-old obsolete system to a current high-speed system over a weekend, there was more testing and calibrating necessary than if it were a simpler, incremental upgrade done over time. 

Learning the client wanted to walk in Monday morning and resume production at 100%, our team had to manage those expectations by communicating the need to ensure functionality at all stages of production, and that they would need to begin at 50% and increase gradually to 100%. This would allow for an easier shut down to solve any potential bugs or hiccups that may arise before reaching full production.

Our team’s clear communication and ability to manage the client’s expectations, coupled with the expert work of our engineers, has opened the door to a multi-year project with the potential for future work. Except now we know that next time, they won’t be in nearly as much of a pickle.