Challenge
Fires are a devastating event for any business. It can result in the loss of equipment, inventory, and even the facility itself. The aftermath of a fire can be challenging for any company as it seeks to rebuild and return to business as usual. This case study highlights how E Tech Group collaborated with an architectural engineering company to support an international almond supplier after a fire destroyed their roasting line. The project team worked together on a tight schedule to replace and upgrade the almond supplier’s roasting system, leveraging PLC, HMI, networking, and control panel expertise.
Solution
The almond supplier needed to replace their out of commission roasters with a new, modernized system. On the old line, the operators had to walk between two floors to start and monitor production. The customer needed a solution that simplified the almond roasting process while providing more visibility from beginning to end. The first step was forming a strong communication strategy between the customer and the project team. This consisted of daily progress meetings about the project’s developments and potential challenges such as equipment procurement, control panel design, and PLC and HMI programing to control the roasting systems. One of the biggest challenges was the “design-build” method to execute the control panel build. This meant the control panel design had to be open to changes while it was still being built in the shop. The project team relied on their strong collaboration skills to successfully merge these steps to support the tight project timeline.
E Tech Group’s principal engineers leveraged their expertise to develop the control modules in CompactLogix PLCs and PanelView Plus HMI for the customer’s various devices as well as configured Moxa managed switches for multi-level network topology. The engineering team also worked closely onsite with electricians to change the control panel design as needed, which included designing auxiliary panels and ordering time sensitive parts for safety and air handling operations. The onsite team also focused on configuring network controls as well as fan VFD motor controllers and burners for the new roasting line. The project manager maintained timely communications with the project team and the customer, securing engineering resources to help with the PLC and HMI code design and managed the team’s progress toward the finish line.
Results
With everything in place, the project team was able to successfully execute the project and trained the customer’s operators and maintenance staff on their new and improved roasting system. The project took about four months to complete from the design phase to site acceptance testing. Rather than walking between two floors to start and monitor the roasting process, the customer’s operators could now start the entire line with the push of a button on the HMI and have production visibility from start to finish. Also, the added controls to the plant network, such as Rockwell Automation’s FactoryTalk AssetCenter, supported the management and protection of the customer’s automation-related asset information. With the project complete, the customer was able to resume production with a more intelligent, modernized roasting line.
Building a PlantPAx 5.0 System for a Pilot Cultivated Meat Plant
A cultivated meat production facility needed to scale up from lab production so they could refine their process of growing cells at bench-top scale to a pilot scale. E Tech Group helped them to replicate their lab-driven results in an increased-volume setting, setting them up for large-scale production and distribution.
The Project: Innovate a Facility-Wide Control System for a Pilot Alternative Protein Plant
Alternative proteins, including lab-grown meats, are the new horizon of ethical, sustainable meat production. Specifically, cultivated meats are just emerging, with the first expected to be available within a few years. Cultivated meat production plants are a new frontier as far as food and beverage automation goes.
A client leading the way in this vein of alternative protein was scaling up from lab production, where they were able to refine their process of growing cells at benchtop scale to a pilot scale. Proving it was possible to replicate their lab driven results in an increased volume setting, if successful, would help move into large-scale production and distribution once necessary approvals came through.
Previously having worked with E Tech Group as an automation partner that helped guide and coach them through thoughtfully identifying their needs for their first automation project, they were confident in our ability to help plan, design, implement, test and support an end-to-end, turnkey automation solution for this new facility.
The Challenges: A New Type of Food Production Requires New Automation Tech
Because this was a new installation of cutting-edge technology, there was no lack of presented challenges on this project. We were all in uncharted waters.
Understanding that nothing in this industry had yet been designed and executed on this scale, the entire process would remain fluid. Change and adaptation would be a constant theme throughout the planning and implementation.
Knowing what was being produced was designed for human consumption, ensuring the developed solution exceeded industry and federal guidelines and standards was non-negotiable. E Tech Group needed to ensure this facility’s automation system took all of the unknown out of the cultivated meat production process.
The Solution: Utilize Rockwell’s PlantPAx 5.0 to Support the Automated Systems
The design of this project needed to be just as creative as the client’s endeavors in this new field of meat production. Communication and flexibility was key in pulling off this leading-edge, facility-wide automation system integration.
Process Design
The plant being a pilot and the organization essentially a start-up, the process engineering environment was naturally ever-evolving and dynamic. The E Tech Group team kept up with the constant changing strategies, provided technical guidance in streamlining the design to a point where it could be automated, and clearly communicated changes to the process design as they went live. This helped maintain open lines of communication with stake holders and create realistic expectations for delivery.
Process Outline
- Media Prep – Media Transfer, CIP of the Media Prep suite, SIP of the Media Prep Suite
- Clean In Place (CIP) – CIP of the Bioreactors and the transfer lines
- Steam in Place (SIP) – SIP of the Bioreactors and the transfer lines
- Inoculation
- Media Filtration
- Cell Growth
- Seeding/Circulation
As part of the design effort, Pin Charts were developed and maintained. These defined the valve alignment and various schemes for Equipment Modules corresponding to each phase step, covering a matrix of over 100 phase steps and over 300 devices, over 200 valves, and a gamut of control modules.
The team also worked to design customized control loops (involving Cascade methodology) for pH, Dissolved Oxygen (DO) and Temperature Control (both Vessel and Jacket) of the Bioreactors. These complex algorithms use shared equipment and phase-specific control strategies.
The process design also involved deciphering functionality of specialized equipment such as Mix Proof Valves – the Equipment Module developed for this valve comprises of 9 schemes in its operation.
Electrical Design
- Panel Layout
- Network Layout
Automation Products
Our process automation team had a daunting task at hand – to synthesize the enormous design and get the system ready for production in less than 6 months. The automation effort included the following tasks:
- PLC Program Design: Rockwell Automation PlantPAx 5.0, ControlLogix L85 EP, Studio 5000 v33
- Mix of 4.x and embedded 5.0 process objects
- Function Block and Ladder Logic used to program Control and Equipment Modules
- Structured Text programming for Phases
- Phase Parameter definitions
- Hold, Abort and Stop Logic
- Operator Prompts and Reporting Logic
- Linking/Messaging with Parallel Phases
- Equipment Module Programming
- Scheme Propagation
- State Model
- Permissive
- Interlocks
- Valve Supervisor Equipment Modules
- Online Configuration Tool – development and maintenance for bulk import of IO
- Handshaking with External Skids (Sanimatic CIP, Alfa Laval Centrifuge)
- Message Instruction programming
- Produce, Consume Tags
- Batch Programming: FactoryTalk Batch
- Area Model Configuration – S88 standardized
- Recipe development – Unit Operations and Unit Procedures
- Handshaking with External Skids (Sanimatic CIP, Alfa Laval Centrifuge)
DCS Design & Implementation
A new DCS System was implemented in a virtualized environment using the FactoryTalk View SE, PlantPAx system. The system is scalable and sustainable, as the infrastructure supports future growth for the pilot plant. The HMS, Historian and Win-911 systems were integrated and deployed.
Components
- SCADA Server: hosts the HMI server, Data server, Alarm and Event server
- ThinManager and Terminal Server: host thin client sessions for HMI operations on the plant floor
- Batch Server: ISA -88 standard Batch engine for Recipe development
- Network Directory Server
- SQL Server
- Win-911: Remote alarm notification to pre-configured user groups via FactoryTalk Alarm and Events
- FactoryTalk Historian SE Server
- Engineering Workstation – hosts the Rockwell Development suite
Features
- Graphics are developed based on the ISA 101 standard.
- Alarms are based on ISA 18.2 standard.
- Node security implemented
- Scope for external Skid Integration
- Custom graphics developed for external skid integration
- Sanimatic CIP Skid
- Centrifuge
Onsite Support & Equipment Testing
In addition to the unparalleled in-house development effort, E Tech Group has provided over 600 hours of hands-on onsite support to the client – starting from instrument checks to Recipe Testing. From beginning to end, our team kept up with dynamic changes happening during the Wet Runs, adjusting the design accordingly and re-integrating the changes into the active system.
E Tech Group’s ability to adapt, adjust, and overcome was critical in the final testing and rollout of this solution. Vital processes we monitored, tested and facilitated included:
- Loop Checks
- Loop Tuning
- Equipment Module Testing
- Phase Testing
- Recipe Testing
- Troubleshooting
- Operator Training
Documentation
Documentation is key in an effective facility-wide automation system, as it allows the users to analyze and identify everything that happens within the system. This has implications for standards, quality control and troubleshooting.
E Tech Group developed the Functional Design Specifications for the entire process – starting from ground zero and then updating all the specs based on the As Built system. Throughout the entirety of the project, our team consolidated information from various vendors – Standalone Skids, construction management and piping and instrumentation vendors – trying our very best to not let any details slip through the cracks.
We also helped develop SOPs to meet the client’s production and compliance requirements. This gives the owner the ability to better support their own system, since it is well-documented. They can find information easily and without having to bring in a control system expert. This reduces downtime, which is inevitable in a development process for cutting-edge products like cultivated meat.
The Results: A Positive Learning Experience for All Involved
After a successful completion of this EPIC project, E Tech Group is fueled with stronger skillsets and hungry for more. We are proud to be part of this journey with the client and extremely thankful to them for trusting and partnering with us in designing and creating the automation solution.
E Tech Group is a Rockwell Automation partner, supporting manufacturers across industries with a unique brand of innovative, adaptable, robust control systems. We design and implement building-wide automation systems that are secure, user-friendly, and sustainable long-term, so you can focus on maximizing quality, capacity and the ability to scale.
Helping a Food & Beverage Client Avoid Obsolescence Concerns
A large food and beverage company recognized there was the potential large-scale obsolescence of their control systems. They reached out to E Tech Group to have our automation team assess and provide plans for remediation and optimization.
The Project: Address the Possibility of Obsolescence Ahead of Time
A large food and beverage client reached out to us, as they were concerned about the state of both their ethernet and non-ethernet connected Industrial Control System Networks at all thirty-six facilities. Understanding there was an inherent risk of obsolescence as the state of their network, physical asset health and security was of an indeterminable nature, they wanted to have an IT/OT Vulnerability Assessment done to identify any potential threats, what was nearing end of life, and areas where they could push for optimization. They needed an in-depth assessment and a targeted plan for remediation.
Having an existing working relationship with this client, E Tech Group was immediately one of the control system integrators considered for this project, and was eventually contacted for a bid. If awarded this contract, E Tech Group would be responsible for walking all thirty-six of their facilities to perform the same type of assessment and provide a plan for remediation.
The Process: Assess, Diagnose & Remediate Control System Concerns
After reviewing proposals, the client selected E Tech Group and another control system integrator to provide the initial assessment and develop their individual plans for remediation based on their findings. We knew our IT/OT assessment and subsequent project design needed to be competitive and cutting-edge while remaining attainable.
This assessment would cover both a network driven and physical on-site inventory and evaluation process. Once completed, the findings would be rolled up into reporting that would be consumed at the corporate level down to the operations team.
For the network portion of the assessment, the client’s IT provided our team documentation around program files for the PLCs, HMIs, Drives, and were granted network vizios to view network topology. They were then given limited network access, and using different protocols were able to show where their PLCs, HMIs and Drives were connected on their IT network. This allowed them to dig a bit deeper into their topology rather than just where their IT switches were.
Lastly, the E Tech Group team worked to compile an asset inventory to help the client understand what equipment was currently living on their network and where in its lifecycle it was. This was a two-prong process that required close collaboration between our automation experts and the client’s operations personnel to ensure that all bases were covered.
Network Assets to be inventoried and assessed:
- Network switches (Not including Plant Network IT devices, For example Cisco)
- Routers
- Firewalls
- Servers
- PCs
- NATs
- HMIs
- PLCs
- VFDs
- IO Racks
- Any other Devices on a communication network
With it now time to walk the facility floor, our E Tech Group team members would be joined by facility engineers, technicians and other staff to assist in locating the critical components that needed to be identified, documented and inventoried. Working closely with their on-site team allowed this process to move more efficiently and ensure nothing was overlooked.
Our automation and IT engineers utilized the existing controls network logical infrastructure drawings to work swiftly, documenting and inventorying each panel and its contents, making sure to note if they were reflected accurately in the provided drawings.
Physical Assets to be inventoried and assessed:
- Physical Network Infrastructure, for both ethernet and non-ethernet.
- Physical Topology and Active Device Inventory
- Switch Selection (Not to include IT Infrastructure Cisco Switches)
- Router Selection (Not to include IT Infrastructure Cisco Routers)
- Communication configurations for both ethernet and non-ethernet (Configuration, Lost Packets, Collisions, etc)
- Environmental Conditions
- Enclosures
- Cable Selection
- Cable Management
- Conduit and Routing
- Cable Labeling
- Power Redundancy
- Grounding
Once all digital and physical assets were documented, the E Tech Group team worked diligently for two weeks to take this massive aggregate of information and transform it into a usable database for the client to digest.
The Solution: Tailor Remediation Plans to Each Facility’s Needs
Upon initial review, the client was painfully unaware so many components were near end of life or patchworked together to keep them running. This targeted view into their IT and OT helped them prioritize their next steps that would allow them to start working more proactively to improve and maintain their network and asset health.
Because the E Tech Group assessment best aligned with the client’s standards, we were awarded the contract. Using this initial work as a benchmark, the templated assessment would now be scaled across all thirty-six sites, allowing for tailored plans of remediation.
The Results: Avoided Complications of Large-Scale Obsolescence
Long term, the client intends to set themselves up to implement a 3D model to monitor their network health and asset lifecycle. This system would allow them to utilize predictive maintenance for planned downtime that has the least impact on production, improving efficiency and uptime as upgrades are performed. Understanding the client’s goals, E Tech Group outlined their first plan for remediation:
Site 1 Phases of Remediation:
- Consolidate network servers across all facilities
- Identify isolated assets not living on the network
- Work to migrate non-network connected assets
- Implement connected shop floor cloud-based monitoring to track asset efficiency and network health
The E Tech Group team will continue to work through the remaining sites, utilizing the templated assessment to help build customized plans for remediation. Since these sites were acquired individually and over time, there is a wide range of equipment manufacturers and assets in various stages of their lifecycle.
Each facility will require a different type and extent of control system upgrade, which is why the assessment design was key in this process. Being able to re-use our assessment strategy across all sites will help in quickly identifying vulnerabilities and being able to move into the remediation process as quickly as possible.
Begin Your Journey to Industry 4.0
Download our Beginner’s Guide to IT/OT Assessments to gain an understanding of what an IT/OT Assessment is, its benefits, and how to get started today.
Challenge
Outdated controls can pose many operational challenges. Companies that operate with outdated controls often experience frequent downtime, expensive or obsolete hardware replacement parts, or few personnel with knowledge to perform maintenance on legacy systems. These challenges were faced by a long-running California winery who needed to modernize the controls for their red fermentation process. The winery needed to address how a system upgrade would impact their operations and engaged E Tech Group to create a PLC and SCADA upgrade solution.
Solution
The first step was to create a well-documented project plan. E Tech Group analyzed the existing system design and detailed existing hardware components, software configurations, and network connections. This step was crucial to have an accurate picture of their current control environment to mitigate production impacts during the implementation phase.
The project team upgraded hardware and reconfigured the customer’s existing Ignition SCADA software for seamless post-upgrade operations. The original red fermentation process was conformed with three areas: pump over area, continuous press area, and decanters area. Previously, each area had their own PLC but the upgraded system connected an Allen-Bradley ControlLogix PLC as the main controller and remote IO was added to control the three areas into one. The system was programmed with ladder logic to the customer’s existing Ignition SCADA software, and seven new PanelView Plus HMIs were also programmed with FactoryTalk View ME as the customer’s choice visualization software.
Results
The project team leveraged their PLC programming, Ignition SCADA software, and controls engineering expertise to resolve software compatibility issues, bugs in the programming code, and electrical connection challenges. The team supported the project to resolve these challenges and performed extensive testing. With the PLC and SCADA upgrade complete, the customer was able to improve the uptime of its red fermentation process and eliminated the key challenges of legacy system maintenance.