News & Knowledge

The Project: Innovate a Facility-Wide Control System for a Pilot Alternative Protein Plant

Alternative proteins, including lab-grown meats, are the new horizon of ethical, sustainable meat production. Specifically, cultivated meats are just emerging, with the first expected to be available within a few years. Cultivated meat production plants are a new frontier as far as food and beverage automation goes.

A client leading the way in this vein of alternative protein was scaling up from lab production, where they were able to refine their process of growing cells at benchtop scale to a pilot scale. Proving it was possible to replicate their lab driven results in an increased volume setting, if successful, would help move into large-scale production and distribution once necessary approvals came through.  

Previously having worked with E Tech Group as an automation partner that helped guide and coach them through thoughtfully identifying their needs for their first automation project, they were confident in our ability to help plan, design, implement, test and support an end-to-end, turnkey automation solution for this new facility.  

The Challenges: A New Type of Food Production Requires New Automation Tech

Because this was a new installation of cutting-edge technology, there was no lack of presented challenges on this project. We were all in uncharted waters. 

Understanding that nothing in this industry had yet been designed and executed on this scale, the entire process would remain fluid. Change and adaptation would be a constant theme throughout the planning and implementation.  

Knowing what was being produced was designed for human consumption, ensuring the developed solution exceeded industry and federal guidelines and standards was non-negotiable. E Tech Group needed to ensure this facility’s automation system took all of the unknown out of the cultivated meat production process.  

The Solution: Utilize Rockwell’s PlantPAx 5.0 to Support the Automated Systems 

The design of this project needed to be just as creative as the client’s endeavors in this new field of meat production. Communication and flexibility was key in pulling off this leading-edge, facility-wide automation system integration.

Process Design

The plant being a pilot and the organization essentially a start-up, the process engineering environment was naturally ever-evolving and dynamic. The E Tech Group team kept up with the constant changing strategies, provided technical guidance in streamlining the design to a point where it could be automated, and clearly communicated changes to the process design as they went live. This helped maintain open lines of communication with stake holders and create realistic expectations for delivery.   

Process Outline 

• Media Prep – Media Transfer, CIP of the Media Prep suite, SIP of the Media Prep Suite  
• Clean In Place (CIP) – CIP of the Bioreactors and the transfer lines 
• Steam in Place (SIP) – SIP of the Bioreactors and the transfer lines  

•  Inoculation  
• Media Filtration  
• Cell Growth  
• Seeding/Circulation  

As part of the design effort, Pin Charts were developed and maintained. These defined the valve alignment and various schemes for Equipment Modules corresponding to each phase step, covering a matrix of over 100 phase steps and over 300 devices, over 200 valves, and a gamut of control modules.  

The team also worked to design customized control loops (involving Cascade methodology) for pH, Dissolved Oxygen (DO) and Temperature Control (both Vessel and Jacket) of the Bioreactors. These complex algorithms use shared equipment and phase-specific control strategies.   

The process design also involved deciphering functionality of specialized equipment such as Mix Proof Valves – the Equipment Module developed for this valve comprises of 9 schemes in its operation.  

Electrical Design  

• Panel Layout  
• Network Layout  

Automation Products

Our process automation team had a daunting task at hand – to synthesize the enormous design and get the system ready for production in less than 6 months. The automation effort included the following tasks:  

• PLC Program Design: Rockwell Automation PlantPAx 5.0, ControlLogix L85 EP, Studio 5000 v33  
  • Mix of 4.x and embedded 5.0 process objects
  • Function Block and Ladder Logic used to program Control and Equipment Modules   
  • Structured Text programming for Phases  
    • Phase Parameter definitions  
    • Hold, Abort and Stop Logic  
    • Operator Prompts and Reporting Logic  
    • Linking/Messaging with Parallel Phases 
  • Equipment Module Programming  
    • Scheme Propagation  
    • State Model  
    • Permissive  
    • Interlocks   

• Valve Supervisor Equipment Modules  
• Online Configuration Tool – development and maintenance for bulk import of IO  
  • Handshaking with External Skids (Sanimatic CIP, Alfa Laval Centrifuge)  
    • Message Instruction programming  
    • Produce, Consume Tags 
  • Batch Programming: FactoryTalk Batch  
    • Area Model Configuration – S88 standardized 
    • Recipe development – Unit Operations and Unit Procedures   

DCS Design & Implementation 

A new DCS System was implemented in a virtualized environment using the FactoryTalk View SE, PlantPAx system. The system is scalable and sustainable, as the infrastructure supports future growth for the pilot plant. The HMS, Historian and Win-911 systems were integrated and deployed.  

Components

• SCADA Server: hosts the HMI server, Data server, Alarm and Event server  
• ThinManager and Terminal Server: host thin client sessions for HMI operations on the plant floor  
• Batch Server: ISA -88 standard Batch engine for Recipe development  
• Network Directory Server  

• SQL Server  
• Win-911: Remote alarm notification to pre-configured user groups via FactoryTalk Alarm and Events  
• FactoryTalk Historian SE Server  
• Engineering Workstation – hosts the Rockwell Development suite  

Features 

• Graphics are developed based on the ISA 101 standard.  
• Alarms are based on ISA 18.2 standard.  
• Node security implemented  
• Scope for external Skid Integration  

• Custom graphics developed for external skid integration  
• Sanimatic CIP Skid   
• Centrifuge  

Onsite Support & Equipment Testing

In addition to the unparalleled in-house development effort, E Tech Group has provided over 600 hours of hands-on onsite support to the client – starting from instrument checks to Recipe Testing. From beginning to end, our team kept up with dynamic changes happening during the Wet Runs, adjusting the design accordingly and re-integrating the changes into the active system. 

E Tech Group’s ability to adapt, adjust, and overcome was critical in the final testing and rollout of this solution. Vital processes we monitored, tested and facilitated included:   

• Loop Checks  
• Loop Tuning  
• Equipment Module Testing  
• Phase Testing

• Recipe Testing  
• Troubleshooting  
• Operator Training  

Documentation

Documentation is key in an effective facility-wide automation system, as it allows the users to analyze and identify everything that happens within the system. This has implications for standards, quality control and troubleshooting.

E Tech Group developed the Functional Design Specifications for the entire process – starting from ground zero and then updating all the specs based on the As Built system. Throughout the entirety of the project, our team consolidated information from various vendors – Standalone Skids, construction management and piping and instrumentation vendors – trying our very best to not let any details slip through the cracks.  

We also helped develop SOPs to meet the client’s production and compliance requirements. This gives the owner the ability to better support their own system, since it is well-documented. They can find information easily and without having to bring in a control system expert. This reduces downtime, which is inevitable in a development process for cutting-edge products like cultivated meat. 

The Results: A Positive Learning Experience for All Involved 

After a successful completion of this EPIC project, E Tech Group is fueled with stronger skillsets and hungry for more. We are proud to be part of this journey with the client and extremely thankful to them for trusting and partnering with us in designing and creating the automation solution.  

E Tech Group is a Rockwell Automation partner, supporting manufacturers across industries with a unique brand of innovative, adaptable, robust control systems. We design and implement building-wide automation systems that are secure, user-friendly, and sustainable long-term, so you can focus on maximizing quality, capacity and the ability to scale.

The Project: Address the Possibility of Obsolescence Ahead of Time

A large food and beverage client reached out to us, as they were concerned about the state of both their ethernet and non-ethernet connected Industrial Control System Networks at all thirty-six facilities. Understanding there was an inherent risk of obsolescence as the state of their network, physical asset health and security was of an indeterminable nature, they wanted to have an IT/OT Vulnerability Assessment done to identify any potential threats, what was nearing end of life, and areas where they could push for optimization. They needed an in-depth assessment and a targeted plan for remediation. 

Having an existing working relationship with this client, E Tech Group was immediately one of the control system integrators considered for this project, and was eventually contacted for a bid. If awarded this contract, E Tech Group would be responsible for walking all thirty-six of their facilities to perform the same type of assessment and provide a plan for remediation.

The Process: Assess, Diagnose & Remediate Control System Concerns

After reviewing proposals, the client selected E Tech Group and another control system integrator to provide the initial assessment and develop their individual plans for remediation based on their findings. We knew our IT/OT assessment and subsequent project design needed to be competitive and cutting-edge while remaining attainable.

This assessment would cover both a network driven and physical on-site inventory and evaluation process. Once completed, the findings would be rolled up into reporting that would be consumed at the corporate level down to the operations team.

For the network portion of the assessment, the client’s IT provided our team documentation around program files for the PLCs, HMIs, Drives, and were granted network vizios to view network topology. They were then given limited network access, and using different protocols were able to show where their PLCs, HMIs and Drives were connected on their IT network. This allowed them to dig a bit deeper into their topology rather than just where their IT switches were. 

Lastly, the E Tech Group team worked to compile an asset inventory to help the client understand what equipment was currently living on their network and where in its lifecycle it was. This was a two-prong process that required close collaboration between our automation experts and the client’s operations personnel to ensure that all bases were covered.

Network Assets to be inventoried and assessed: 

• Network switches (Not including Plant Network IT devices, For example Cisco)
• Routers
• Firewalls
• Servers
• PCs

• NATs
• HMIs
• PLCs
• VFDs
• IO Racks
• Any other Devices on a communication network

With it now time to walk the facility floor, our E Tech Group team members would be joined by facility engineers, technicians and other staff to assist in locating the critical components that needed to be identified, documented and inventoried. Working closely with their on-site team allowed this process to move more efficiently and ensure nothing was overlooked. 

Our automation and IT engineers utilized the existing controls network logical infrastructure drawings to work swiftly, documenting and inventorying each panel and its contents, making sure to note if they were reflected accurately in the provided drawings. 

Physical Assets to be inventoried and assessed:

• Physical Network Infrastructure, for both ethernet and non-ethernet.
• Physical Topology and Active Device Inventory
• Switch Selection (Not to include IT Infrastructure Cisco Switches)
• Router Selection (Not to include IT Infrastructure Cisco Routers)
• Communication configurations for both ethernet and non-ethernet (Configuration, Lost Packets, Collisions, etc)
• Environmental Conditions

• Enclosures
• Cable Selection
• Cable Management
• Conduit and Routing
• Cable Labeling
• Power Redundancy
• Grounding

Once all digital and physical assets were documented, the E Tech Group team worked diligently for two weeks to take this massive aggregate of information and transform it into a usable database for the client to digest. 

The Solution: Tailor Remediation Plans to Each Facility’s Needs

Upon initial review, the client was painfully unaware so many components were near end of life or patchworked together to keep them running. This targeted view into their IT and OT helped them prioritize their next steps that would allow them to start working more proactively to improve and maintain their network and asset health.  

Because the E Tech Group assessment best aligned with the client’s standards, we were awarded the contract. Using this initial work as a benchmark, the templated assessment would now be scaled across all thirty-six sites, allowing for tailored plans of remediation.

The Results: Avoided Complications of Large-Scale Obsolescence

Long term, the client intends to set themselves up to implement a 3D model to monitor their network health and asset lifecycle. This system would allow them to utilize predictive maintenance for planned downtime that has the least impact on production, improving efficiency and uptime as upgrades are performed. Understanding the client’s goals, E Tech Group outlined their first plan for remediation:

Site 1 Phases of Remediation:

• Consolidate network servers across all facilities
• Identify isolated assets not living on the network
• Work to migrate non-network connected assets
• Implement connected shop floor cloud-based monitoring to track asset efficiency and network health

The E Tech Group team will continue to work through the remaining sites, utilizing the templated assessment to help build customized plans for remediation. Since these sites were acquired individually and over time, there is a wide range of equipment manufacturers and assets in various stages of their lifecycle. 

Each facility will require a different type and extent of control system upgrade, which is why the assessment design was key in this process. Being able to re-use our assessment strategy across all sites will help in quickly identifying vulnerabilities and being able to move into the remediation process as quickly as possible. 

Begin Your Journey to Industry 4.0

Download our Beginner’s Guide to IT/OT Assessments to gain an understanding of what an IT/OT Assessment is, its benefits, and how to get started today.