News & Knowledge

The Project: Implement Building-Wide Automation for Manufacturing Scale-Up

A promising biotech start-up with drug candidates progressing from Stage 2 to Stage 3 clinical trials and limited automation staffing, turned to E Tech Group to help with their transition to full-scale manufacturing. At this crucial time, collection of data to be used in their license application and a transition from laboratory protocol to full cGMP manufacturing was critical.   

The plant selected for production of clinical material and initial launch had islands of automation that often predated 21 CFR part 11 requirements, and were from numerous OEM manufacturers that were not designed to be integrated into a plant network. Some of the equipment did not even have Ethernet connectivity. 

The Challenge: Compliant, Consistent Integration Across Varied Systems

The obstacles in this project had two major facets:

• Logistical limitations: The facility was not currently set up for a plant-wide control system that could accommodate market-level manufacturing, and the client’s dedicated in-house automation employees were limited.
• Biotech industry standards: Along with food and beverage production, pharmaceutical and life science production has some of the strictest regulations to adhere to. This requires automation systems with extra-secure, redundant design, continuous monitoring and more-than-thorough documentation.

Because of the diverse and disparate OEM systems in the plant, AVEVA  PI was chosen as the plant historian, as it has documented success for integrating the required systems. For the cGMP SCADA, HMI backbone, Rockwell’s Factory Talk View Site Edition (ViewSE) was chosen because of the installed base of Rockwell controlled equipment in the plant and the migration path of PanelView HMIs to an FDA 21 CFR Part 11-compliant system.  

The Process: SCADA System Standup & Historian Integration

Through a collaborative effort, E Tech Group designed a highly-scalable system for the biotech client by starting small to fit into their budget constraints but can easily be expanded to a large multi-site manufacturing network once the drug candidates are approved. Although the new Data Historian system is currently licensed for only 3000 tags, the underlying infrastructure is designed so that the system can expand to 100,000 tags with only license upgrades and integration of the new tags and systems. 

This process was comprised of a new SCADA control system, skid integration with the PI Historian, and integration and deployment of the PI system:

NEW SCADA SYSTEM DESIGN AND IMPLEMENTATION 

E Tech Group designed and implemented a new 21 CFR Part 11 compliant SCADA system that is scalable and sustainable for the biotech manufacturing plant. The centralized server/thin client architecture facilitates authentication and logging of operator actions in a cGMP environment. 

This SCADA system utilizes a SCADA server that hosts a Data server, an Alarm & Event server and a Terminal server that hosts thin client sessions for the day-to-day HMI operations. The Terminal server also hosts Rockwell’s ThinManager software that is used to set up and manage the thin clients. 

An Engineering Workstation (EW) hosts the configuration tools for Rockwell and PI software. All servers run as virtual machines under VMWare for ease of maintenance by the biotech client’s IT department and so that resources can seamlessly be reallocated as the system grows. 

User Requirements Specification (URS), Functional Specification (FS) and Software Design Specification (SDS) were developed for the new SCADA system and will serve as living documents for future integrations. 

As part of this effort, the E Tech Group team replaced a legacy PanelView HMI with a thin client HMI and converted the legacy Panel Builder software to ViewSE. E Tech Group also converted a legacy SLC 504 controller to a new CompactLogix controller integrated with the new plant SCADA system. 

Components that were not converted properly by the Rockwell conversion tools were documented and manually modified to fit the new configuration. 

In the HMI conversion process, features of the converted HMI were enhanced by cleaning out the legacy tag database of obsolete tags and revamping the graphics to be compatible with touch screen with easily readable modern fonts. Built-in security features of ViewSE such as autologin logout were implemented and integrated with Windows Active directory. Screen security was also implemented based on domain login.   

In the PLC program conversion process, all the key rungs in the legacy ladder logic were commented with the appropriate descriptions in the new program which will aid in code comprehension and maintenance.   

E Tech Group provided total support during the installation and commissioning of the new controller and the HMI panel. The team also performed the Factory Acceptance Test (FAT) and Site Acceptance Test (SAT) on the complete SCADA system and controller installation and supported validation activities adhering to GMP. 

SKID INTEGRATION INTO AVEVA PI HISTORIAN 

For the data historian portion of the project, 26 skid systems were integrated into a central Kepware server to streamline data collection to the PI interface. The following are the types of data sources integrated in the process: 

• Allen-Bradley/Rockwell Controllers 
• ControlLogix 
• CompactLogix 
• MicroLogix 
• SLC- 5/05 

• Parker Scilog B&R 
• GE ATKA Unicorn 
• GE iFix 
• GE XDR Wonderware 
• Siemens 

As part of this effort, E Tech Group defined the architecture to connect all the skids to the client’s manufacturing network and assigned the appropriate IPs to be on the same subnet as the OPC Server machine. A Kepware server was installed to serve as a centralized collection point and the OPC server for PI. The Kepware server was configured to add appropriate channels and devices to establish data source connection and polled tags for each system. 

Based on customer requirement, only the tags to be historicized for each system were polled. For legacy systems, this process was particularly challenging due to lack of documentation. E Tech Group studied and simulated individual system software (PLC and HMI codes) to determine the PLC tag address for each tag which will provide the data to be historicized in the PI system. 

OSI PI INSTALLATION AND IMPLEMENTATION 

E Tech Group designed, installed, configured, and tested a new AVEVA PI Historian system compliant with FDA’s 21 CFR Part 11 regulation and EU GMP Annex 11 on electronic signatures and records.  

The PI system was deployed in a virtualized environment utilizing four virtual machines including three servers and one engineering workstation that is shared with the SCADA configuration tools. The PI system is licensed for historicizing 3000 tags and provides backup capacity. The underlying infrastructure design is capable of handling 100,000 tags. 

The network architecture was designed with the PI core components communicating on a secured manufacturing network to be able to poll data from various data sources. PI Vision web application was configured and deployed on the corporate network to provide an interface to authorized users to be able to view and analyze trend data and generate batch reports. 

E Tech Group configured the PI Vision application on the corporate network for authorized users to trend data as required. Notifications and Analysis were enabled to facilitate notifying the operators of critical alarms and events. 

The Solution: Comprehensive Control System Installation & Integration

The scope of work for this project required thorough planning, design and start up. Despite technically being a control systems upgrade, the reality of the project was more like a new system build, which was reflected in E Tech Group’s approach:

1. Specification Development: E Tech Group developed new URS, FS and SDS, all adhering to GMP. 

2. Design/architecture: E Tech Group designed the PI system architecture utilizing VMWare virtual machine servers and defined the volumes for each machine to be scalable and sustainable to handle multiple data sources. 

3. Installation/Configuration: E Tech Group installed PI core components and configured applications to poll and historicize data perpetually. Configured buffering on the PI Server utilizing the PI Buffer subsystem to prevent loss of data in the event of a failure. Enabled Audit trail on the PI Server to facilitate auditing for GMP purposes. 

The following software components were installed and implemented: 

• PI ICU 
• PI Data Archive 
• PI Asset Framework 
• PI SQL Server 
• PI SMT 

• PI System Explorer 
• PI Buffering Subsystem 
• PI Vision 
• PI Notifications and Analysis 

4. Security Configuration:  

E Tech Group implemented security settings on the PI Server to allow only authorized users to access and view historicized data. This effort involved coordination with the client’s IT department to safely integrate the virtual servers on the network. 

5. System Integration and Data Collection: 

Integrated a gamut of process systems which hosted a variety of data sources. 

The communication to the various controllers was established by performing the necessary configuration including DCOM. All the data sources were integrated into a Kepware OPC server to streamline data polling in the PI Interface via standard OPC protocol. 

E Tech Group created PI Points for each system based on the client requirements and published the data to the PI Data Archive for historicization. The exported tag list serves as a living Appendix to the Software Design Specification which facilitates easy integration, testing, and historicization of future systems. 

The ability to control and monitor automated processes across the entire facility was an absolute must in this biotech company’s endeavors to scale to full manufacturing. E Tech Group organized the data by creating a structure for each system in PI Asset Framework to facilitate easy access and viewing of data in PI clients such as PI Vision. 

5a. DREAM REPORT INSTALLATION AND CONFIGURATION FOR BATCH REPORTS 

E Tech Group installed and configured a new batch reporting software for the client, from Ocean Data Systems called “Dream Report.” The reporting software was configured to automatically generate the reports whenever a new batch was run. This effort needed key information such as the batch start time and stop times and unique cycle Ids so that appropriate batch definitions could be set up for robust and automatic report generation to meet the GMP requirements. 

Procuring this information was especially challenging because the OEM code design for each system was fundamentally different from all others, and the supporting code for each system was legacy software with no clear documentation of the appropriate tags. E Tech Group’s automation team coordinated with client’s manufacturing to monitor every batch run. Key tag updates were studied and correct triggers were deciphered to set up the batch configuration. 

For this implementation, E Tech Group opted to obtain and present the historicized data from the newly-installed AVEVA PI data historian in the batch reports. This reduced the cost for the client without having to purchase any additional licensing software for the interface, as the built-in driver for AVEVA  PI AF sever from Dream Report was used.

This method also enhanced the report quality, as the historicized data is pre-filtered and optimized before it is represented on the batch report. Using the PI historical data provides a single source of truth for all data and allows the client to reproduce reports ad-hoc from the historical data.  

Analysis and calculations were performed in the AVEVA PI AF to make the reporting more robust and sustainable for automatic report generation. Due to the solid integration and interface with the PI system, all the batch reports are now streamlined and generated without any intervention, able to support the client’s audit requirements. 

5b. BATCH REPORTING 

To allow the client to generate batch reports, E Tech Group configured an interface to third-party Dream Report batch reporting software. 

5c. ONGOING IMPROVEMENTS

The client has purchased two additional pieces of equipment that were designed based on the functional requirements of the plant SCADA system installed for this project. Installation of the new skids is scheduled for later this year. 

The HMI screens for the new equipment are built on Rockwell’s ViewSE platform and use identical thin clients to the skid that was upgraded as part of the project. This will allow E Tech Group to import the tags and graphics into the plant SCADA and integrate the new skids into the existing centralized server/thin client architecture to utilize the central authentication services and logging of operator actions in the plant SCADA system. ThinManager will be used to integrate the new thin clients into the network and assure that the correct graphics go to the correct thin clients. Tags from the new skids will be added to Kepware and PI and new reports using Dream Reports will be created for the new systems. 

The Results: A Facility-Wide Automation System Capable of Progressive Scaling

It is expected that over time as systems are replaced or upgraded the existing PanelView HMIs will be replaced with thin clients and the systems will be brought under the 21 CFR Part 11 compliant security umbrella of the ViewSE plant SCADA system. 

Additionally, efforts are underway to upgrade the plant’s Vaisala temperature monitoring system to add Vaisala’s OPC server which will allow E Tech Group to integrate the approximately 100 Vaisala temperature points into the PI historian. 

Although the PI system is currently limited to the initial site installation, the future vision is that this system will be used enterprise wide for research, product development, and future largescale manufacturing plants. 

E Tech Group has deep domain expertise in the life sciences industry, including pharmaceuticals, biologics, medical devices, alternative protein production, and more. Our automation teams are adept at building, validating and supporting integrated control systems that are compliant, secure and scalable.

The Project: Innovate a Facility-Wide Control System for a Pilot Alternative Protein Plant

Alternative proteins, including lab-grown meats, are the new horizon of ethical, sustainable meat production. Specifically, cultivated meats are just emerging, with the first expected to be available within a few years. Cultivated meat production plants are a new frontier as far as food and beverage automation goes.

A client leading the way in this vein of alternative protein was scaling up from lab production, where they were able to refine their process of growing cells at benchtop scale to a pilot scale. Proving it was possible to replicate their lab driven results in an increased volume setting, if successful, would help move into large-scale production and distribution once necessary approvals came through.  

Previously having worked with E Tech Group as an automation partner that helped guide and coach them through thoughtfully identifying their needs for their first automation project, they were confident in our ability to help plan, design, implement, test and support an end-to-end, turnkey automation solution for this new facility.  

The Challenges: A New Type of Food Production Requires New Automation Tech

Because this was a new installation of cutting-edge technology, there was no lack of presented challenges on this project. We were all in uncharted waters. 

Understanding that nothing in this industry had yet been designed and executed on this scale, the entire process would remain fluid. Change and adaptation would be a constant theme throughout the planning and implementation.  

Knowing what was being produced was designed for human consumption, ensuring the developed solution exceeded industry and federal guidelines and standards was non-negotiable. E Tech Group needed to ensure this facility’s automation system took all of the unknown out of the cultivated meat production process.  

The Solution: Utilize Rockwell’s PlantPAx 5.0 to Support the Automated Systems 

The design of this project needed to be just as creative as the client’s endeavors in this new field of meat production. Communication and flexibility was key in pulling off this leading-edge, facility-wide automation system integration.

Process Design

The plant being a pilot and the organization essentially a start-up, the process engineering environment was naturally ever-evolving and dynamic. The E Tech Group team kept up with the constant changing strategies, provided technical guidance in streamlining the design to a point where it could be automated, and clearly communicated changes to the process design as they went live. This helped maintain open lines of communication with stake holders and create realistic expectations for delivery.   

Process Outline 

• Media Prep – Media Transfer, CIP of the Media Prep suite, SIP of the Media Prep Suite  
• Clean In Place (CIP) – CIP of the Bioreactors and the transfer lines 
• Steam in Place (SIP) – SIP of the Bioreactors and the transfer lines  

•  Inoculation  
• Media Filtration  
• Cell Growth  
• Seeding/Circulation  

As part of the design effort, Pin Charts were developed and maintained. These defined the valve alignment and various schemes for Equipment Modules corresponding to each phase step, covering a matrix of over 100 phase steps and over 300 devices, over 200 valves, and a gamut of control modules.  

The team also worked to design customized control loops (involving Cascade methodology) for pH, Dissolved Oxygen (DO) and Temperature Control (both Vessel and Jacket) of the Bioreactors. These complex algorithms use shared equipment and phase-specific control strategies.   

The process design also involved deciphering functionality of specialized equipment such as Mix Proof Valves – the Equipment Module developed for this valve comprises of 9 schemes in its operation.  

Electrical Design  

• Panel Layout  
• Network Layout  

Automation Products

Our process automation team had a daunting task at hand – to synthesize the enormous design and get the system ready for production in less than 6 months. The automation effort included the following tasks:  

• PLC Program Design: Rockwell Automation PlantPAx 5.0, ControlLogix L85 EP, Studio 5000 v33  
  • Mix of 4.x and embedded 5.0 process objects
  • Function Block and Ladder Logic used to program Control and Equipment Modules   
  • Structured Text programming for Phases  
    • Phase Parameter definitions  
    • Hold, Abort and Stop Logic  
    • Operator Prompts and Reporting Logic  
    • Linking/Messaging with Parallel Phases 
  • Equipment Module Programming  
    • Scheme Propagation  
    • State Model  
    • Permissive  
    • Interlocks   

• Valve Supervisor Equipment Modules  
• Online Configuration Tool – development and maintenance for bulk import of IO  
  • Handshaking with External Skids (Sanimatic CIP, Alfa Laval Centrifuge)  
    • Message Instruction programming  
    • Produce, Consume Tags 
  • Batch Programming: FactoryTalk Batch  
    • Area Model Configuration – S88 standardized 
    • Recipe development – Unit Operations and Unit Procedures   

DCS Design & Implementation 

A new DCS System was implemented in a virtualized environment using the FactoryTalk View SE, PlantPAx system. The system is scalable and sustainable, as the infrastructure supports future growth for the pilot plant. The HMS, Historian and Win-911 systems were integrated and deployed.  

Components

• SCADA Server: hosts the HMI server, Data server, Alarm and Event server  
• ThinManager and Terminal Server: host thin client sessions for HMI operations on the plant floor  
• Batch Server: ISA -88 standard Batch engine for Recipe development  
• Network Directory Server  

• SQL Server  
• Win-911: Remote alarm notification to pre-configured user groups via FactoryTalk Alarm and Events  
• FactoryTalk Historian SE Server  
• Engineering Workstation – hosts the Rockwell Development suite  

Features 

• Graphics are developed based on the ISA 101 standard.  
• Alarms are based on ISA 18.2 standard.  
• Node security implemented  
• Scope for external Skid Integration  

• Custom graphics developed for external skid integration  
• Sanimatic CIP Skid   
• Centrifuge  

Onsite Support & Equipment Testing

In addition to the unparalleled in-house development effort, E Tech Group has provided over 600 hours of hands-on onsite support to the client – starting from instrument checks to Recipe Testing. From beginning to end, our team kept up with dynamic changes happening during the Wet Runs, adjusting the design accordingly and re-integrating the changes into the active system. 

E Tech Group’s ability to adapt, adjust, and overcome was critical in the final testing and rollout of this solution. Vital processes we monitored, tested and facilitated included:   

• Loop Checks  
• Loop Tuning  
• Equipment Module Testing  
• Phase Testing

• Recipe Testing  
• Troubleshooting  
• Operator Training  

Documentation

Documentation is key in an effective facility-wide automation system, as it allows the users to analyze and identify everything that happens within the system. This has implications for standards, quality control and troubleshooting.

E Tech Group developed the Functional Design Specifications for the entire process – starting from ground zero and then updating all the specs based on the As Built system. Throughout the entirety of the project, our team consolidated information from various vendors – Standalone Skids, construction management and piping and instrumentation vendors – trying our very best to not let any details slip through the cracks.  

We also helped develop SOPs to meet the client’s production and compliance requirements. This gives the owner the ability to better support their own system, since it is well-documented. They can find information easily and without having to bring in a control system expert. This reduces downtime, which is inevitable in a development process for cutting-edge products like cultivated meat. 

The Results: A Positive Learning Experience for All Involved 

After a successful completion of this EPIC project, E Tech Group is fueled with stronger skillsets and hungry for more. We are proud to be part of this journey with the client and extremely thankful to them for trusting and partnering with us in designing and creating the automation solution.  

E Tech Group is a Rockwell Automation partner, supporting manufacturers across industries with a unique brand of innovative, adaptable, robust control systems. We design and implement building-wide automation systems that are secure, user-friendly, and sustainable long-term, so you can focus on maximizing quality, capacity and the ability to scale.