The Project: Implement Building-Wide Automation for Manufacturing Scale-Up

A promising biotech start-up with drug candidates progressing from Stage 2 to Stage 3 clinical trials and limited automation staffing, turned to E Tech Group to help with their transition to full-scale manufacturing. At this crucial time, collection of data to be used in their license application and a transition from laboratory protocol to full cGMP manufacturing was critical.   

The plant selected for production of clinical material and initial launch had islands of automation that often predated 21 CFR part 11 requirements, and were from numerous OEM manufacturers that were not designed to be integrated into a plant network. Some of the equipment did not even have Ethernet connectivity. 

The Challenge: Compliant, Consistent Integration Across Varied Systems

The obstacles in this project had two major facets:

• Logistical limitations: The facility was not currently set up for a plant-wide control system that could accommodate market-level manufacturing, and the client’s dedicated in-house automation employees were limited.
• Biotech industry standards: Along with food and beverage production, pharmaceutical and life science production has some of the strictest regulations to adhere to. This requires automation systems with extra-secure, redundant design, continuous monitoring and more-than-thorough documentation.

Because of the diverse and disparate OEM systems in the plant, AVEVA  PI was chosen as the plant historian, as it has documented success for integrating the required systems. For the cGMP SCADA, HMI backbone, Rockwell’s Factory Talk View Site Edition (ViewSE) was chosen because of the installed base of Rockwell controlled equipment in the plant and the migration path of PanelView HMIs to an FDA 21 CFR Part 11-compliant system.  

The Process: SCADA System Standup & Historian Integration

Through a collaborative effort, E Tech Group designed a highly-scalable system for the biotech client by starting small to fit into their budget constraints but can easily be expanded to a large multi-site manufacturing network once the drug candidates are approved. Although the new Data Historian system is currently licensed for only 3000 tags, the underlying infrastructure is designed so that the system can expand to 100,000 tags with only license upgrades and integration of the new tags and systems. 

This process was comprised of a new SCADA control system, skid integration with the PI Historian, and integration and deployment of the PI system:

NEW SCADA SYSTEM DESIGN AND IMPLEMENTATION 

E Tech Group designed and implemented a new 21 CFR Part 11 compliant SCADA system that is scalable and sustainable for the biotech manufacturing plant. The centralized server/thin client architecture facilitates authentication and logging of operator actions in a cGMP environment. 

This SCADA system utilizes a SCADA server that hosts a Data server, an Alarm & Event server and a Terminal server that hosts thin client sessions for the day-to-day HMI operations. The Terminal server also hosts Rockwell’s ThinManager software that is used to set up and manage the thin clients. 

An Engineering Workstation (EW) hosts the configuration tools for Rockwell and PI software. All servers run as virtual machines under VMWare for ease of maintenance by the biotech client’s IT department and so that resources can seamlessly be reallocated as the system grows. 

User Requirements Specification (URS), Functional Specification (FS) and Software Design Specification (SDS) were developed for the new SCADA system and will serve as living documents for future integrations. 

As part of this effort, the E Tech Group team replaced a legacy PanelView HMI with a thin client HMI and converted the legacy Panel Builder software to ViewSE. E Tech Group also converted a legacy SLC 504 controller to a new CompactLogix controller integrated with the new plant SCADA system. 

Components that were not converted properly by the Rockwell conversion tools were documented and manually modified to fit the new configuration. 

In the HMI conversion process, features of the converted HMI were enhanced by cleaning out the legacy tag database of obsolete tags and revamping the graphics to be compatible with touch screen with easily readable modern fonts. Built-in security features of ViewSE such as autologin logout were implemented and integrated with Windows Active directory. Screen security was also implemented based on domain login.   

In the PLC program conversion process, all the key rungs in the legacy ladder logic were commented with the appropriate descriptions in the new program which will aid in code comprehension and maintenance.   

E Tech Group provided total support during the installation and commissioning of the new controller and the HMI panel. The team also performed the Factory Acceptance Test (FAT) and Site Acceptance Test (SAT) on the complete SCADA system and controller installation and supported validation activities adhering to GMP. 

SKID INTEGRATION INTO AVEVA PI HISTORIAN 

For the data historian portion of the project, 26 skid systems were integrated into a central Kepware server to streamline data collection to the PI interface. The following are the types of data sources integrated in the process: 

• Allen-Bradley/Rockwell Controllers 
• ControlLogix 
• CompactLogix 
• MicroLogix 
• SLC- 5/05 

• Parker Scilog B&R 
• GE ATKA Unicorn 
• GE iFix 
• GE XDR Wonderware 
• Siemens 

As part of this effort, E Tech Group defined the architecture to connect all the skids to the client’s manufacturing network and assigned the appropriate IPs to be on the same subnet as the OPC Server machine. A Kepware server was installed to serve as a centralized collection point and the OPC server for PI. The Kepware server was configured to add appropriate channels and devices to establish data source connection and polled tags for each system. 

Based on customer requirement, only the tags to be historicized for each system were polled. For legacy systems, this process was particularly challenging due to lack of documentation. E Tech Group studied and simulated individual system software (PLC and HMI codes) to determine the PLC tag address for each tag which will provide the data to be historicized in the PI system. 

OSI PI INSTALLATION AND IMPLEMENTATION 

E Tech Group designed, installed, configured, and tested a new AVEVA PI Historian system compliant with FDA’s 21 CFR Part 11 regulation and EU GMP Annex 11 on electronic signatures and records.  

The PI system was deployed in a virtualized environment utilizing four virtual machines including three servers and one engineering workstation that is shared with the SCADA configuration tools. The PI system is licensed for historicizing 3000 tags and provides backup capacity. The underlying infrastructure design is capable of handling 100,000 tags. 

The network architecture was designed with the PI core components communicating on a secured manufacturing network to be able to poll data from various data sources. PI Vision web application was configured and deployed on the corporate network to provide an interface to authorized users to be able to view and analyze trend data and generate batch reports. 

E Tech Group configured the PI Vision application on the corporate network for authorized users to trend data as required. Notifications and Analysis were enabled to facilitate notifying the operators of critical alarms and events. 

The Solution: Comprehensive Control System Installation & Integration

The scope of work for this project required thorough planning, design and start up. Despite technically being a control systems upgrade, the reality of the project was more like a new system build, which was reflected in E Tech Group’s approach:

1. Specification Development: E Tech Group developed new URS, FS and SDS, all adhering to GMP. 

2. Design/architecture: E Tech Group designed the PI system architecture utilizing VMWare virtual machine servers and defined the volumes for each machine to be scalable and sustainable to handle multiple data sources. 

3. Installation/Configuration: E Tech Group installed PI core components and configured applications to poll and historicize data perpetually. Configured buffering on the PI Server utilizing the PI Buffer subsystem to prevent loss of data in the event of a failure. Enabled Audit trail on the PI Server to facilitate auditing for GMP purposes. 

The following software components were installed and implemented: 

• PI ICU 
• PI Data Archive 
• PI Asset Framework 
• PI SQL Server 
• PI SMT 

• PI System Explorer 
• PI Buffering Subsystem 
• PI Vision 
• PI Notifications and Analysis 

4. Security Configuration:  

E Tech Group implemented security settings on the PI Server to allow only authorized users to access and view historicized data. This effort involved coordination with the client’s IT department to safely integrate the virtual servers on the network. 

5. System Integration and Data Collection: 

Integrated a gamut of process systems which hosted a variety of data sources. 

The communication to the various controllers was established by performing the necessary configuration including DCOM. All the data sources were integrated into a Kepware OPC server to streamline data polling in the PI Interface via standard OPC protocol. 

E Tech Group created PI Points for each system based on the client requirements and published the data to the PI Data Archive for historicization. The exported tag list serves as a living Appendix to the Software Design Specification which facilitates easy integration, testing, and historicization of future systems. 

The ability to control and monitor automated processes across the entire facility was an absolute must in this biotech company’s endeavors to scale to full manufacturing. E Tech Group organized the data by creating a structure for each system in PI Asset Framework to facilitate easy access and viewing of data in PI clients such as PI Vision. 

5a. DREAM REPORT INSTALLATION AND CONFIGURATION FOR BATCH REPORTS 

E Tech Group installed and configured a new batch reporting software for the client, from Ocean Data Systems called “Dream Report.” The reporting software was configured to automatically generate the reports whenever a new batch was run. This effort needed key information such as the batch start time and stop times and unique cycle Ids so that appropriate batch definitions could be set up for robust and automatic report generation to meet the GMP requirements. 

Procuring this information was especially challenging because the OEM code design for each system was fundamentally different from all others, and the supporting code for each system was legacy software with no clear documentation of the appropriate tags. E Tech Group’s automation team coordinated with client’s manufacturing to monitor every batch run. Key tag updates were studied and correct triggers were deciphered to set up the batch configuration. 

For this implementation, E Tech Group opted to obtain and present the historicized data from the newly-installed AVEVA PI data historian in the batch reports. This reduced the cost for the client without having to purchase any additional licensing software for the interface, as the built-in driver for AVEVA  PI AF sever from Dream Report was used.

This method also enhanced the report quality, as the historicized data is pre-filtered and optimized before it is represented on the batch report. Using the PI historical data provides a single source of truth for all data and allows the client to reproduce reports ad-hoc from the historical data.  

Analysis and calculations were performed in the AVEVA PI AF to make the reporting more robust and sustainable for automatic report generation. Due to the solid integration and interface with the PI system, all the batch reports are now streamlined and generated without any intervention, able to support the client’s audit requirements. 

5b. BATCH REPORTING 

To allow the client to generate batch reports, E Tech Group configured an interface to third-party Dream Report batch reporting software. 

5c. ONGOING IMPROVEMENTS

The client has purchased two additional pieces of equipment that were designed based on the functional requirements of the plant SCADA system installed for this project. Installation of the new skids is scheduled for later this year. 

The HMI screens for the new equipment are built on Rockwell’s ViewSE platform and use identical thin clients to the skid that was upgraded as part of the project. This will allow E Tech Group to import the tags and graphics into the plant SCADA and integrate the new skids into the existing centralized server/thin client architecture to utilize the central authentication services and logging of operator actions in the plant SCADA system. ThinManager will be used to integrate the new thin clients into the network and assure that the correct graphics go to the correct thin clients. Tags from the new skids will be added to Kepware and PI and new reports using Dream Reports will be created for the new systems. 

The Results: A Facility-Wide Automation System Capable of Progressive Scaling

It is expected that over time as systems are replaced or upgraded the existing PanelView HMIs will be replaced with thin clients and the systems will be brought under the 21 CFR Part 11 compliant security umbrella of the ViewSE plant SCADA system. 

Additionally, efforts are underway to upgrade the plant’s Vaisala temperature monitoring system to add Vaisala’s OPC server which will allow E Tech Group to integrate the approximately 100 Vaisala temperature points into the PI historian. 

Although the PI system is currently limited to the initial site installation, the future vision is that this system will be used enterprise wide for research, product development, and future largescale manufacturing plants. 

E Tech Group has deep domain expertise in the life sciences industry, including pharmaceuticals, biologics, medical devices, alternative protein production, and more. Our automation teams are adept at building, validating and supporting integrated control systems that are compliant, secure and scalable.