Implementation of an FDA-Compliant Biologics Process Control System
E Tech Group’s process control solutions included a manufacturing execution system for FDA-compliant data collection and reporting, as well as a building management system and other vendor-supplied control systems.
Project: Turnkey Factory Automation for a Biologics Client
E Tech Group implemented a $1.8M process control system (PCS) for the Massachusetts Biologic Laboratories (MBL) state-of-the-art facility in Mattapan, MA. The MBL facility performs cell culture and purification of monoclonal antibodies (MAb) and filling of licensed and R&D stage liquid parenteral (injectable) products that cannot be terminally sterilized. The monoclonal antibody production centers around cell culture and purification.
Solution: Build a Custom, Fully-Integrated Automation System
E Tech Group installed a PLC- and SCADA-based PCS system utilizing redundant Allen-Bradley ControlLogix 5550 processors with 1756 series I/O, 1794 Flex Series I/O, Festo solenoid blocks, and Mettler Toledo scale indicators. The I/O is distributed between 13 control panels which are interconnected with DeviceNet and ControlNet I/O networks. The PCS communicates approximately 400 data points over Ethernet to the various equipment vendor-supplied systems.
The PCS consists of a SCADA/Historian fault-tolerant server, manufacturing execution system (MES) fault tolerant server, web reporting server, 11 portable client workstations, 7 fixed client workstations, and 2 engineering workstations. The 11 portable workstations contain wireless Ethernet capability and are mounted on a portable cart with an onboard battery and barcode scanner.
The following software packages were used as part of the PCS system:
- SCADA – GE/Proficy iFix
- Historian – GE/Proficy Historian
- Data Analysis – GE/Proficy Real Time Information Portal
- Reporting – Crystal Reports
- Manufacturing Execution System – Elan XFP, Oracle database
- Alarm/Event Storage – SQL Server
The following equipment is located in the MAb production area:
- Buffer preparation tank
- Portable pH adjustment tank
- 55L, 500L, and 2500L bioreactors
- Portable Q-eluate tank
- Harvest tank
- Portable nanofiltration tank
- Chromatography skid
- Final bulk tank
- Ultrafiltration / diafiltration skid
- Fill / finish production
- Vial washer, parts washer, and autoclave
- Depyrogenation tunnel and filling line
- Labeling and packaging equipment
The following utility systems are located in the MPL Mattapan Facility:
- Water for injection (WFI) pre-treatment / soft water generation
- Acid day tank
- WFI still, and hot WFI storage tank
- Caustic day tank
- Compressed air generation
- Clean in place (CIP) skid
- Chilled process glycol generation
- Clean steam generation
- Hot process glycol generation and distribution
Results: An Efficient, Compliant, Comprehensive Plant-Wide Control System
Our engineers implemented a PCS to control process and cleaning steps for the buffer preparation tank, harvest tank, pH adjustment tank, Q-eluate tank, and nanofiltration tank through hardwired I/O. The PCS coordinates CIP operations with the CIP skid, controls CIP and WFI distribution, and interfaces with the various equipment vendor-supplied systems over Ethernet. The PCS stores alarms and process data from the skids and hardwired I/O.
The PCS provides a MES that includes material tracking, equipment management, barcode recognition and creation, and electronic work instructions (EWI) for execution of process steps in the MBL facility. Operators use barcode scanners to verify and record raw materials, semi-finished goods, finished goods, and equipment that is created and/or used in the MBL facility.
The system also consists of a number of equipment vendor-supplied control systems and a building management system (BMS). The PCS provides supervisory control and data acquisition (SCADA), batch execution, electronic work instructions, material tracking, equipment management, barcode recognition and creation, historical data collection, and reporting. All data collection is in accordance with 21 CFR Part 11.
With deep domain expertise in the life sciences and biopharma industry, E Tech Group creates process automation solutions that keep you compliant with R&D, production, and quality control regulations while retaining the ability to scale and adapt in this ever-changing industry. Improve processes and reduce costs with strategic, robust building automation.
Provides a MES that includes material tracking, equipment management, barcode recognition and creation, and electronic work instructions (EWI) for execution of process steps in the MBL facility. Operators use barcode scanners to verify and record raw materials, semi-finished goods, finished goods, and equipment that is created and/or used in the MBL facility.
The system also consists of a number of equipment vendor-supplied control systems and a building management system (BMS). The PCS provides supervisory control and data acquisition (SCADA), batch execution, electronic work instructions, material tracking, equipment management, barcode recognition and creation, historical data collection, and reporting. All data collection is in accordance with 21 CFR Part 11.
Increased Flexibility & Better Quality for a Pharmaceutical Manufacturer
E Tech Group’s control system engineers implemented a low-cost Rockwell Automation PlantPAx process automation system to upgrade a single-product semi work facility.
The Project: Accomplish Two Automation Capabilities at Once
AMPAC Fine Chemicals (AFC) A California pharmaceutical manufacturer hired E Tech Group to upgrade the process control system of a single-product semiwork facility. During the project, the client asked E Tech to also convert the equipment to a flexible midscale, multi-product production facility… with no increase in project budget.
This presented several obstacles for our automation engineers; we needed to implement a multi-purpose and comprehensive control system that would accomplish both goals for the client while staying within budget and deadline constraints.
The Solution: Kill Two Birds with… Rockwell’s PlantPAx?
The existing hard-coded control system was built in the 1990s, and was not capable of handling the flexible manufacturing desired. Our automation engineers designed and installed an updated process control system that met the strict FDA regulatory requirements and high level of security required for the production of active pharmaceutical ingredients (APIs). This approach to their integration upgrade was a cost-effective alternative to a traditional server-based batch solution.
E Tech’s engineers selected the PlantPAx process automation system from Rockwell Automation for the complete control system upgrade. PlantPAx provided a control solution that allowed the flexibility to easily change the manufacturing schedule and produce more than one product on the equipment.
Rockwell’s PlantPAx software is industry-leading in terms of plant-wide controls integration, scalability, adaptability, and perhaps most important of all in pharmaceutical automation, consistency. With support from Allen Bradley automation products, we created an integrated control system that won’t require entire retrofits each time it needs an upgrade, saving the client money.
Additionally, the use of PlantPAx Logix Batch & Sequence Manager allowed a controller-based solution and saved money by removing the need for costly custom code or additional server infrastructure. FactoryTalk View Site Edition software was chosen so that personnel can configure recipes directly in the controller.
Technology We Used:
- PlantPAx
- PlantPAx Logix Batch & Sequence Manager
- FactoryTalk View Site Edition
- EtherNet/IP network
- Allen-Bradley FLEX I/O modules
- Allen-Bradley IntelliCENTER motor control center
- Allen-Bradley PowerFlex VFDs
The Results: Efficient, Scalable, Versatile Pharmaceutical Automation
The new control system has improved the quality, consistency and speed of manufacturing the client’s APIs by automating processes that were once manual. The new system has reduced reliance on paper-based documentation, which offers additional cost savings. The success and quick ROI of this project has led to the client pursuing additional upgrade projects with E Tech Group and Rockwell Automation in other plants.
“With the automated batch solution from Rockwell Automation, we’re improving the quality and consistency of our APIs, and avoiding the delays that naturally occur during manual operations,” said Director of Automated Systems and Calibration, for our client. “The Rockwell Automation solution allows us to meet customer needs more quickly than our competitors while still staying in full compliance with industry regulations.”
The director continued, “We need to align ourselves with leading partner companies in order to be competitive and each project we complete with [E Tech Group] and Rockwell Automation is a perfect example of that strategy.”
An industry leader in control systems integration and industrial automation, E Tech Group offers comprehensive controls system services for clients across a wide breadth of industries. With deep domain expertise in the life sciences industry, we provide pharmaceutical manufacturers with turnkey factory automation solutions that look toward the future of the company and the industry, so our clients can stay competitive, no matter how active the market.
Customized Validated Critical Monitoring System
E Tech Group supported a biopharmaceutical’s expansion with a new validated control system. The company focuses on serious, debilitating and genetic diseases.
The Project: Implement a Highly-Customized Facility-Wide Control System
A biopharmaceutical company committed to bringing to patients novel products for the treatment of rare and ultra-rare diseases with a focus on serious, debilitating genetic diseases, was looking to expand. When they began building a new pharmaceutical research laboratory at their New England facility with an extremely condensed timeline, they reached out to E Tech Group to provide the qualified automated monitoring and alarm system (QMAS).
The new validated control system would monitor 78 relocated or new pieces of equipment. The critical monitoring and alarm system project completion date was just three months after the project kickoff, which was a very tight schedule for a customized system such as this.
The laboratory equipment to be controlled by the QMAS included:
- Incubators
- Lab refrigerators
- Lab freezers
- Cryogenic freezers
- Water generation skid
- Autoclave
- Glass washer
The Challenges: Too Little Time, Too Much Noise & a Global Pandemic
Challenge #1: A Tight Timeline
With the extremely short timeline, An E Tech Group Project Manager front loaded the project to stay ahead of schedule. The PM outlined the functional specification and got initial customer buy-off to prevent any unwanted surprises later on.
Part of the strategy to condense the schedule was to conduct the factory acceptance test (FAT) on the software functionality at E Tech Group. The control panel was tested at the fabricator, then the wiring was tested at the site with the control panel after it was installed.
Challenge #2: Construction Zone
Along with the tight timeframe, another project challenge was that the laboratory was being constructed concurrently with the development of the QMAS and delivery of the equipment. The laboratory was constructed on one floor of a five-story building and was gutted down to the concrete walls before building began. The space needed to be highly customized for the laboratory, with specialized equipment, air filtration suitable for biopharmaceuticals, and more.
Challenge #3: Site Acceptance Testing
The customer notified E Tech Group that three pieces of laboratory equipment were indefinitely delayed due to the onset of the pandemic. We quickly pivoted to create a cascading site acceptance testing (SAT) schedule, allowing the partially complete system to be validated and functional until the final equipment arrived from overseas.
Our engineers were able to break the system apart such that when they incorporated the utility equipment, it did not affect anything that was already tested. They used the modular design of the system with the notification to make a modular test.
E Tech Group’s system integrators also tested the automation system with all pieces of equipment that arrived on time so that they wouldn’t have to be retested when the remaining three machines arrived. They were able to incorporate the new equipment without impacting any testing that was already done.
Solution: Create a QMAS Tailored to the Pharmaceutical Client’s Needs
Our engineers proposed a Rockwell Automation-based solution. The off-the-shelf equipment was customized to provide the critical alarming, control and reporting to the client’s specifications. The hardware provided a single control cabinet containing an Allen-Bradley ControlLogix PLC with ControlLogix I/O. The software platform included a FactoryTalk View SE server with local HMI clients, FactoryTalk Historian, FactoryTalk VantagePoint, FactoryTalk ViewPoint, and WIN-911.
E Tech Group designed a customized, validated QMAS to provide centralized monitoring and alarming for all 78 pieces of equipment, featuring local audio-visual beacons throughout the facility with an email notification system.
With the new system, the client no longer has to be present in front of a machine to know it is in alarm. Those employees on site are notified by the beacons alert to check the equipment status. Employees both on- and off-site receive alarm notifications by email.
The client created a detailed hierarchy of permissions and multiple notification lists that are triggered by which specific area has the equipment fault. At the facility, notifications are sent by room, and each area beacon covers a group of rooms.
In addition, the system provides data historian and reporting functions. It gathers data from all equipment and stores it for historical reference. To facilitate cloud data storage, the system includes a hardware module as a local data collection and storage site before it is uploaded. The client requested a minimum of 24 hours of local data storage, and the hardware module provided is able to capture more than 10 days of data, for redundancy.
E Tech Group built the entire control panel, tested it, and developed all of the automation and controls software for the project. Once the new control system was integrated, it was tested and implemented on site. Control modules were developed in-house by E Tech Group based on past experience and projects to reduce development and test time. The functionality of each module has been tested and validated. Taking one day per control module to test, the five pre-validated control modules saved five days of testing.
Results: Exactly What the Customer Ordered, But Better
With the efforts of the teams at E Tech Group as well as the client, the system was constructed, programmed, installed on site, and tested in according to the aggressive schedule. Had it not been for the COVID-19 equipment delay, they would have finished the project on time, exactly three months from the project kickoff. Considering the obstacles we faced from inside and outside the facility, our automation team worked very hard and the project went well.
The client was able to run the new automated alarm system even before the last three machines arrived, and they seem very happy with its performance. With the QMAS in place, future additions to lab and storage equipment can be installed at the customer’s own pace.
E Tech Group offers top-tier process automation solutions to entities across the life sciences industry, as well as data centers, metals, material handling, and more. Using the best in automation products, like Rockwell Automation and Siemens Global, our engineers and staff design, build and implement strategic control systems that increase your production capacity and capability while decreasing safety and security risks.
BioTech Manufacturing Scale-Up
A promising biotech startup with drug candidates progressing from stage 2 to stage 3 clinical had limited automation staffing. The startup turned to E Tech Group to transition their processes to full-scale manufacturing.
Project: Create Scalable, Integrated Controls out of a Piecemealed Automation System
The plant selected for production of clinical material and initial launch had islands of automation that often predated 21 CFR part 11 requirements from numerous OEM manufacturers that were not designed to be integrated into a plant network. Some of the equipment did not even have Ethernet connectivity.
Through a collaborative effort, the automation team designed a highly-scalable system for the biotech client by starting small to fit into their budget. This system can be expanded to a large multi-site manufacturing network when the drugs are approved.
Although the new Data Historian system is currently licensed for only 3000 tags, the underlying infrastructure is designed so that the system can expand to 100,000 tags with only license upgrades and integration of the new tags and systems.
Because of the diverse disparate OEM systems in the plant, OSIsoft’s PI was chosen as the plant historian as it has documented success for integrating the required systems. For the cGMP HMI plant SCADA system backbone, Rockwell’s Factory Talk View Site Edition (ViewSE) was chosen because of the installed base of Rockwell controlled equipment in the plant and the migration path of PanelView HMIs to a 21 CFR Part 11 compliant system.
Solution: A Scalable SCADA System that Keeps Up With Compliance Specs
E Tech Group designed and implemented a new 21 CFR Part 11-compliant SCADA system that is scalable and sustainable for the biotech manufacturing plant.
- The centralized server/thin client architecture facilitates authentication and logging of operator actions in a cGMP environment.
- This SCADA system utilizes a SCADA server that hosts a Data server, an Alarm & Event server and a Terminal server that hosts thin client sessions for the day to day HMI operations.
- The Terminal server also hosts Rockwell’s ThinManager software that is used to setup and manage the thin clients.
- An Engineering Workstation (EWS) hosts the configuration tools for Rockwell and PI software.
- All servers run as virtual machines under VMWare for ease of maintenance by the biotech client’s IT department and so that resources can seamlessly be reallocated as the system grows.
User Requirements Specification(URS), Functional Specification(FS) and Software Design Specification (SDS) were developed for the new SCADA system and will serve as living documents for future integrations.
As part of this effort, the E Tech Group automation team replaced a legacy PanelView HMI with a thin client HMI and converted the legacy Panel Builder software to ViewSE. We also converted a legacy SLC 504 controller to a new CompactLogix controller integrated with the new plant SCADA system. Components that were not converted properly by the conversion tools were documented and manually modified to fit the new configuration.
In the HMI conversion process, features of the converted HMI were enhanced by cleaning out the legacy tag database of obsolete tags and revamping the graphics to be compatible with touch screen with easily readable modern fonts. Built-in security features of ViewSE such as autologin logout were implemented and integrated with Windows Active directory. Screen security was also implemented based on domain logins.
In the PLC program conversion process, all the key rungs in the legacy ladder logic were commented with the appropriate descriptions in the new program which will aid in code comprehension and maintenance.
Result: End-to-End System Support for Plant-Wide Coverage
E Tech Group provided total support during the installation and commissioning of the new controller and the HMI panel. We also performed the Factory Acceptance Test (FAT) and Site Acceptance Test (SAT) on the complete SCADA system and controller installation and supported validation activities adhering to GMP.