We’re your source for automation news. Keep up with the latest industry updates and E Tech employee spotlights, as well as tips and guidance from our manufacturing experts.
E Tech Group, a control system integrator that delivers turnkey automation solutions, and Apera AI, a company that offers AI vision guidance, partnered together to produce a webinar covering the capabilities of robotic integration with 4D vision technology, which you can watch below. AI & Vision-Guided Robotics: Talking Points What if we told you that vision-guided robotics powered by AI could do all of this? Identify objects in bins and give a robot picking instructions in only 0.3 seconds. Operate in changing light conditions with no special lighting or sensors. Remove the custom programming that makes projects expensive. Traditional vision systems required a lot of expertise to train and implement, and were not very resilient to lighting changes. The new system from Apera AI is self-programming and can handle variations in lighting conditions. Some key points from the webinar: The system uses two camera sets mounted overhead to look at a bin of parts. It can learn the entire process, including the different parts and the end-of-arm tooling, by itself. It is very fast and can be used in a variety of applications, including process or discrete applications. E Tech Group can also help with the digital modeling of the solution to test it before it’s implemented. Check out the video below to view a recording of the webinar, which was originally presented on May 9, 2023. We hope you’ll gain some new ideas about making your facilities more productive!
Scaling a Biotech Startup from Trial to Manufacturing
A new biotech company with limited staff needed to pivot to full-scale manufacturing. E Tech Group designed, installed, configured, and tested a new AVEVA PI Historian system compliant with FDA’s 21 CFR Part 11 regulation and EU GMP Annex 11 on electronic signatures and records.
The Project: Implement Building-Wide Automation for Manufacturing Scale-Up
A promising biotech start-up with drug candidates progressing from Stage 2 to Stage 3 clinical trials and limited automation staffing, turned to E Tech Group to help with their transition to full-scale manufacturing. At this crucial time, collection of data to be used in their license application and a transition from laboratory protocol to full cGMP manufacturing was critical.
The plant selected for production of clinical material and initial launch had islands of automation that often predated 21 CFR part 11 requirements, and were from numerous OEM manufacturers that were not designed to be integrated into a plant network. Some of the equipment did not even have Ethernet connectivity.
The Challenge: Compliant, Consistent Integration Across Varied Systems
The obstacles in this project had two major facets:
Logistical limitations: The facility was not currently set up for a plant-wide control system that could accommodate market-level manufacturing, and the client’s dedicated in-house automation employees were limited.
Biotech industry standards: Along with food and beverage production, pharmaceutical and life science production has some of the strictest regulations to adhere to. This requires automation systems with extra-secure, redundant design, continuous monitoring and more-than-thorough documentation.
Because of the diverse and disparate OEM systems in the plant, AVEVA PI was chosen as the plant historian, as it has documented success for integrating the required systems. For the cGMP SCADA, HMI backbone, Rockwell’s Factory Talk View Site Edition (ViewSE) was chosen because of the installed base of Rockwell controlled equipment in the plant and the migration path of PanelView HMIs to an FDA 21 CFR Part 11-compliant system.
The Process: SCADA System Standup & Historian Integration
Through a collaborative effort, E Tech Group designed a highly-scalable system for the biotech client by starting small to fit into their budget constraints but can easily be expanded to a large multi-site manufacturing network once the drug candidates are approved. Although the new Data Historian system is currently licensed for only 3000 tags, the underlying infrastructure is designed so that the system can expand to 100,000 tags with only license upgrades and integration of the new tags and systems.
This process was comprised of a new SCADA control system, skid integration with the PI Historian, and integration and deployment of the PI system:
NEW SCADA SYSTEM DESIGN AND IMPLEMENTATION
E Tech Group designed and implemented a new 21 CFR Part 11 compliant SCADA system that is scalable and sustainable for the biotech manufacturing plant. The centralized server/thin client architecture facilitates authentication and logging of operator actions in a cGMP environment.
This SCADA system utilizes a SCADA server that hosts a Data server, an Alarm & Event server and a Terminal server that hosts thin client sessions for the day-to-day HMI operations. The Terminal server also hosts Rockwell’s ThinManager software that is used to set up and manage the thin clients.
An Engineering Workstation (EW) hosts the configuration tools for Rockwell and PI software. All servers run as virtual machines under VMWare for ease of maintenance by the biotech client’s IT department and so that resources can seamlessly be reallocated as the system grows.
User Requirements Specification (URS), Functional Specification (FS) and Software Design Specification (SDS) were developed for the new SCADA system and will serve as living documents for future integrations.
As part of this effort, the E Tech Group team replaced a legacy PanelView HMI with a thin client HMI and converted the legacy Panel Builder software to ViewSE. E Tech Group also converted a legacy SLC 504 controller to a new CompactLogix controller integrated with the new plant SCADA system.
Components that were not converted properly by the Rockwell conversion tools were documented and manually modified to fit the new configuration.
In the HMI conversion process, features of the converted HMI were enhanced by cleaning out the legacy tag database of obsolete tags and revamping the graphics to be compatible with touch screen with easily readable modern fonts. Built-in security features of ViewSE such as autologin logout were implemented and integrated with Windows Active directory. Screen security was also implemented based on domain login.
In the PLC program conversion process, all the key rungs in the legacy ladder logic were commented with the appropriate descriptions in the new program which will aid in code comprehension and maintenance.
E Tech Group provided total support during the installation and commissioning of the new controller and the HMI panel. The team also performed the Factory Acceptance Test (FAT) and Site Acceptance Test (SAT) on the complete SCADA system and controller installation and supported validation activities adhering to GMP.
SKID INTEGRATION INTO AVEVA PI HISTORIAN
For the data historian portion of the project, 26 skid systems were integrated into a central Kepware server to streamline data collection to the PI interface. The following are the types of data sources integrated in the process:
Allen-Bradley/Rockwell Controllers
ControlLogix
CompactLogix
MicroLogix
SLC- 5/05
Parker Scilog B&R
GE ATKA Unicorn
GE iFix
GE XDR Wonderware
Siemens
As part of this effort, E Tech Group defined the architecture to connect all the skids to the client’s manufacturing network and assigned the appropriate IPs to be on the same subnet as the OPC Server machine. A Kepware server was installed to serve as a centralized collection point and the OPC server for PI. The Kepware server was configured to add appropriate channels and devices to establish data source connection and polled tags for each system.
Based on customer requirement, only the tags to be historicized for each system were polled. For legacy systems, this process was particularly challenging due to lack of documentation. E Tech Group studied and simulated individual system software (PLC and HMI codes) to determine the PLC tag address for each tag which will provide the data to be historicized in the PI system.
OSI PI INSTALLATION AND IMPLEMENTATION
E Tech Group designed, installed, configured, and tested a new AVEVA PI Historian system compliant with FDA’s 21 CFR Part 11 regulation and EU GMP Annex 11 on electronic signatures and records.
The PI system was deployed in a virtualized environment utilizing four virtual machines including three servers and one engineering workstation that is shared with the SCADA configuration tools. The PI system is licensed for historicizing 3000 tags and provides backup capacity. The underlying infrastructure design is capable of handling 100,000 tags.
The network architecture was designed with the PI core components communicating on a secured manufacturing network to be able to poll data from various data sources. PI Vision web application was configured and deployed on the corporate network to provide an interface to authorized users to be able to view and analyze trend data and generate batch reports.
E Tech Group configured the PI Vision application on the corporate network for authorized users to trend data as required. Notifications and Analysis were enabled to facilitate notifying the operators of critical alarms and events.
The Solution: Comprehensive Control System Installation & Integration
The scope of work for this project required thorough planning, design and start up. Despite technically being a control systems upgrade, the reality of the project was more like a new system build, which was reflected in E Tech Group’s approach:
1. Specification Development: E Tech Group developed new URS, FS and SDS, all adhering to GMP.
2. Design/architecture: E Tech Group designed the PI system architecture utilizing VMWare virtual machine servers and defined the volumes for each machine to be scalable and sustainable to handle multiple data sources.
3. Installation/Configuration: E Tech Group installed PI core components and configured applications to poll and historicize data perpetually. Configured buffering on the PI Server utilizing the PI Buffer subsystem to prevent loss of data in the event of a failure. Enabled Audit trail on the PI Server to facilitate auditing for GMP purposes.
The following software components were installed and implemented:
PI ICU
PI Data Archive
PI Asset Framework
PI SQL Server
PI SMT
PI System Explorer
PI Buffering Subsystem
PI Vision
PI Notifications and Analysis
4. Security Configuration:
E Tech Group implemented security settings on the PI Server to allow only authorized users to access and view historicized data. This effort involved coordination with the client’s IT department to safely integrate the virtual servers on the network.
5. System Integration and Data Collection:
Integrated a gamut of process systems which hosted a variety of data sources.
The communication to the various controllers was established by performing the necessary configuration including DCOM. All the data sources were integrated into a Kepware OPC server to streamline data polling in the PI Interface via standard OPC protocol.
E Tech Group created PI Points for each system based on the client requirements and published the data to the PI Data Archive for historicization. The exported tag list serves as a living Appendix to the Software Design Specification which facilitates easy integration, testing, and historicization of future systems.
The ability to control and monitor automated processes across the entire facility was an absolute must in this biotech company’s endeavors to scale to full manufacturing. E Tech Group organized the data by creating a structure for each system in PI Asset Framework to facilitate easy access and viewing of data in PI clients such as PI Vision.
5a. DREAM REPORT INSTALLATION AND CONFIGURATION FOR BATCH REPORTS
E Tech Group installed and configured a new batch reporting software for the client, from Ocean Data Systems called “Dream Report.” The reporting software was configured to automatically generate the reports whenever a new batch was run. This effort needed key information such as the batch start time and stop times and unique cycle Ids so that appropriate batch definitions could be set up for robust and automatic report generation to meet the GMP requirements.
Procuring this information was especially challenging because the OEM code design for each system was fundamentally different from all others, and the supporting code for each system was legacy software with no clear documentation of the appropriate tags. E Tech Group’s automation team coordinated with client’s manufacturing to monitor every batch run. Key tag updates were studied and correct triggers were deciphered to set up the batch configuration.
For this implementation, E Tech Group opted to obtain and present the historicized data from the newly-installed AVEVA PI data historian in the batch reports. This reduced the cost for the client without having to purchase any additional licensing software for the interface, as the built-in driver for AVEVA PI AF sever from Dream Report was used.
This method also enhanced the report quality, as the historicized data is pre-filtered and optimized before it is represented on the batch report. Using the PI historical data provides a single source of truth for all data and allows the client to reproduce reports ad-hoc from the historical data.
Analysis and calculations were performed in the AVEVA PI AF to make the reporting more robust and sustainable for automatic report generation. Due to the solid integration and interface with the PI system, all the batch reports are now streamlined and generated without any intervention, able to support the client’s audit requirements.
5b. BATCH REPORTING
To allow the client to generate batch reports, E Tech Group configured an interface to third-party Dream Report batch reporting software.
5c. ONGOING IMPROVEMENTS
The client has purchased two additional pieces of equipment that were designed based on the functional requirements of the plant SCADA system installed for this project. Installation of the new skids is scheduled for later this year.
The HMI screens for the new equipment are built on Rockwell’s ViewSE platform and use identical thin clients to the skid that was upgraded as part of the project. This will allow E Tech Group to import the tags and graphics into the plant SCADA and integrate the new skids into the existing centralized server/thin client architecture to utilize the central authentication services and logging of operator actions in the plant SCADA system. ThinManager will be used to integrate the new thin clients into the network and assure that the correct graphics go to the correct thin clients. Tags from the new skids will be added to Kepware and PI and new reports using Dream Reports will be created for the new systems.
The Results: A Facility-Wide Automation System Capable of Progressive Scaling
It is expected that over time as systems are replaced or upgraded the existing PanelView HMIs will be replaced with thin clients and the systems will be brought under the 21 CFR Part 11 compliant security umbrella of the ViewSE plant SCADA system.
Additionally, efforts are underway to upgrade the plant’s Vaisala temperature monitoring system to add Vaisala’s OPC server which will allow E Tech Group to integrate the approximately 100 Vaisala temperature points into the PI historian.
Although the PI system is currently limited to the initial site installation, the future vision is that this system will be used enterprise wide for research, product development, and future largescale manufacturing plants.
E Tech Group has deep domain expertise in the life sciences industry, including pharmaceuticals, biologics, medical devices, alternative protein production, and more. Our automation teams are adept at building, validating and supporting integrated control systems that are compliant, secure and scalable.
In the same way you use automation to make consistent, quality products, why not use CIP to ensure germ-free cleaning every time? You’ve automated your process and packaging and have computer-generated paper trails to show food safety and quality throughout your facility. Why wouldn’t you do the same with cleaning—wherever it’s practical? Granted, there are certain pieces of equipment you just have to take out of the process to clean manually, but for other components—like piping, some conveyor belting, certain pumps, mixers, tanks and other equipment—you can clean them in place (CIP) and have automated recordkeeping to show proof of cleaning and sanitation to regulating bodies when they ask. An Expert Opinion on Sanitation Safety in Food & Beverage Automation “CIP is an integral part of cleaning and sanitation, which affects the food safety of any facility. Cleaning and sanitation not only help you comply with regulatory requirements, but also protect consumers and your brand from contamination and recalls,” says Pablo Coronel, Ph.D., CRB Senior Fellow in Food Process & Food Safety. “CIP is carried out in process lines without dismantling and must reach every nook and cranny of the process equipment; thus, well-designed equipment that is amenable to CIP (as opposed to COP—clean out of place) is a must for any installation. Materials of construction, drainability, ease of cleaning, and prevention of dead areas/zones are some of the criteria for design,” adds Coronel. Besides the equipment design, a successful CIP system depends on several variables, such as water flow rate and temperature, chemicals used (concentration) and time of cleaning. These variables are set in cooperation with the chemical supplier by the characteristics of the product to be cleaned (viscosity, stickiness, amount of fouling and presence of allergens) and the equipment that needs to be cleaned. The presence of allergens, … Continued
Taking cells from livestock to grow meat in a lab might seem like science fiction, but E Tech Group is making these processes a reality. Alternative proteins and cultivated meat are the ethical, sustainable future of meat production, and require new innovations in automation technology and control system design. Cassy Gardner, Director of Operations, and Anees Chamarty, Lead Automation Engineer at E Tech Group, Life Sciences West, discussed their work to develop a scalable and sustainable solution for cultivated meat. The first-of-its-kind pilot plant is essentially growing meat in a lab, with the help of automation. “This is something that’s not been done before, so our clients are in a very innovative space,” said Gardner. “This made early involvement with the client imperative to reducing design changes throughout the course of the project. We really had to stay, from an automation perspective, in lockstep with the clients as they evolved what they were looking to accomplish from a process perspective,” said Gardner. Utilizing Control System Software that’s Accessible but Flexible To design, program and test the distributed control system (DCS) in about six months, E Tech Group needed a platform with an out-of-the-box solution, but one that was also flexible. E Tech Group turned to Rockwell Automation’s PlantPAx platform, which is known for its robust features and adaptability. It was a time- and cost-effective solution for this control system, as it needed to be highly customizable. In addition to a tight deadline and the need for flexibility, the project included some 1,100 individual I/O points. The PlantPAx process library helped with relatively easy and quick programming, and using module programming where the state machines were embedded also helped speed things along. “So at a high level, we deliver a code that’s reusable and scalable,” said Chamarty. Batch Processing Automation: System … Continued
