We’re your source for automation news. Keep up with the latest industry updates and E Tech employee spotlights, as well as tips and guidance from our manufacturing experts.
When you look at supporting industrial utility systems like ammonia refrigeration, you may wonder how often you might actually encounter them. They appear far more in your daily life than you realize. Look at the rooftop of an industrial facility and see if you notice orange pipes or windsocks. It is likely that this facility has an ammonia refrigeration system. Now, when it comes to the maintenance of these systems, there are a few pain points that tend to plague facilities more than others. Aging Systems: These systems are decades old in most cases, many around 20–30 years old, and tend to require human labor occasionally. By comparison, the average industrial boiler system in the United States is around 30-40 years old, and many of these systems require large amounts of human interaction more often. The thing both these systems share is a lower level of automation, utilizing control panels dependent on relays and contactors to implement control schemes. Incorrect Sequencing Causing Inefficiency: Not only primitive automation, but the inability to adapt to changes in demand or load can present a challenging issue. This can often result in incorrect sequencing equipment due to simple control implementations, leading to lower overall system efficiency. A common instance of incorrect sequencing is having the largest piece of equipment either always running or brought online when demand is lower, which can be less efficient in its operation and energy consumption. Keeping in mind these systems can also be slow to start or restart if personnel are not quickly notified when a fault or issue occurs. Lack of Integration into Larger Control Systems: Even if a modern system has been implemented, it is less likely that it is part of a BMS (Building Management System) or DCS (Distributed Control System) which could better control … Continued
Laurie Cavanaugh, VP of Business Development from E Tech Group, and Nick Hobbs, Senior Regional Sales Manager from Apera AI, recently held a webinar covering the benefits of integrating Apera AI’s 4D vision system into your automation project. There were some great follow-up questions asked by our attendees and we wanted to make sure we shared the As to your Qs! How long does it take for the AI to go through a million learning cycles? Nick: At this time, once all required information is submitted, you’ll receive your vision program back within a week to two weeks. How many different objects can the system handle at one time? What if they have a mix of different finishes? Nick: So right now, the PC alone can handle learning up to 30 different parts. However, other users that have their own internal network, which I know E Tech Group can help you to set up, have hundreds of parts and it takes a mere 20 seconds to simultaneously learn if they’re not part of those original 30. Now as far as different finishes, I have been working with foundries and casting plants where they have metal billets. We can pick clean or corroded, as they have varying finishes. We don’t care. We really don’t mind picking different colors, things of that nature. However, we’re not looking for color at that point, so you just need to let us know what we need and don’t need to look for. But in that sense, it won’t affect us. Laurie: There are some things too that we take into consideration. We do have our own internal industrial IT group that has helped numerous companies’ IT groups set up the proper OT network. Between the hardware and software, servers, and systems, we can help ensure that those … Continued
E Tech Group, a control system integrator that delivers turnkey automation solutions, and Apera AI, a company that offers AI vision guidance, partnered together to produce a webinar covering the capabilities of robotic integration with 4D vision technology, which you can watch below. AI & Vision-Guided Robotics: Talking Points What if we told you that vision-guided robotics powered by AI could do all of this? Identify objects in bins and give a robot picking instructions in only 0.3 seconds. Operate in changing light conditions with no special lighting or sensors. Remove the custom programming that makes projects expensive. Traditional vision systems required a lot of expertise to train and implement, and were not very resilient to lighting changes. The new system from Apera AI is self-programming and can handle variations in lighting conditions. Some key points from the webinar: The system uses two camera sets mounted overhead to look at a bin of parts. It can learn the entire process, including the different parts and the end-of-arm tooling, by itself. It is very fast and can be used in a variety of applications, including process or discrete applications. E Tech Group can also help with the digital modeling of the solution to test it before it’s implemented. Check out the video below to view a recording of the webinar, which was originally presented on May 9, 2023. We hope you’ll gain some new ideas about making your facilities more productive!
Scaling a Biotech Startup from Trial to Manufacturing
A new biotech company with limited staff needed to pivot to full-scale manufacturing. E Tech Group designed, installed, configured, and tested a new AVEVA PI Historian system compliant with FDA’s 21 CFR Part 11 regulation and EU GMP Annex 11 on electronic signatures and records.
The Project: Implement Building-Wide Automation for Manufacturing Scale-Up
A promising biotech start-up with drug candidates progressing from Stage 2 to Stage 3 clinical trials and limited automation staffing, turned to E Tech Group to help with their transition to full-scale manufacturing. At this crucial time, collection of data to be used in their license application and a transition from laboratory protocol to full cGMP manufacturing was critical.
The plant selected for production of clinical material and initial launch had islands of automation that often predated 21 CFR part 11 requirements, and were from numerous OEM manufacturers that were not designed to be integrated into a plant network. Some of the equipment did not even have Ethernet connectivity.
The Challenge: Compliant, Consistent Integration Across Varied Systems
The obstacles in this project had two major facets:
Logistical limitations: The facility was not currently set up for a plant-wide control system that could accommodate market-level manufacturing, and the client’s dedicated in-house automation employees were limited.
Biotech industry standards: Along with food and beverage production, pharmaceutical and life science production has some of the strictest regulations to adhere to. This requires automation systems with extra-secure, redundant design, continuous monitoring and more-than-thorough documentation.
Because of the diverse and disparate OEM systems in the plant, AVEVA PI was chosen as the plant historian, as it has documented success for integrating the required systems. For the cGMP SCADA, HMI backbone, Rockwell’s Factory Talk View Site Edition (ViewSE) was chosen because of the installed base of Rockwell controlled equipment in the plant and the migration path of PanelView HMIs to an FDA 21 CFR Part 11-compliant system.
The Process: SCADA System Standup & Historian Integration
Through a collaborative effort, E Tech Group designed a highly-scalable system for the biotech client by starting small to fit into their budget constraints but can easily be expanded to a large multi-site manufacturing network once the drug candidates are approved. Although the new Data Historian system is currently licensed for only 3000 tags, the underlying infrastructure is designed so that the system can expand to 100,000 tags with only license upgrades and integration of the new tags and systems.
This process was comprised of a new SCADA control system, skid integration with the PI Historian, and integration and deployment of the PI system:
NEW SCADA SYSTEM DESIGN AND IMPLEMENTATION
E Tech Group designed and implemented a new 21 CFR Part 11 compliant SCADA system that is scalable and sustainable for the biotech manufacturing plant. The centralized server/thin client architecture facilitates authentication and logging of operator actions in a cGMP environment.
This SCADA system utilizes a SCADA server that hosts a Data server, an Alarm & Event server and a Terminal server that hosts thin client sessions for the day-to-day HMI operations. The Terminal server also hosts Rockwell’s ThinManager software that is used to set up and manage the thin clients.
An Engineering Workstation (EW) hosts the configuration tools for Rockwell and PI software. All servers run as virtual machines under VMWare for ease of maintenance by the biotech client’s IT department and so that resources can seamlessly be reallocated as the system grows.
User Requirements Specification (URS), Functional Specification (FS) and Software Design Specification (SDS) were developed for the new SCADA system and will serve as living documents for future integrations.
As part of this effort, the E Tech Group team replaced a legacy PanelView HMI with a thin client HMI and converted the legacy Panel Builder software to ViewSE. E Tech Group also converted a legacy SLC 504 controller to a new CompactLogix controller integrated with the new plant SCADA system.
Components that were not converted properly by the Rockwell conversion tools were documented and manually modified to fit the new configuration.
In the HMI conversion process, features of the converted HMI were enhanced by cleaning out the legacy tag database of obsolete tags and revamping the graphics to be compatible with touch screen with easily readable modern fonts. Built-in security features of ViewSE such as autologin logout were implemented and integrated with Windows Active directory. Screen security was also implemented based on domain login.
In the PLC program conversion process, all the key rungs in the legacy ladder logic were commented with the appropriate descriptions in the new program which will aid in code comprehension and maintenance.
E Tech Group provided total support during the installation and commissioning of the new controller and the HMI panel. The team also performed the Factory Acceptance Test (FAT) and Site Acceptance Test (SAT) on the complete SCADA system and controller installation and supported validation activities adhering to GMP.
SKID INTEGRATION INTO AVEVA PI HISTORIAN
For the data historian portion of the project, 26 skid systems were integrated into a central Kepware server to streamline data collection to the PI interface. The following are the types of data sources integrated in the process:
Allen-Bradley/Rockwell Controllers
ControlLogix
CompactLogix
MicroLogix
SLC- 5/05
Parker Scilog B&R
GE ATKA Unicorn
GE iFix
GE XDR Wonderware
Siemens
As part of this effort, E Tech Group defined the architecture to connect all the skids to the client’s manufacturing network and assigned the appropriate IPs to be on the same subnet as the OPC Server machine. A Kepware server was installed to serve as a centralized collection point and the OPC server for PI. The Kepware server was configured to add appropriate channels and devices to establish data source connection and polled tags for each system.
Based on customer requirement, only the tags to be historicized for each system were polled. For legacy systems, this process was particularly challenging due to lack of documentation. E Tech Group studied and simulated individual system software (PLC and HMI codes) to determine the PLC tag address for each tag which will provide the data to be historicized in the PI system.
OSI PI INSTALLATION AND IMPLEMENTATION
E Tech Group designed, installed, configured, and tested a new AVEVA PI Historian system compliant with FDA’s 21 CFR Part 11 regulation and EU GMP Annex 11 on electronic signatures and records.
The PI system was deployed in a virtualized environment utilizing four virtual machines including three servers and one engineering workstation that is shared with the SCADA configuration tools. The PI system is licensed for historicizing 3000 tags and provides backup capacity. The underlying infrastructure design is capable of handling 100,000 tags.
The network architecture was designed with the PI core components communicating on a secured manufacturing network to be able to poll data from various data sources. PI Vision web application was configured and deployed on the corporate network to provide an interface to authorized users to be able to view and analyze trend data and generate batch reports.
E Tech Group configured the PI Vision application on the corporate network for authorized users to trend data as required. Notifications and Analysis were enabled to facilitate notifying the operators of critical alarms and events.
The Solution: Comprehensive Control System Installation & Integration
The scope of work for this project required thorough planning, design and start up. Despite technically being a control systems upgrade, the reality of the project was more like a new system build, which was reflected in E Tech Group’s approach:
1. Specification Development: E Tech Group developed new URS, FS and SDS, all adhering to GMP.
2. Design/architecture: E Tech Group designed the PI system architecture utilizing VMWare virtual machine servers and defined the volumes for each machine to be scalable and sustainable to handle multiple data sources.
3. Installation/Configuration: E Tech Group installed PI core components and configured applications to poll and historicize data perpetually. Configured buffering on the PI Server utilizing the PI Buffer subsystem to prevent loss of data in the event of a failure. Enabled Audit trail on the PI Server to facilitate auditing for GMP purposes.
The following software components were installed and implemented:
PI ICU
PI Data Archive
PI Asset Framework
PI SQL Server
PI SMT
PI System Explorer
PI Buffering Subsystem
PI Vision
PI Notifications and Analysis
4. Security Configuration:
E Tech Group implemented security settings on the PI Server to allow only authorized users to access and view historicized data. This effort involved coordination with the client’s IT department to safely integrate the virtual servers on the network.
5. System Integration and Data Collection:
Integrated a gamut of process systems which hosted a variety of data sources.
The communication to the various controllers was established by performing the necessary configuration including DCOM. All the data sources were integrated into a Kepware OPC server to streamline data polling in the PI Interface via standard OPC protocol.
E Tech Group created PI Points for each system based on the client requirements and published the data to the PI Data Archive for historicization. The exported tag list serves as a living Appendix to the Software Design Specification which facilitates easy integration, testing, and historicization of future systems.
The ability to control and monitor automated processes across the entire facility was an absolute must in this biotech company’s endeavors to scale to full manufacturing. E Tech Group organized the data by creating a structure for each system in PI Asset Framework to facilitate easy access and viewing of data in PI clients such as PI Vision.
5a. DREAM REPORT INSTALLATION AND CONFIGURATION FOR BATCH REPORTS
E Tech Group installed and configured a new batch reporting software for the client, from Ocean Data Systems called “Dream Report.” The reporting software was configured to automatically generate the reports whenever a new batch was run. This effort needed key information such as the batch start time and stop times and unique cycle Ids so that appropriate batch definitions could be set up for robust and automatic report generation to meet the GMP requirements.
Procuring this information was especially challenging because the OEM code design for each system was fundamentally different from all others, and the supporting code for each system was legacy software with no clear documentation of the appropriate tags. E Tech Group’s automation team coordinated with client’s manufacturing to monitor every batch run. Key tag updates were studied and correct triggers were deciphered to set up the batch configuration.
For this implementation, E Tech Group opted to obtain and present the historicized data from the newly-installed AVEVA PI data historian in the batch reports. This reduced the cost for the client without having to purchase any additional licensing software for the interface, as the built-in driver for AVEVA PI AF sever from Dream Report was used.
This method also enhanced the report quality, as the historicized data is pre-filtered and optimized before it is represented on the batch report. Using the PI historical data provides a single source of truth for all data and allows the client to reproduce reports ad-hoc from the historical data.
Analysis and calculations were performed in the AVEVA PI AF to make the reporting more robust and sustainable for automatic report generation. Due to the solid integration and interface with the PI system, all the batch reports are now streamlined and generated without any intervention, able to support the client’s audit requirements.
5b. BATCH REPORTING
To allow the client to generate batch reports, E Tech Group configured an interface to third-party Dream Report batch reporting software.
5c. ONGOING IMPROVEMENTS
The client has purchased two additional pieces of equipment that were designed based on the functional requirements of the plant SCADA system installed for this project. Installation of the new skids is scheduled for later this year.
The HMI screens for the new equipment are built on Rockwell’s ViewSE platform and use identical thin clients to the skid that was upgraded as part of the project. This will allow E Tech Group to import the tags and graphics into the plant SCADA and integrate the new skids into the existing centralized server/thin client architecture to utilize the central authentication services and logging of operator actions in the plant SCADA system. ThinManager will be used to integrate the new thin clients into the network and assure that the correct graphics go to the correct thin clients. Tags from the new skids will be added to Kepware and PI and new reports using Dream Reports will be created for the new systems.
The Results: A Facility-Wide Automation System Capable of Progressive Scaling
It is expected that over time as systems are replaced or upgraded the existing PanelView HMIs will be replaced with thin clients and the systems will be brought under the 21 CFR Part 11 compliant security umbrella of the ViewSE plant SCADA system.
Additionally, efforts are underway to upgrade the plant’s Vaisala temperature monitoring system to add Vaisala’s OPC server which will allow E Tech Group to integrate the approximately 100 Vaisala temperature points into the PI historian.
Although the PI system is currently limited to the initial site installation, the future vision is that this system will be used enterprise wide for research, product development, and future largescale manufacturing plants.
E Tech Group has deep domain expertise in the life sciences industry, including pharmaceuticals, biologics, medical devices, alternative protein production, and more. Our automation teams are adept at building, validating and supporting integrated control systems that are compliant, secure and scalable.
