A commercial drug manufacturer in collaboration with E Tech Group implemented a critical monitoring system that required environmental data to be recorded in a 21 CFR Part 11 compliant manner.
What Was at Stake: The Role of Critical Control Data in This Facility
In this facility, critical control data played a central role in maintaining uptime, coordinating production, and ensuring regulatory compliance. This included:
- Real-time process values and system setpoints
- Historical logs used for performance benchmarking
- Alarm and event data essential for troubleshooting
Losing access to this data — even temporarily — could have disrupted operations, reduced batch quality, or triggered costly downtime.
The Plan to Mitigate Risk
With an intent to mitigate the risk of data loss, the drug manufacturer chose Vaisala instrumentation to read and locally store environmental data. The Vaisala instrumentation chosen can store environmental data locally on the device and report environmental data to an OPC-compliant SCADA system.
The drug manufacturer had a need for a solution that moves data stored on the devices to the Rockwell Automation SCADA system in the event of data loss. E Tech Group provided a solution that detects data loss, recovers the lost data from the devices, and moves it to the SCADA system.
The solution included a Windows service that polls the SCADA for gaps in historian data. If a gap is found, the solution pulls data from the buffers and populates it into the SCADA system. This provides redundancy and verification that critical data resides in a single repository, ensuring data integrity. The customer relied solely on the existing SCADA and used the most robust instrumentation to protect against critical data loss.
The Need for a Professional System Integrator
The end-user chose Vaisala instrumentation equipped with loggers that provide data buffering to protect against critical data loss. However, the buffering capability did not integrate “off the shelf” with the existing SCADA, Rockwell Automation FactoryTalk. The buffered data from the loggers did not automatically populate data to the historian after a SCADA outage.
The architecture selection allowed the customer to deviate from the more typical control system implementation, which would include PLCs and remote I/O nodes. The architecture consisted of 150+ Vaisala temperature and humidity sensors and data loggers mounted throughout the facility. The data loggers have a 10-year internal battery and a sample storage capacity that can store from days to years’ worth of data based on the number of connected sensors and the configurable sample interval. Each sensor was connected to a data logger, which buffered data and transmitted real-time data to an OPC server. The OPC server was also the home of the custom solution described within.
The Solution: How E Tech Responded to the Hardware Failure
Once the server failure was reported, E Tech’s team took immediate action to restore critical operations. The process involved:
- Accessing the customer’s secure off-site backup, which had been configured to archive daily system images
- Deploying the restored system image to new hardware, avoiding the need to reprogram or manually reconfigure central applications
- Verifying tag databases and communication pathways, ensuring all systems were fully functional before handing operations back to the facility team
This rapid response minimized downtime and demonstrated the value of proactive backup planning.
The Details
The software solution developed was a Windows service designed to actively poll the historian to find gaps in critical data. This polling occurred on a configurable, timed interval. A gap in historical data could occur for a variety of reasons, including power loss, communication loss, or servers not running. Once a gap in data was identified, the service retrieved data from the instrument data loggers and populated the historian with missing data. If a critical data gap is detected and the solution is unable to retrieve data or unable to populate the data gap, a SCADA alarm is generated and relevant personnel are notified using WIN-911 notification software. The service is multithreaded, so it takes full advantage of the VM’s available resources, reducing the time to recover from a critical data gap. This is especially useful when multiple critical data gaps have been created from different loggers, as data can be read from more than one logger simultaneously. The service is also able to fully recover from any system failure, such as an unexpected shutdown of the VM. The software solution decision tree that highlights the functionality is detailed below.
This service allowed the client to use their selected data loggers with their existing industry-standard historian and SCADA system. It requires no action on the part of the client. The alternative that had been provided to the client was a standalone system based around the data logger selection, which would have required separate specialized historian and reporting software to be purchased and maintained. This would mean separate reporting and trending tools for data for just one section of the plant. This would require additional training to use and considerable expansion of the virtual infrastructure. Additionally, a custom solution in the event of non-recoverable data setting off an HMI alarm would have been required.
Success in Critical Data Restoration
The solution has been validated, is now running, and has successfully inserted otherwise lost critical data. Currently, power/communication could be lost for 3 months and the provided solution would insert critical data for all 150+ sensors upon communication restoration. This solution has allowed the end-user to implement a critical monitoring system that consists of instrumentation that connects directly to the existing SCADA system. This eliminated the need for multiple sources of critical data, as well as a programmable logic controller with remote input/output nodes spread across the facility.
Lessons Learned: What This Incident Highlights About Automation Resilience
While no one anticipates a sudden hardware failure, this incident underscores the importance of prepping for the unexpected. Key takeaways include:
- Daily off-site backups are not a luxury — they’re essential
- A well-documented restoration plan can shave hours or days off recovery
- Working with a trusted automation partner ensures expertise is available when it matters most
By prioritizing these practices, manufacturers can maintain business continuity even when faced with unexpected infrastructure loss.s of critical data, as well as a programmable logic controller with remote input/output nodes spread across the facility.