The Project: Apply a Scalable Automation Solution to a Shallow Data Lake

Our client had an existing data lake solution they quickly outgrew. As they increased the number of operational facilities and concurrent users accessing reports, the old design experienced an unanticipated bottleneck: with one API and one database servicing needs across the globe, adding more data and accessing the data in reports would take an average of thirty seconds per page, proving a barricade for future growth.

The client reached out to E Tech Group’s Mission Critical and Advanced Software teams to help with this issue; they trusted our long-standing partnership and expertise, giving us a small proof-of-concept budget for this new challenge.

The Challenge: Getting Toes Back in the Water of Systems Upgrades

Part of the obstacles to this aging data lake issue is that the client had already tried two existing solutions, both of which had failed. Understandably, they were gun-shy about undertaking a huge system retrofit with no surety of the result. Our Mission Critical and Advanced Software engineers would need a proof-of-concept solution that was foolproof at any scale.

Through this discovery effort, we identified that their current reporting practices were a pain point for them. They were using PowerBI reports and had thousands of single files, one per customer per report. Each one had to be manually managed and updated. We proposed a solution built upon a microservice architecture in Azure. This gave them data redundancy, fail-over, and a solution that scales horizontally with their business.

The Solution: Clouds, Communication & Consistent Reporting

The remedy designed by our automation and IT engineers was multi-pronged:

High-Performance Data Storage & Access

First, we started with the cloud solution. The team built an API in Azure using ASP.NET 8, InfluxDB, and a time series database instead of the traditional choice of SQL. Since their data is measured and consumed over time, the purpose-built InfluxDB served the previously defined client needs, having a proven record of high performance and availability.

Centralized Reporting & Management

The last piece was to replace their PowerBI reports with a single-page application using React and TypeScript. The team tied into their existing systems and authentication to automatically retrieve customer-specific data and dynamically update the SPA. This allowed standardized reporting to be managed in a single location, one time.

Organized Communication Without Data Loss

Second, the client needed a way to get their operational data from their machines and buildings into their cloud solution. They had several disparate devices from multiple vendors, some using different communication protocols.

The team installed KepServerEX as a middleware between the hardware and Ignition that enabled the collection of information from varying devices into a central system. Then Ignition polled the data at a high frequency and writing to a store-forward database before aggregating it and sending the reduced set to the cloud for long-term storage and retrieval by client reporting.

This solution enabled the client to have data redundancy in case of communication failure or outages while also having high data granularity at the location most critical to operations.

Each of these pieces were deployed per facility and architected in such a way that multiple installations can service a single location, while a central gateway in Azure controls routing to needed resources. Due to the horizontal architecture, the client can scale their solution indefinitely without hitting a ceiling that would impact performance.

The Results: Unlimited Applications, Unlimited Potential for Growth

The client was thrilled with the initial success and decided to move to full implementation of the proof-of-concept, especially for the high-availability and SPA report system, which represents a massive quality-of-life-increase for multiple departments in the company. Our automation team successfully delivered on E Tech Group’s “Zero Defects” promise, even in the proof-of-concept phase.

The most promising prospect for their future with this new solution is the elimination of scaling barricades. The client will be able to increase the amount of telemetry data collected, and the number of facilities supported, and deliver a single, standardized reporting experience to their users. This new solution is faster, more intuitive, offers better insights, and more importantly, has the intelligence to automatically adapt to their business needs.