The Project

Our client had an existing data lake solution they quickly outgrew. As they increased the number of operational facilities and concurrent users accessing reports, the old design experienced an unanticipated bottleneck: with one API and one database servicing needs across the globe, adding more data and accessing the data in reports would take an average of thirty seconds per page and proving a barricade for future growth.

The client reached out to E Tech Group’s Mission Critical and Advanced Software teams to help with this issue. They were nervous to try again after now two failed data lake attempts with existing solutions. to try again, But they trusted our long-standing partnership and expertise, having then given us a small proof-of-concept budget. After discussing the problem, we proposed a solution built upon a microservice architecture in Azure. This gave them data redundancy, fail-over, and a solution that scales horizontally with their business. Through this discovery effort, we identified that their current reporting practices were a pain point for them. They were using PowerBI reports and had thousands of single files, one per customer per report. Each one had to be manually managed and updated.

Solution:

First, we started with the cloud solution. The team built an API in Azure using ASP.NET 8,  InfluxDB, and a timeseries database instead of the traditional choice of SQL. Since their data is measured and consumed over time, the purpose-built InfluxDB served the previously defined client needs, having a proven record of high performance and availability. The last piece was to replace their PowerBI reports with a single-page application using React and TypeScript. The team tied into their existing systems and authentication to automatically retrieve customer-specific data and dynamically update the SPA This allowed standardized reporting to be managed in a single location, one time.

Second, the client needed a way to get their operational data from their machines and buildings into their cloud solution. They had several disparate devices from multiple vendors, some using different communication protocols. The team installed KepServerEX as a middleware between the hardware and Ignition that enabled the collection of information from varying devices into a central system. Then Ignition polled the data at a high frequency and writing to a store-forward database before aggregating it and sending the reduced set to the cloud for long-term storage and retrieval by client reporting. This solution enabled the client to have data redundancy in case of communication failure or outages, while also having high data granularity at the location most critical to operations.

Each of these pieces were deployed per facility and architected in such a way that multiple installations can service a single location, while a central gateway in Azure controls routing to needed resources. Due to the horizontal architecture, the client can scale their solution indefinitely without hitting a ceiling that would impact performance.

Results:

The client was thrilled with the initial success and decided to move to full implementation of the proof-of-concept, especially for the high-availability and SPA report system, which represents a massive quality-of-life-increase for multiple departments in the company. The team successfully delivered on E Tech Group’s “Zero Defects” promise, even in the proof-of-concept phase.

The most promising prospect for their future with this new solution is the elimination of scaling barricades. The client will be able to increase the amount of telemetry data collected, and the number of facilities supported, and deliver a single, standardized reporting experience to their users. This new solution is faster, more intuitive, offers better insights, and more importantly, has the intelligence to automatically adapt to their business needs.