News & Knowledge

Author contributor: Solutions Architect, Kevin Romer

When a food and beverage industry manufacturer faced a cyberattack, the previously implemented best practices for network architecture minimized the damage. E Tech Group had conducted a thorough IT/OT assessment for the company, based on which they also designed and installed essential network improvements. Consequently, all OT assets remained inaccessible to the attacker. The facility continued operating at full capacity while mitigating the limited damage to the enterprise network.

Introduction

In a previous article, the argument for tackling cybersecurity in the manufacturing sector was made. It included alarming data about the rising frequency of attacks on critical infrastructure and the large costs associated with them. Manufacturers must continue to evaluate, manage, and secure their networks and consequently, their vital assets, even as they strive to reap the benefits of digitalization.

An E Tech Group food and beverage client recently had a successful breach of their enterprise network. However, because this manufacturer had previously worked with E Tech Group associates to complete recommendations stemming from an IT/OT assessment, the scope of the attack was limited to the IT side of the network and was caught early. As a result, critical assets remained out of reach and production was not impacted. 

This article discusses the networking strategies that were implemented by this client well ahead of the cyberattack. It then presents the timeline of events that occurred due to the attack, the consequences, as well as the potential consequences had the network not been properly configured and secured. Finally, it should be stated that a great deal of network security best practices involves some amount of hidden information, about both a network’s structure, as well as the security tools that guard it. As a result, the brands of equipment installed within this facility are not identified.

The Decision to Implement OT Network Best Practices

A predominant high-level characteristic of the management team within this company was their forward-thinking and proactive approach. As a team, this group thrived on learning about how new technology and trends within the industry could benefit their facility, including those surrounding security.

In 2018, an E Tech Group associate assessed the facility’s network, uncovering significant vulnerabilities in its architecture. Of particular concern was the lack of proper isolation between the IT and OT networks, which posed a serious risk of allowing attackers to breach the IT network and gain access to critical infrastructure in the OT network. In response to these findings, management adopted recommended security measures, specifically implementing a demilitarized zone (DMZ) architecture as part of their proactive approach to network security.

Foundational DMZ Architecture Strategies

The following list outlines some of the basic strategies for industrial network security. These strategies include the following:

• A firewall with restricted access control lists is positioned at the edge of the enterprise network, connecting to the internet.
• An additional firewall, serving as the DMZ zone, is implemented to block all network connections between the IT and OT networks.
• Servers and a jump box are essential within the DMZ to facilitate secure data connections.
• Separate domain controllers are established for the IT and OT networks.

This design helps to ensure that in the event of a breach in the outer layer, the supplementary layers will remain effective in safeguarding the OT network including the industrial control systems and the critical infrastructure dependent on it. This particular manufacturer further strengthened their security posture by implementing a network threat detection engine.

Each component of the DMZ architecture and the threat detection engine is discussed in more detail below.

OT Network Firewall

The OT network firewall is designed specifically to protect critical infrastructure and will typically include:

• Enhanced access control: Access control configurations restrict access to sensitive areas of the building network, allowing only necessary communications to pass through.

• Protocol filtering: The firewall can be configured to filter communication protocols specific to the OT network, such as industrial protocols (MODBUS, PROFIBUS, OPC UA, Ethernet/IP, etc.), and block unauthorized protocols.

• Segmentation and zoning: The firewall segments the OT network into security zones based on function or security requirements, enforcing distinct policies and access controls for each zone to mitigate the risk of breaches spreading across the network.

• Monitoring and logging: The firewall offers advanced monitoring and logging capabilities, enabling detailed diagnostics of OT network issues and early detection of potential security incidents.

Separate Domains for the IT and OT Networks

By segregating the facility’s network into distinct IT and OT domains, unique access policies can be enforced through separate domain controllers. In this specific instance, despite an attacker compromising the enterprise network, access to the OT network was effectively prevented because it operated independently within its own domain and required completely separate login credentials.

Cyberattack Sequence of Events

The following list outlines the sequence of events that occurred during the cyberattack at the food and beverage facility:

• Initially, a suspicious log entry flagged by the threat detection engine was overlooked, as it did not impact operations.
• A few days later, another anomaly was detected close to 2 am. This time, the network engineer investigated upon arriving at 6 am.
• The investigation revealed unexpected traffic between the IT domain controller and the backup system, a common target in ransomware attacks to disable systems and pressure victims to pay ransom.
• The network administrator responded by changing the administrator password, only to find it changed back automatically, indicating full network access by the attacker.
• Realizing the severity, the administrator promptly disconnected the IT network to contain the breach.
• An independent Incident Response Team was called for immediate assistance in identifying the breach scope and guiding remediation.
• The remediation of the IT network took several days. The process involves containment of the cyberattack by isolating affected systems and preventing further access to the network by the attacker. Once contained, recovery efforts can be undertaken with the goal of restoring normal operations and repairing any damage caused by the attack.
• Crucially, due to proactive measures in the OT network—such as the use of a jump box, separate IT/OT domains with domain controllers, and a DMZ firewall—the cyberattack was contained within the IT network. This ensured uninterrupted manufacturing operations while the IT network was restored.

These measures prevented the attack from spreading to critical OT infrastructure, highlighting the effectiveness of preemptive cybersecurity strategies in maintaining operational continuity during security incidents.

Worst Case Scenario: What Could Have Been

Had this facility not prioritized implementing OT networking best practices, the outcome of this cyberattack would likely have been much worse. 

Without the separation of the IT and OT domains and distinct domain controllers, the facility would have been governed by a singular domain. Upon obtaining access to it, the threat actor would have easily been able to traverse the entire network of the facility, including into the OT network. Some consequences within the OT network may have included:

• Deleting rights for some of the servers: the attacker could have altered or deleted legitimate users or administrators of the servers.
• Shutting down HMI servers: halting control and monitoring of the process.
• Control of PLCs: gaining control of the PLC would allow the attacker to directly alter the process.
• Compromising the data servers: in this facility, the batch system was dependent on the data servers. If the attackers had compromised or disabled them, they could have disrupted the batch system and consequently the entire manufacturing process. Note that this dire consequence can be achieved without directly interfacing with the PLCs.
• Use of RDP sessions: had the attackers used RDP sessions to remotely control other servers on the OT network, they would have been able to continue further damage within the system.

Initial Attack Vector

The analysis completed during the remediation process revealed that the threat actor had entered the enterprise network through an incorrectly configured cloud-based server that had access to the enterprise. They had obtained a valid username and password, allowing them to access the network. Once they had access, they began gathering information about the network and its vulnerabilities. This learning phase is known as reconnaissance. It is a very common strategy used by cyber attackers to maximize the amount of damage they deliver to the victim. By maximizing damages, the victim is highly compelled to pay exorbitant ransom fees to reestablish their business. 

The Threat Detection Engine Identified the Threat Actor

In this breach, the network threat detection engine detected traffic that the threat actor was initiating to gather information about the backup system. This traffic, deviating from expected patterns, was flagged as an anomaly by the engine. Without the threat detection engine monitoring the network, the attackers could have remained unnoticed for a much longer period. This delay would have allowed them to gather extensive knowledge about the facility’s operations before executing a full-scale attack.

Conclusion

In conclusion, although this food and beverage manufacturer faced the challenge of a cyberattack, their proactive cybersecurity posture significantly limited the breach’s impact. By adhering to the recommendations from an IT/OT assessment, they ensured the OT network remained secure and operational, preventing any disruption to their critical infrastructure. This case exemplifies the importance of forward-thinking security strategies in protecting vital industrial operations and maintaining business continuity amidst cyber threats.

Below is helpful industry definitions and information to provide additional context and a deeper understanding of complexities of a cybersecurity protection.

What is a domain controller?

A domain controller is a server in a Windows Active Directory domain that manages network security and enforces security policies for a network. It authenticates users, stores their account information, and controls their access to network resources such as files, printers, and applications. Domain controllers play a crucial role in centralized network management, ensuring secure access and efficient administration of user accounts and permissions within an organization’s network infrastructure.

What is a Firewall?

A firewall acts as the first line of defense for a network, effectively creating a barrier between an external untrusted network (such as the internet) and an internal network. It monitors and filters incoming and outgoing network traffic based on pre-determined rules to control what traffic can enter or leave the network.

In comparison to a traditional firewall, a Next Generation Firewall (NGFW) adds advanced monitoring capabilities including:

• Deep packet inspection (DPI): Network traffic is partitioned into portions known as packets. Each packet includes a header and a data (or payload) component. Traditional firewalls typically only monitor the contents of the packet’s header including source and destination IP addresses and port numbers, and protocol type.  NGFWs go further by monitoring the entire contents of the packet, known as deep packet inspection (DPI). DPI allows the firewall to detect and block far more sophisticated threats. One particularly troublesome threat that stands a better likelihood of detection using DPI is known as a zero-day exploit – a type of cyberattack that targets a previously unknown software or hardware vulnerability on the same day it is discovered, thus leaving no time for a vendor to create and release a patch.
• Application Awareness: Unlike traditional firewalls, NGFWs can identify and control specific applications or services that access the network, regardless of the port used. This capability allows it to block potentially harmful traffic originating from risky applications that may be known for spreading malware, or involved in phishing.
• Intrusion prevention: This capability involves real-time detection and prevention of malicious activity. By continuously monitoring network traffic and performing DPI, NGFWs are able to detect anomaly behavior on the network. In addition, they typically access databases identifying specific features of known threats, called signature-based detection, to further identify and stop malicious activity.
• Advanced threat protection: This capability combines the lower-level detection techniques described above with data analytics such as machine learning. It helps the NGFW identify and respond to highly sophisticated cyber-threats in real time.
• SSL/TLS inspection: these protocols encrypt data, providing greater security and privacy. NGFWs decrypt, inspect, and then re-encrypt this traffic, allowing for threat detection that may have been embedded within encrypted communication.
• Integration with threat intelligence services: This integration improves security efficacy by detecting and mitigating threats from known malicious sources.

Overall, NGFWs provide robust network protection and visibility, empowering organizations to secure their networks from a wide range of cyber threats while maintaining control over application usage and network access.  

What is an OT Threat Detection Engine?

An OT Threat detection engine is a specialized tool used in industries like manufacturing and utilities to actively search for and detect potential security threats in Operational Technology (OT) networks. It monitors network traffic for unusual activity, focusing on how OT devices communicate to identify unauthorized access or malicious actions. By leveraging databases of known threats, it enhances its ability to prevent issues before they impact operations. Real-time alerts enable prompt action by teams to safeguard industrial systems, ensuring they operate securely and without disruption.

Unlike a network firewall, which primarily acts as a barrier to prevent unauthorized access, an OT threat detection engine operates deeper within the OT network. It can monitor and analyze network traffic that a firewall may not see, searching for unusual or suspicious activity. While a firewall enforces access rules at the network perimeter, a threat detection engine monitors internal traffic, capable of detecting threats that slip through the firewall or originate internally.

Threat detection engines utilize advanced data analytics and machine learning algorithms to model traffic patterns and effectively identify anomalies. Unlike firewalls that rely on static rules, threat detection engines adapt based on evolving threat intelligence, enhancing their ability to detect emerging threats. They generate alerts and provide detailed information to support the mitigation process during an attack, ensuring proactive defense measures can be implemented swiftly.

What is a Jump box?

In IT/OT networking, a jump box is a secure computer acting as the sole entry point into the isolated OT network. Users must connect to the jump box first before accessing critical infrastructure within the OT network. Access to the jump box is tightly regulated using advanced security measures including multi-factor authentication and strict access controls. Authorized users who successfully authenticate through the jump box are granted access to manage and interact with devices and servers on the OT network. By serving as the exclusive gateway, the jump box minimizes the risk of successful cyberattacks entering the highly sensitive OT network.

Examples of unusual, suspicious transactions between domain controller and back-up system may include:

• Excessive data transfer outside of regularly scheduled backup times. This action may indicate an attempt to exfiltrate data.
• Off-schedule backup attempts. May suggest an unauthorized user attempting to create a backup.
• Access attempts from unexpected, atypical IP addresses. May suggest that an attacker has gained access to administrative credentials and is attempting to access the backup system.
• Backup configuration changes, such as to the destination folder, compression settings, or type of data to store. Suggests that an attempt to manipulate backup data is occurring.
• Unexpected file types such as an executable file, may indicate that there is an attempt to use the backup system to spread malware.

Exfiltration: The transfer of sensitive information from a network

During a ransomware attack, it’s common for data from the system to be exfiltrated. The process of exfiltration involves the transfer of sensitive or confidential information or data off the network to a location under the control of the threat actor. This data breach can have wide reaching impact, including the loss of:

• Proprietary information
• Personal information
• Intellectual property
• Recipes and other trade secrets

IT/OT Assessment: Completed by the Cyber Attacker to Maximize Damages

A modern cyberattack will typically go through distinct phases as the attacker moves from reconnaissance to gaining access and then to full-scale execution of the attack. It may be several months before the actual attack occurs, as the threat actor will spend this time learning everything they can about the company, their operation, their process, and their network structure. During each phase, an attacker may use a variety of network analysis tools to better understand the victim’s facility, associated vulnerabilities, location of sensitive data, and best means of data exfiltration. The learning phase allows an attacker to strategize the best approach to disrupting a facility.