Published 11 May 2023

Abstract

Operational Technologies (OT) and Industrial Control Systems (ICS) have traditionally been managed separately from systems and services that fall within the Information Technology (IT) area. OT and ICS are the systems that control and operate manufacturing plants, control critical infrastructure and increasingly Internet of Things (IoT) sensors and devices. These OT/ICS systems require dedicated security and management support, but also need to connect to overarching security capabilities.

This report focuses on the people, processes, information, and technology to ensure OT/ICS systems are fully protected from all forms of cybersecurity threats. As more machines are connected to networks to support real-time controls, monitoring, and management, there is a corresponding growth in cybersecurity risks.

This report starts with a level set about OT/ICS security then presents an OT security maturity framework to let our customers know where they are and where they may want to move towards. We further break out maturity for the core OT/ICS security categories including basic grade capabilities, standard grade and premium. We also describe some of the unique challenges associated with OT security and describe a framework and strategy for addressing OT/ICS risks.

Reviews of several top vendors in OT/ICS security are provided in this paper along with a strategy to apply the tools in a hybrid on premises and cloud deployment.

Author:

Sorell Slaymaker Principal Consulting Analyst [email protected]

 

Executive Summary

OT/ICS systems are increasingly connected to IT, Internet of Things (IoT), and Industrial Internet of Things (IIoT) systems, making it imperative to secure all systems in an organization’s environment. Malicious organizations commit frequent ransomware attacks and engage in nation-state cyber warfare, making OT/ICS security critical. As more machines are connected to networks to support real-time controls, monitoring, and management, there is a corresponding growth in cybersecurity risks.

OT/ICS platforms are built and deployed with an initial focus on asset discovery, visibility, and network topology. New features are being added that include threat intelligence, vulnerability management, risk scoring and secure remote access.

The proprietary protocols, legacy equipment, and several other factors make OT/ICS security more difficult and requires different solutions than traditional IT security. All enterprises will benefit from a well-thought-out OT strategy. Some notable early industries adopting and building out OT ecosystems include energy and utilities, healthcare, manufacturing, supply chains, and government agencies.

TechVision Research has built an OT security maturity framework to guide enterprises on their journey. This comprehensive framework provides a foundation and roadmap for OT security that is summarized in a single graphic below.

Figure 1: OT Security Framework

As shown in the lower left of the figure, the basic foundation starts by building a static inventory and grows toward using AI/ML for anomaly detection as part of the premium grade (shown in the upper left). Not all machines and environments need the same level of security.

This research creates a framework for enterprises to follow to do the basics well before spending more money for additional tools and people. Security is a journey and, in this case, starts at the bottom left with a solid inventory and then matures to use the latest AI/ML technology to automate more of the SOC functions in near real-time. Reviews of several top vendors in OT/ICS security are provided in this paper along with a strategy to apply the tools in a hybrid on premises and cloud deployment.

Introduction

Operational Technologies (OT) involve machines that move or move objects through manufacturing, hospital, utility, and other critical infrastructures. Securing these machines has its own nuances that differ from the standard IT security of people, common devices (i.e. personal computers or smart phones), or the Internet of Things (IoT). Many OT systems have not been patched for years. The cost of updating these systems is a risk some businesses may not be willing to accept provided the risk is appropriately managed.

OT systems control and monitor physical processes such as Industrial Control Systems (ICS) and Supervisory Control and Data Acquisition (SCADA) systems. These systems include sensors, Programmable Logic Controllers (PLCs), and other devices that interact with physical equipment. For OT cybersecurity practitioners, the focus is on the reliability, productivity, and safety of these systems and their environments.

As more machines are connected to networks to support real-time controls, monitoring, and management, there is a corresponding growth in cybersecurity risks. These risks include malware and malicious actors working on behalf of organized crime syndicates and foreign adversaries. More and more manufacturing, hospital, and utility enterprises are reporting ransomware, vandalism, and cybersecurity breaches that can damage critical infrastructure and lead to the injury or death of people.

OT/ICS Cybersecurity incidents cost U.S. firms currently over $100M for a single event. The number of cybersecurity incidents continues to grow as malware targeting OT increases and there is an accelerating proliferation of connected devices. Many enterprises lack the security controls and management in OT as they have in IT and, as we’ve discussed, IT and OT often remain separate environments.

Machines span the gamut of technology with some being new and complex while others are old with proprietary operating systems and command line interfaces. In many cases, there is also an IoT (Internet of Things) component to OT security for those devices and sensors that are directly connected and used to support machines. The TechVision Research report “The Identity of Things (IDoT)” can provide additional background information.

Most enterprises benefit from a well-thought-out OT strategy. Some notable early industries adopting and building out OT ecosystems include energy and utilities, healthcare, manufacturing, supply chains, and government agencies. As with any type of environment, one is only as secure as the weakest link, which can be compromised and used to target higher value machines.

While every industry has a wide range of use cases, enterprises have an opportunity to build an inventory of OT machines and provide additional security controls, processes, and tools. For instance, a steel plant needs to ensure the furnace that heats the metal is kept at a very specific temperature and pours the hot steel into modes in an exact way. Any variation needs to be closely monitored. Compromising these machines will not only hurt the bottom line but potentially put lives at risk.

Many organizations are under the false assumption that if their machines are not connected to the Internet, they are safe. This has proven not to be true time and time again. Operators and devices access these machines to provide monitoring, maintenance, and management. Both operators and devices can be compromised and result in disastrous outcomes. One well publicized example were the nuclear centrifuges that Iran was using to enrich uranium.

Besides organizational security, there are several new laws, government mandates, and best practices being issued to protect critical infrastructure. One example is the US government Cybersecurity and Infrastructure Security Agency (CISA). Another example is the International Society of Automation (ISA) and the International Electro-technical Commission (IEC) which created 62443 standards for security of industrial automation.

This research paper focuses on the people, processes, information, and technology to ensure OT/ICS systems are fully protected from all forms of cybersecurity threats. Security is a journey and enterprises should not only understand where they are at, but also plan for continued maturation. The amount of effort and the costs in security and automation tools is high, but the risks of not doing it are higher.

We have built an OT security maturity framework to help guide enterprises on their journey. This comprehensive framework provides a foundation and roadmap for OT security that is summarized in a single graphic. As always, TechVision’s goal is to provide a set of pragmatic recommendations and action plans that enterprises can directly leverage.

Connecting OT Systems to IT

More and more machines are being fully connected to the enterprise network driven by the desire to improve enterprise efficiencies. Proactive maintenance, just-in-time inventory management, Enterprise Resource Planning (ERP) applications, and additional sensors supporting OT machines also result in greater connectivity.

When there is no network connectivity in/out of a network that a system is connected to, it is considered to be air-gapped. However, it is a fallacy to assume air-gapped systems are more secure. Infected devices may be added to the air-gapped network or there may be an unknown back door network in place. One electrical utility that was breached by hackers was not worried because the utility did not have any Internet connectivity for their power distribution center. Unbeknownst to them, their connected maintenance provider had a connection to the Internet.

As OT systems get connected, they need to be secured. For examples of notable OT breaches, click here[1]. The attack surface is expanding as more OT operations and management are moving into the cloud. Today, manufacturers collect data in real time and use it to run analytics in the cloud for immediate results. Applications and data-generating activities, once reserved for local operations and warehouse management systems for logistics, are finding their way to the cloud as well, increasing OT network complexity. Securing OT technology has its own challenges that are different from IT.

Unique Challenges of OT Security

Many IT security experts approach OT security the same way they approach IT security. However, OT security has several idiosyncrasies that make it a challenge. These challenges include:

• Old Machines – Many OT systems are decades old and have outdated operating systems (OS) if they use a standard OS at all. The cost of updating the machines is not worth the risk, so the network must be air-gapped or protected upstream at the physical site. Many of these machines only have command line access and making simple changes like an IP address challenging.
• Non-Standard Operating Systems– Many industrial machines, sensors, and systems do not utilize Microsoft Windows, Apple IOS, Unix/Linux, or other standard operating systems. This means that the endpoints are missing end point security software to identify and mitigate viruses and malware.
• Non–Standard Protocols – Unlike IT, where all traffic runs on IP, OT has over 200 vendor proprietary protocols, with many of them running on top of layer 2 Ethernet for wired and wireless connectivity. An example list of OT protocols can be found here[2].
• Lack of Homogenous Ownership and Control – Many hospitals and other high-tech facilities buy equipment that is maintained by the manufacturer. While the enterprise can use the equipment, how it works and is supported is purely up to the manufacturer. In many cases a separate ISP and/or 4/5G network is used to remotely access and support the system.
• No Centralized Directory – Many of the users and devices are not in a corporate directory unlike in IT where all users and devices are in a directory such as Microsoft’s Active Directory (AD).
• Unique Attacks – There is malware that is being designed specifically for OT devices and the signatures used to identify it are different than what one sees with standard Operating Systems. Palo Alto, Fortinet, and other firewall vendors sell different security signature feeds for OT devices.
• Use of Removable Drives – Thumb drives and other removable media offer a back door to OT systems where malware can be injected and not detected. Organizations may have policies against the use of removable drives, but those who operate the OT systems find that removable drives provide an easier way to get data or apply a patch.
• System Within Systems – Poor Security of Sub-Components – Many large OT systems may have thousands of processors used internally and most of the communication is within. The enterprise does not have any visibility to the inter-workings of the system.
• Technical Debt – OT security was not a priority for many decades. As a result, many machines do not have passwords. Some may use a default password, and, in some cases, the password is sent on the network as clear text.
• Downtime is Expensive – In critical environments such as healthcare or public infrastructure, downtime is extremely expensive. Scheduled maintenance is infrequent and doing network scanning can cause machines to freeze up.
• Different Risk Profile – While IT is often consumed by privacy and data breach concerns, the thing that keeps OT operators up at night is disruption or malfunction of systems that could threaten the business or even people’s safety.
• The Rapid Proliferation of Sensors and IoT Devices – This increases complexity, the scale and the number of attack surfaces.

With millions of OT systems and their uniqueness, it is difficult for enterprises to develop secure best practice guidelines on OT system set-up and support similar to what is done in IT with Windows and Linux servers. For this reason, TechVision has created an OT security framework to help address OT/ICS security challenges in a standard way.

The OT/ICS Security Maturity Framework Overview

Below is a diagram of the OT security framework that provides structure to the level of breadth and depth of OT security.

Figure 2: TechVision’s OT Security Maturity Framework Overview

The capabilities in this framework are supported by processes and policies at the level of enterprise security as described in our “Multicloud Cybersecurity Reference Architecture” report. The technology capabilities in this diagram may be provided through a combination of dedicated tools within the OT/ICS environment or they may be part of the enterprise security environment. That is, there may be embedded network security capabilities within OT/ICS, but UTM or SIEM capabilities may be provided at the enterprise level and therefore OT/ICS practitioners must ensure that the enterprise tools meet OT/ICS security requirements or integrate well with embedded OT/ICS security capabilities.

The left side depicts the levels of security and is shaped as a pyramid to represent the number of devices in each level. The level of security maturity is dependent on the assessed risk of the devices being protected.

• Basic Grade – This grade includes OT machines that are not of high value and/or do not provide much risk if they are compromised. Basic security measures are taken to protect them and to ensure that they are not used to target higher value assets.  This is where enterprise should start in providing minimally viable OT security.
• Standard Grade – As the name suggests, this is the level of security that all machines should have. If an enterprise does not have this level of security, then there should be a roadmap in place to get there. Like with all frameworks in security, many of the premium security features will move down into standard grade as cybersecurity risks grow.
• Premium Grade – High value and/or high-risk assets should strive to achieve the best security processes, controls, and tools. Unfortunately, there is not a single product that will provide everything an enterprise needs. Even within the premium grade, there are different sub-tiers.

Across the top of the framework are the six characteristics of OT security and the level for each tier. A characteristic represents a single category of security processes, controls, and tools.  Together, all six categories work in concert to provide the end-to-end solution required to secure machines. These characteristics can also be found in IT and IoT security, but the details and nuances of this framework are specific to OT security.

• Inventory – Identifying and classifying all OT machines and their directly associated devices is critical to set up the inventory. This starts at the basic grade with a static inventory which is a snapshot in time of what is in place. The standard grade is dynamic and all changes in machines and their configurations are tracked in real-time. Premium grade ensures Least Privileged Access that defines which other systems a machine is allowed to talk to by name, IP address, and TCP/UDP or other protocol. This whitelist of access is then monitored, and any exceptions or anomalies are reported.
• Network – Segmenting OT traffic onto its own network is a critical first step in OT security. At the basic grade, this starts with putting all OT machines and directly used devices onto their own network and away from IT, IoT, Guest, or other networks. The standard grade provides additional north-south micro-segmentation in the OT segment following the Purdue reference model. At the premium grade, east-west segmentation in a Zero Trust Security model is added as described in our “Zero Trust Networking” report.
• UTM – Unified Threat Management (UTM) provides security controls between network segments. At the basic grade, using a stateful firewall to control traffic based on IP address, TCP/UDP port, and other protocol mechanisms allows permitted communications and blocks everything else. At the standard grade, Intrusion Prevention Systems (IPS) are added to continuously inspect all network traffic and look for malicious activity based on IP address, signature, or pattern, and block it when it occurs. At the premium grade, having a proxy gateway for all communication and leveraging Data Loss Prevention (DLP) software will ensure no private or confidential information is permitted outside of the OT segment.
• SPAN – Switch Port Analyzer (SPAN), also known as Ethernet switch port mirror, copies all the network packets passing through an Ethernet switch to a designated SPAN port where it is analyzed by a 3^rd party tool. A passive SPAN means that the tool is just monitoring, whereas an active SPAN means a monitoring tool will query the device to find out more information about it including the operating system details, type and version of protocols, and industrial control specifics. At the basic grade, this is only passive monitoring of traffic between the switches (Uplinks) on the site to the Main Distribution Frame (MDF) where the security appliances sit. This is the North-South (N-S) traffic. At the standard grade, SPAN the traffic within every switch and capture all East-West (E-W) traffic. At the premium grade, provide active spanning as described above.
• IAM – Identity and Access Management (IAM) is the foundation of IT security. At the basic grade, identity and access control is based on the Media Access Control (MAC) address. The MAC address is a unique 12-character alphanumeric assigned to every connected device. At the standard grade, Network Admission Control (NAC) is used to provide IAM directory-based Authentication, Authorization, and Accounting (AAA) access to a network. At the premium grade, Multi-Factor Authentication (MFA) is added based on up to six attributes which are described later in this paper.
• SIEM – Security Incident and Event Management (SIEM) starts at the basic grade with storing logs, including a syslog, in a separate and secured system and storing them for 90 days in case an event occurs. At the standard grade, event identification and classification are added to minimize false positives tied to planned changes or other approved activities. The premium grade represents the “holy grail” by establishing a baseline of data collected across all characteristics and tiers and leveraging Artificial Intelligence and Machine Learning (AI/ML) to provide anomaly detection.

Implementing this full stack of OT security can be challenging. It takes multiple products that must be integrated together and many of the technologies are proprietary and vary across vendors. With the security landscape under continued threats, there will be additional security controls in the future. In the next sections we will go into further depth on each character and tier.

Inventory

As with the NIST Cybersecurity framework, the first step is to identify everything in the environment. While this may seem obvious, real-world execution is difficult. Most enterprises do not have an up-to-date inventory of all the devices connected to the network at a given site. The asset inventory is a start, but a lot of additional information needs to be gathered.

Classifying OT Machines/Systems

Determining what is OT vs. IT can be tricky. IT refers to the use of computers for information processing of data and includes things like servers, networking devices, and common endpoint devices. OT encompasses the hardware and machines responsible for the physical processes of a given business and includes industrial computing equipment like control systems.

Many organizations will classify OT systems as anything in a specific location such as a plant floor, a hospital operating room, or an electrical sub-station. Other organizations will first classify IT systems tied directly to a user or application and then classify everything else as OT. Approximately 90% of classifications are based on common sense and are easy, but the last 10% can be tricky. For instance, a timecard machine could be an IT or OT device. If it is in a directory like Active Directory (AD) and has a common OS, one could argue that it is an IT device. On the other hand, if it sits on the plant floor and is part of day-to-day operations, one could argue that it is an OT device.

Differentiating between OT and IoT devices is also complicated. Are surveillance cameras used for site security or monitoring outputs OT or IoT? In a crawl, walk, run methodology, enterprises are encouraged to do a good job of OT and IT security and include IoT with OT before further separating these functions.

Creating a Standard Naming Schema

The first step is to develop a common naming schema for every OT device. Many systems allow for a name to be up to 40 characters long and, due to system compatibility, use dashes between fields instead of periods. The goal of a standard naming schema is to uniquely identify every OT system, its location, and the type of device it is. Many enterprises will use a format like the example shown in figure 3.

• Region – 2-digit code for a world region such as NA for North America
• Country – 2-digit country code such as US for United States
• Branch – 4* digits for the branch number – most enterprises have site IDs
• Building – 8* digits for the building number, floor number, room number
• Device – 10* digits for the device description summary –
  • First 4 digits for the company – One can use the stock sticker symbol.
  • Next 6 digits for product/device/system description – There should be a glossary that provides a greater description.
• Seq Number – 4* digits for the sequential number of devices of that specific type

 

* (the number of digits can be smaller or larger depending on the size of the organization and/or buildings but not to exceed an overall total of 40 characters including dashes)

Figure 3: Example Standard OT Naming Schema

An example would look like this:  AP-KR-1234-0204IDF1-CSCOSwitch-0002 where the second Cisco Ethernet switch is in building 2 on floor 4 in the IDF1 closet at site 1234 in South Korea. Following the naming standard is critical and helps when searching the inventory database. Once an enterprise has the naming schema defined, the next step is to build the inventory.

Basic Grade—Static Inventory

Unfortunately, in most enterprises, inventory information is spread across many disparate applications such as asset management, site operations, and IT Service Management (ITSM).  Using tools for discovery is an option, but this will provide only some of the information that is required. Later in this report we will review some of the more popular tools used to scan and identify all connected machines and devices.

Network scanning in OT/ICS environments can have unintended consequences, such as systems freezing up during the scan or even worse, going offline. The best way to discover OT assets is to start with passive monitoring of the network by spanning the network traffic. The next step is to intelligently communicate with identified machines on their native protocol(s), one by one.

Many enterprises start their OT inventory by building a spreadsheet or a simple database to track all their OT assets and importing information from other systems and/or manually entering important information. Once a device is given a name as described above, the following information needs to be gathered.

• Device Details – Description, manufacture, product SKU, maintenance contact, website link
• Asset Details – When it was purchased, how much, maintenance, serial number, and location
• Business Details – Function, criticality, lifecycle maintenance
• IT Details – MAC address, VLAN, IP address or network, Operating System, OS Version, administrator contact, whether backups are done and if so, where they are sent
• Access Details – Who has access to the device and what levels of access, if applicable, login credentials, local and remote access
• Recovery Details – If the system fails, machine process and contacts to get it up and working

After the above data is collected, OT machines should be classified, and their vulnerabilities identified. As part of the vulnerability management process, Vulnerability Assessment (VA) identifies and prioritizes vulnerabilities in devices, networks, and software. With an asset inventory, security risk managers identify CVEs (Common Vulnerabilities and Exposures).

Once the vulnerabilities are identified and prioritized, organizations can take steps to address them, such as patching or mitigation through security controls like encryption or access controls. Additionally, vulnerability management also includes periodic scans of networks to detect new threats or changes in existing vulnerabilities to ensure proper security posture is maintained over time.

The National Vulnerability Database (NVD), which is maintained by National Institute of Standards and Technology (NIST), is currently a good source for tracking CVEs. An example of a CVE on the NVD website can be seen in Figure 4.

Figure 4: Example of a CVE in the NVD

The first step of any OT security assessment involves the maturing of inventory management. You can’t manage or secure what you don’t know about and this inventory discovery, classification and assessment process is critical. Once this data has been compiled in a structural way, IT can provide some pragmatic steps to further secure the environment from cybersecurity risks without forcing the business to pay for major machine upgrades. The next sections of this report will provide further details about this process.

Standard Grade—Active Inventory

Enterprises should embrace Operational Technology Systems Management (OTSM) in a way that is similar to how many organizations have handled ITSM. Achieving a mature level of OTSM is not only critical in terms of improving overall ROI from increasingly connected industrial systems, but also in ensuring the foundational elements of OT cybersecurity necessary to protect critical infrastructure from targeted attacks.

Traditionally, industrial controls systems have been seen as long-term capital investments that will last 15-20 years between major upgrades. OTSM requires regular management including updates, configuration management, access management, and vulnerability management, among other activities. In many cases, this requires changes to the mindsets and behaviors of team members as well as the more functional training and procedural requirements. Senior leadership is key to making this change effective within already stretched operational organizations.

A good OTSM solution tracks and manages all changes in the OT environment including adding and removing machines and sensors and all the maintenance and upgrades. Bringing IT and Cybersecurity change control into an OT environment can be a cultural challenge. A good IT change management process following ITIL requires a lot of upfront planning and documentation along with formal management approvals. Many OT environments do not have this level of rigor, especially when upgrading the computing/OS side of the machine.

Premium Grade—Least Privileged Access

Least Privileged Access (LPA) requires that one understands exactly what a machine should communicate with, when, and how. This is required in highly secure environments that have very valuable assets and/or are vulnerable to cybersecurity attacks. A whitelist is created that specifies exactly what a machine is allowed to talk with. All other communications are monitored and dropped if it is not on the whitelist. An alert is generated to correlate with other alerts to determine the level of vulnerability or risk, whether a cybersecurity event should be launched, and the associated severity.

A monitoring tool is required to create a baseline of the devices and systems with which a machine talks. Any anomalies should generate an event and be reported to a SIEM for further analysis and correlation with other events. This is also a good way to ensure that all change management is planned and approved from the standard grade. LPA is tied to Zero Trust Networking (ZTN) which is covered in the next section as well as in our report of the same name.

Network

Besides underfunding and not paying attention to OT security, many enterprises have not updated their Local Area Network (LAN) infrastructure in over a decade. In this section, we will discuss the levels of network segmentation required. This is a topic also covered in our report “Zero Trust Networking”. Investing in a LAN upgrade can cost millions of dollars per site in hardware/software, implementation, tools, and in many cases additional fiber runs. This investment is a foundational one to ensure OT networks can be isolated, controlled, and monitored.

OT environments tend to use flat networks and equipment from multiple vendors. Yet deploying network-level segmentation with physical devices can mean significant downtime—especially if systems are too outdated. Moreover, most OT professionals don’t know IT best practices or advanced network security concepts, and their job is to prioritize uptime and people safety, not security.

Ethernet is still the backbone of most factories and warehouses, but wireless connectivity is gaining traction as private cellular brings mobility, reliability, deterministic networking, and standardized technology. This will enable autonomous mobile robots, asset tracking, smart glasses, and other Industry 4.0 applications that are rapidly proliferating.

Basic Grade—OT Segmentation

The first step at a site with OT machines is to segment OT from IT.  This is important because many times malware or malicious people will infiltrate on the IT side and then move over to the OT side where there are higher value targets. The second reason is once OT machines are on their own network, specific networking security controls and monitoring can be applied such as those covered in this research paper.

One question that comes up often is whether segmentation should be physical, whereby separate Ethernet switches are used for the OT network, or if segmentation should be logical using separate VLANs on the same physical Ethernet switches. In most cases, logical separation is good enough and the cost for physical separation is not worth it. Logical separation offers great security with the primary vulnerability being misconfiguration allowing traffic to bridge VLANs without the proper security controls. While misconfiguration of systems is one of the top causes of vulnerabilities, this problem is better solved through a comprehensive change management process.

Another question is how many VLANs should a typical OT site have and what are these VLANs.   At the basic level many enterprises find 12 VLANs adequate to meet their current needs. Figure 5 is an example of the number, type, and controls of VLANs at a typical OT site.

 

Figure 5: Example of VLANs Used at an OT Facility

Each of the VLANs shown in Figure 5 must go through a firewall to talk to another VLAN. Controls are based on IP addresses and TCP/UDP ports. A zone-based firewall is used for inter-VLAN routing. The 12 different VLANs in this example include:

• Management – Privileged Access Management (PAM) network with access allowed only for system administrators of routers, switches, firewalls, servers, and other IT systems.
• Servers – The IT computing and storage used to run a manufacturing, hospital, or other type of OT site.
• Voice – IP based phone system to ensure high network performance and keep this traffic isolated.
• Users – Authenticated users and their devices within the site.
• Video – Surveillance and monitoring video which can consume vast amounts of network bandwidth.
• OT – Operational technology and everything in the Purdue models 0-3 (The Purdue model is discussed more in detail later in this report.)
• OT DMZ – The IT systems used to support the OT environment including patch management along with proxies to control and monitor traffic in/out of the OT environment.
• OT Support – Remote access including 3^rd party to the OT environment. (This remote access should be through a Zero Trust VPN which will allow a 3^rd party one-time access from their specific device to only the OT devices during a change/maintenance period.)
• IoT – Consumer and non-OT related IoT devices, for instance facility monitoring.
• Industrial IoT – Industrial Internet of Things (IIoT) is the use of smart sensors and actuators to enhance manufacturing and industrial processes.
• Guest – Users who are guests to the facility and need Internet access, with all IT or OT access going through a VPN.
• Vending – 3^rd party devices that need Internet access but are not part of the OT systems at the facility, such as a vending machine.

These 12 VLANs are a common example, and many mature facilities may have a few more (or less) VLANs. The different color codes shown in Figure 5 represent the different levels of access.  The VLANs span across both the physical and Wireless LAN (WLAN). Many enterprises will use the same vendor for LAN and WLAN management so that access controls and monitoring are the same. Next, we’ll progress to standard grade capabilities.

Standard Grade—Micro-Segmentation

Micro-segmentation refers to creating North-South segmentation between various functions in a facility. The most common form of this segmentation is the Purdue model as shown in figure 6.

The Purdue model, part of the Purdue Enterprise Reference Architecture (PERA), was designed as a reference model for data flows in computer-integrated manufacturing (CIM) where a plant’s processes are automated. It came to define the standard for building an OT/ICS network architecture in a way that supports OT security, separating the layers of the network to maintain a hierarchical flow of data between them.

Figure 6: Purdue Model for OT Security[3]

The model shows how the typical elements of an OT/ICS architecture interconnect, dividing them into six zones that contain IT and OT systems. The goal is to separate ICS/OT and IT systems so an enterprise can enforce effective access controls without hindering business.  The Purdue model has six levels/zones:

Level 4/5: Enterprise Zone – These 2 zones house the IT network where the primary business functions occur, including the orchestration of site operations. In manufacturing, for instance, Enterprise resource planning (ERP) systems will drive plant production schedules, material use, shipping, and inventory levels.

Level 3.5: Demilitarized Zone (DMZ) – This zone includes security systems such as firewalls and proxies that are used to prevent lateral threat movement between IT and OT. The rise of automation has increased the need for bidirectional data flows between OT and IT systems.

Level 3: Manufacturing Operations Systems Zone – This zone contains customized OT devices that manage production workflows at the site. For instance, at a manufacturing site:

• Manufacturing operations management (MOM) systems manage production operations.
• Manufacturing execution systems (MES) collect real-time data to help optimize production.
• Data historians store process data and (in modern solutions) perform contextual analysis.

Level 2: Control Systems Zone – This zone contains systems that supervise, monitor, and control physical processes. Examples at a manufacturing site include:

• Supervisory control and data acquisition (SCADA) software oversees and controls physical processes, locally or remotely, and aggregates data to send to historians.
• Distributed control systems (DCS) perform SCADA functions but are usually deployed locally.
• Human-machine interfaces (HMIs) connect to DCS and PLCs to allow for basic controls and monitoring.

Level 1: Intelligent Devices Zone – This zone contains instruments that send commands to the devices at Level 0. In a manufacturing environment this can include:

• Programmable logic controllers (PLCs) monitor automated or human input in industrial processes and make output adjustments accordingly.
• Remote terminal units (RTUs) connect hardware in Level 0 to systems in Level 2.

Level 0: Physical Process Zone – This zone contains sensors, actuators, and machines that are the core to the OT environment.

While the Purdue model was introduced in 1992 by Theodore J. Williams and the Purdue University Consortium, few other models have yet outlined a clear information hierarchy for CIM, which began to take hold of the industry in the mid-to-late 1980s.  Many organizations will use a zone-based firewall to separate the levels versus having a separate physical firewall going in/out of the DMZ.

Mobile devices can make purely enforcing the model a challenge such as engineer laptops which may plug into the IT network than into the OT network. It is highly recommended that these mobile devices have end point protection software.

Premium Grade—Zero Trust

As we advance to Premium Grade capabilities, we move to Zero Trust. Zero Trust Networking (ZTN) segmentation refers to creating East-West segmentation between various functions in a facility. Just like in a data center where there are many database servers within the data layer, each database server is isolated from the others so that if one is compromised, the others are not vulnerable.

In a manufacturing plant, this may include isolating each cell with its own private IP address and using Network Address Translation (NAT) to connect it to layers 3 or 4. Having to re-IP address a site can cost millions of dollars, so many organizations will shoehorn a legacy IP addressing schema into a new site that is doing ZTN. NATing between network segments has its challenges but is quite common. For more information on Zero Trust Networking, see our report.

Many large facilities are adding public and/or private 4/5G cellular networks for support machines and devices. The advantages of 5G networking over 802.11 based WLANs are greater coverage and lower power requirements for endpoints. Citizens Broadband Radio Service (CBRS) frequency band in the United States enables organizations to use the 3.5 GHz to 3.7 GHz radio spectrum to build wireless networks based on 4G LTE and 5G cellular technologies. Other radio spectrums can be used, but there are licensing and power restrictions to them.

Network coverage of a large facility is challenging and costly. Many plants today installed multi-mode fiber decades ago to support 100M and 1G fiber links. With 4K video and other high bandwidth applications, 10 & 100G uplinks between buildings is required, which requires spending millions on installing new single mode fiber connections.

Roaming users can also have challenges in losing cellular and or WLAN coverage with the facility due to steel ceilings, thick concrete walls, and high electromagnetic fields. Additional access points and/or repeaters will need to be added. Many enterprises also have traditional copper phone lines that can be used for emergency communications.

UTM

It is important to control security between a facility’s internal VLANs and network connectivity in and out of the site via the Wide Area Network (WAN). These days with Software Defined WANs (SD-WAN), most sites have both Internet offload—where traffic can be sent from the facility to anywhere on the Internet, including cloud services—and SD-WAN tunnels that connect to other sites within the organization, including data centers. For more information on SD-WANs, see our research report “The Path to SD-WAN”.

Basic Grade—Firewall

A firewall controls what users, devices, and machines are allowed to talk to each other and which services and applications they may use to communicate. At the basic level, the controls are based on names such as Domain Naming Standard (DNS), IP address, and/or TCP/UDP port. The goal is to minimize the attack surface by restricting the number of devices that can access a target along with controlling ports/protocols, who can initiate/establish a session, and whether the session is TLS encrypted. Table 1 below represents how the various VLAN and WAN links—or in firewall terms, zones—are controlled.

Table 1: Firewall Zone Rule Example

The top row of the table represents something within that zone initiating a session to another zone represented on the left. The zones on the left are initiating a session to a zone on the top. The color codes represent:

• Green – Full access which means all users and ports.
• Yellow – Filtered access based on DNS, IP, and/or TCP/UDP port.
• Red – No access is granted under any condition.

Some examples to call out include:

• OT – The OT zone is only allowed (permitted) to initiate sessions to the OT DMZ zone. For instance, if a historian server that records a machine’s output for reporting purposes is in the server zone, it must proxy its connection through the OT DMZ to get to the machine.
• IoT – Local IoT devices are only allowed to talk to a local server such as a server hosting a building HAVC management application.
• Servers – Controlling what servers with key applications and data have access to is critical for every organization. There are shared services such as NTP, DNS, AD, Logging, backups, and others to which every server needs access. Beyond that, access should be restricted only to a whitelist of other systems.
• Guest – Guests are allowed to go out to the Internet and are filtered based on approved DNS sites, blocked sites or categories, ports such as HTTPS, and IP addresses with good reputations. The guest network is not allowed to talk with any other zone.

Large enterprises will utilize a tool to manage these rules which can number over 1,000 per facility. Algosec, Skybox, Solarwinds, and Tufin are a sample of vendors who manage these rules across leading firewall vendors such as Palo Alto and Fortinet. These tools also automate rule changes and create reports for audits to ensure industry compliance. Once an organization has tackled this process, the next step is to ensure the firewall can inspect all TLS encrypted traffic.

Standard Grade—IPS

An intrusion prevention system (IPS) is a network security solution that continuously monitors a network for malicious activity and takes action to prevent it, including reporting, blocking, or dropping it when it does occur. Next generation firewalls provide full Unified Threat Management (UTM) solutions which include IPS functions. An IPS has multiple functions including:

• Decryption – All TLS traffic is decrypted and inspected. The firewall/IPS sits in the middle of the session and decrypts the traffic, inspects it, then encrypts it back using a certificate that is managed by the enterprise.
• Inspection – It is important to compare session behavior, patterns, signatures, and source/destination IPs against known vulnerabilities. The list of signatures and trusted public IPs should be continuously updated. The top firewall vendors have their own security feeds along with feeds from the government including from the U.S. Government CyberSecurity and Infrastructure Security Agency (CISA) which can be found on the web at cisa.gov.
• Prevention – Suspicious traffic is monitored or stopped once it is detected. Many IPSs come with a sandbox where suspicious attachments are executed to see if they are truly malicious.

The top firewall vendors also have signature OT-specific feeds for which they charge an extra licensing fee. While these are great additional features, providing full UTM requires a lot more hardware (CPUs and Memory) to perform these tasks while maintaining traffic speed and avoiding jitter. There typically is a 100x hit when going from firewalling to UTM between LAN segments.   Most enterprises do not decrypt and inspect video traffic because of the resources required to do so.

One of the most common problems with an IPS is the detection of false positives or false negatives, which occurs when the system blocks an activity on the network because it is suspicious and thus assumes it is malicious, causing denial of service to a valid user trying to do a valid procedure. IPSs take a lot of care and feeding to ensure that they are working properly and not impacting valid traffic that keeps the enterprise running.

Many enterprises will use one set or service of firewalls for communication in and out of the site and a second set for security within the plant. This is being done for organizational reasons with one team being responsible for the WAN and another for the LAN. DLP is more often done just on the WAN side as an organization wants to ensure that specific types of data are not leaking out.

Premium Grade—Proxy and DLP

A proxy server acts as an intermediary between a client requesting a resource and the server providing a resource. Instead of connecting directly to a server, the client is directed to a proxy server, which evaluates the request and performs the required action. This serves as a method to control requests and provide load balancing, privacy, or security functions. A reverse proxy works in the opposite direction such that it appears to clients to be an ordinary server.

Modern OT/ICS systems that use APIs to talk to other systems should go through an API proxy similar to IT APIs. For instance, SCADA instrumentation will allow one to see what transactions are taking place, what data are out there, and if the transactions are problematic. Having an additional point of data and defining thresholds that shouldn’t be crossed will allow for an environment to operate safely.

Data loss prevention seeks to prevent the leakage, corruption, or the deletion of confidential or proprietary data systems. DLP is most associated with compliance and privacy and ensures confidential and private data does not leave an organization. If it does, it is masked, or other security measures are in place. What does DLP have to do with OT/ICS security? In this case, it has a lot to do with data integrity. Data integrity refers to information property that has not been altered or modified by an unauthorized person.

In many cases, the ability to modify the data is worth more than the ability to steal the data. For example, if a sewer plant monitors both toxins coming in and out of it, and this data is modified, it is bad for the environment, and someone is getting away with illegal dumping of hazardous waste.

Data quality includes examining data accuracy, consistency, completeness, and relevance. Data integrity refers to the accuracy, reliability, and consistency of data over its life cycle. In many cases, sensors start going bad and over time their accuracy declines. Manual or other types of testing is required to ensure all data is accurate and is not prone to being modified.

SPAN

The best way to secure older OT/ICS equipment is to monitor, capture, and analyze all network traffic. Products that do this work are covered later in this report. The advantages of utilizing an OT monitoring solution are:

Visibility – Ability to discover the assets, assess their associated vulnerability, and track the changes to the environment as machines are added, removed, or changed.

Detection – To quickly pinpoint security threats and any anomalies.

Response – To accelerate remediation efforts and minimize risks and downtime.

Logging – Ability to continuously monitor what is happening and create the equivalent of a syslog for older OT systems that do not have a logging function.

It is critical for organizations to start with passive monitoring of OT/ICS systems. Many industrial systems operate 24/7/365 and involve processes with significant safety risks. Classic IT cyber security tools that generate significant network traffic are not suitable due to the risk of industrial device failure and/or latency which can disrupt the industrial processes.

OT/ICS systems use many protocols that are unknown in the IT world, and these protocols are inherently insecure. Analyzing communications using these protocols for security threats requires specialized evaluation techniques and must be done at a very detailed level. To start, passive north-south monitoring of the OT/ICS environment is recommended.

Basic Grade—Passive N-S

North-South monitoring spans all traffic moving through the MDF switch, which is inter-VLAN traffic, or traffic leaving the site via the Internet or WAN. To start, a virtual server should be put into place to run security tools for OT monitoring and scanning. Incident Rapid Response (IRR) software should be in place in case a machine is compromised. Figure 7 depicts a typical OT environment. The bottom section represents the machines being monitored and their connectivity starting at the cell switch and then going up through the IDF switch to the MDF switch for inter-VLAN traffic or traffic going offsite.

Figure 7: Monitoring OT/ICS Traffic via Network SPAN & RSPAN

The OT monitoring solution will see all the network traffic via the SPAN and will summarize this information into meta data on every device and session. The summary is sent to a centralized management server for analysis. A variety of situations are detected including:

• Asset details with known vulnerabilities.
• No password, or cleartext, default, or weak passwords.
• Machine anomalies such as state changed, uses ports, trying to talk with something it has not talked with before.
• Bad configurations (NTP/DNS/DHCP, etc.) or IP address conflicts.
• Unencrypted communications such as Telnet, FTP, HTTP.

The amount of data collected is impressive. The secret is then being able to take the data and turn it into actionable information. Once this is accomplished, an enterprise can go deeper into the environment by monitoring the traffic within an IDF and cells.

Standard Grade—Passive E-W

To truly see all OT/ICS traffic, the East-West traffic has to be monitored. This involves Remote SPAN (RSPAN) of traffic from all the ethernet ports at a facility. RSPAN allows traffic that is sourced from a switch to be mirrored to a remote switch within a layer 2 network over trunk ports. To make this happen, the destination VLAN must be configured across the entire path between the switches. In Figure 7, the cell and IDF switch ports are RSPANed up to the MDF switch into the security server that runs the OT monitoring software.

To RSPAN the traffic from all the ethernet ports at a facility requires an Ethernet switch upgrade throughout the site which can be expensive. RSPAN of all traffic may require upgrading the fiber plant to be able to support 10 & 100Gbps Ethernet up-links. Also, Quality of Service (QoS) should be implemented to ensure the RSPAN data does not overrun the uplinks.

Most of the traffic within a cell is based on a proprietary protocol. One of the big differentiators between OT/ICS monitoring tools is how many different protocols they understand. The protocols vary by both vendor and industrial vertical such as power distribution or healthcare treatment. One example is Siemens with some of their protocols that include APOGEE, DIGSI, Sinec H1, S7, S7 Plus, and CAMP.

Premium Grade—Active

In active monitoring, the OT monitoring software sends packets to, and talks with, an OT device. This needs to be done with caution to avoid impacting the machine while allowing active communication. Additional and useful information can be obtained as shown in Table 2 below.

Table 2: Comparing Passive & Active Monitoring

As OT security solutions continue to mature, the top OT/ICS asset discovery and monitoring solutions now blend elements of both active and passive technologies to maximize visibility into the ICS environment and enable OT security teams to deploy the right approach for each network segment. The best of these solutions also incorporates fail-safe technologies to reduce the risk of disruption. Examples include reducing the risk of an endpoint malfunction by passively monitoring the network and mapping which firmware versions and communication protocols are present before sending active queries to gather more granular data.

There are also agentless solutions on the market that do not require all the spanning network infrastructure and security server at the facility. These solutions are targeted toward those facilities where quick OT/ICS security is wanted at a low price point. They are, however, not as comprehensive as a spanning based solution.

The next component of securing an OT facility is to ensure all the machines are authenticated and authorized to be on the network in addition to authorizing to whom they talk.

IAM

Identity and access management (IAM) are the core foundation of IT and OT security. OT IAM focuses on various machines and devices talking to one another. IT IAM security focuses on IT systems’ access to OT devices to gather ERP information.

Basic Grade—MAC

In accordance with the IEEE 802 LAN standards, the medium access control layer controls the hardware responsible for interaction with the wired, optical, or wireless transmission medium. On the wired side, this layer 2 networking is Ethernet using copper or fiber connectivity. Figure 8 depicts a standard Ethernet packet.

Figure 8: Standard Ethernet Protocol Frame

Many of the proprietary OT protocols run on top of this layer 2 connectivity, hence the focus on Ethernet which is the only standard that can be effectively leveraged across the OT environment. As mentioned earlier, every MAC address is unique. The first 6 bytes of the Ethernet header represent the manufacture of the Ethernet interface being used. For this reason, all OT inventory should record the MAC address for a machine or device.

Standard Grade—NAC

Network access control (NAC), also called network admission control, is used to improve the security, visibility, and access management of an IP network. It restricts the availability of network resources to endpoint devices and users that comply with a defined security policy. NAC can also provide endpoint security protection such as antivirus software, firewall, and vulnerability assessment with security enforcement policies and system authentication methods. OT NAC applies only to those more modern machines and devices that are leveraging Internet Protocol.

NAC is one aspect of network security. It provides visibility into the devices and users trying to access the enterprise network. It controls who can access the network and denies access to those users and devices that don’t comply with security policies. NAC solutions and tools help companies control network access, ensure compliance, and strengthen their IT and OT/ICS infrastructure. There are two types of NAC:

• Pre-admission: Evaluates access attempts and only allows entry to authorized devices and users.
• Post-admission: Re-authenticates users trying to enter a different part of the network; also restricts lateral movement to limit the damage from cyber-attacks.

NAC devices enforce security policies across all users and devices on a network through multiple capabilities. These devices:

• Limit network access to users, machines, and devices to specific network areas.
• Prevent data access by unauthorized employees and cybercriminals.
• Block access from endpoint devices that don’t comply with enterprise security policies.
• Manage the policy lifecycle for multiple operating scenarios such as OS and end point protection to ensure the latest supported version is being used.
• Recognize and profile users and devices to protect them from malicious code.
• Integrate with other security solutions through APIs.

Figure 9:  Mindsight’s NAC Reference Example[4]

Figure 9 above represents a NAC framework developed by Mindsight. One of the keys to a successful NAC solution is the ability to integrate with the other components within the OT/ICS stack such as inventory, network segmentation, AAA, and SIEM. Because validation of authentication is a key NAC attribute, multi-factor authentication (MFA) is covered in the following section.

Premium Grade—MFA (6x)

At the premium grade, Multi-Factor Authentication (MFA) is required for access to all machines and devices, whether the access is local or remote.  The additional factors for authentication are based on up to six attributes, including the traditional ones:

• Something You Know – Password or other knowledge-based information.
• Something You Are – Biometrics such as facial or fingerprint recognition.
• Something You Have – Token, personal computer, or smartphone.

The newer forms of authentication include:

• Location – Specific place based on IP network address, GPS, or RF signature.
• Context – History of previous access including time and day of week along with other business or transactional context.
• Behavior – Tracking a user, system, or machine’s activity and monitoring patterns and deviations such as typing or repeating a process 110 times per minute.

Up to six factor authentication is important for secure access. When changes are made to machines and devices, they are most vulnerable to outages, malware, or malicious access. When access is remote and especially by a 3^rd party, additional zero trust security measures should be taken including:

• Logging – Recording every keystroke of what an administrator is doing for the entire change along with all outputs. Screen recording is also an option.
• Continuous – Periodically checking the authentication and authorization and ensuring there is a session timeout set after no action which is typically 60-90 minutes and for the duration of a change window.
• One Time Password – Having a password or passcode that is only good during a specific window of time.
• Restrict Access – Allowing access only to the machines and/or devices in scope for the change and nothing else.
• Change Control Approval – Tying into a change management system, such as ServiceNow, with a change ticket and following the change approval process.

BeyondTrust is one example of a vendor with a product that provides a zero-trust remote access solution that meets the above requirements. TechVision has additional reports on IAM, including Privileged Access Management (PAM), in our portfolio.

SIEM

Security incident and event management (SIEM) is a core competency for any security team. Typically, a Security Operations Center (SOC) monitors all security events and acts on those deemed major or higher. A significant amount of data is collected through logs and tools are required to adequately process the data and classify events. The industry record for false positives is high and more work is being done to improve cybersecurity efficiencies.

Basic Grade—Logging

According to the Center for Internet Security (CIS) Version 7 Top 20 Controls (CIS, 2018), the sixth most important control is to monitor and analyze event logs. This activity is just as important for Industrial Control Systems. The widely-adopted ICS Cyber Security Framework (CSF) Version 1.1 from NIST lists the “Anomalies and Events” category under the core Detect Function (NIST, 2018). Energy companies operating ICS equipment under regulation of the North American Electric Reliability Corporation (NERC) standards require asset owners to log in CIP-007-6 R4 in order to implement and review systems for events (NERC, 2016). Furthermore, NIST 800-82 (2015) states “the security architecture of an ICS must also incorporate mechanisms to monitor, log, and audit activities occurring on various systems and networks.”

As companies’ operating ICS systems seek to comply with industry regulations, standards, and best practices, there often lacks detailed implementation guidance for establishing logging architectures, especially considering the diverse ICS install base.

Once a hacker has gained access to a system, one of their first tasks is to erase logs that hint that the machine, system, or device has been breached. Logs are used to monitor and detect system functions and access and if a system is breached, to identify the impact.  Logs are typically kept for a minimum of 90 days. A large hospital, manufacturing site, or power plant can generate over a Terabyte of logs in a single day.

Logging methods used in OT/ICS, such as Windows Management Instrumentation (WMI), Syslog, and Windows Event Forwarding (WEF), are common to the IT industry. Many ICS and SCADA systems provide logging through Windows Event Log. Each log source in Windows Event Log has a set of Event IDs associated with it. These logs can be filtered and parsed based on Event IDs by using the native Windows Event Log API.

Syslog is a commonly used logging protocol for network routers, switches, firewalls, and Unix or Linux operating systems. Syslog can also be used to gather logs from Programmable Logic Controllers (PLC), Remote Terminal Units (RTU), Intelligent Electronic Devices (IED) and other ICS devices, given they support Syslog logging functionality.

Most IT based logging systems require an agent be added to the server. As mentioned earlier, this can be difficult in an OT/ICS due to the proprietary nature of the systems or the chance the manufacturer will void the warranty/support of the system if 3^rd party software is added.

Auditing requires that logging is used. Most audits track account login events, the success or failure of login events, privileged access, and any changes made. What needs to be collected various by industry, country, and associated regulations.

Standard Grade—Event ID, Classification

Once all logging data is collected from the network, OT/ICS Tools, machines, systems, and devices, the data is sent to a SIEM to normalize, filter, assemble, and correlate it. A typical SIEM will then conduct additional correlation for the minor and above events.  While there are many different SIEM platforms on the market, they all provide the following features:

• Timestamp of security events – Timestamps are used to analyze and later correlate events with other information to better see the entire security picture. SCADA environments face a unique challenge in that there are multiple components with different time clocks. These clocks should be synchronized as much as possible to provide accurate data and to distinguish between simultaneous security events. More and more PTP is being used instead of NTP for clock synchronization.
  1. Precision Time Protocol (PTP) replaces Network Time Protocol (NTP) on complex systems to keep logs synchronized. PTP is 1000x more precise than NTP and will keep systems in sync to 10 microseconds vs 10-20 milliseconds found with NTP. A Stratum one clock source must be acquired. Most hosting solutions and carrier hotels such as Equinix offer PTP service.
  2. To put this in context, a single credit card transaction may go through 40 different networks and servers in the process of getting approved. If something goes wrong in the process, it is important to quickly identify where the transaction failed across the various systems that are talking back and forth. In another example from a Fortune 100 company, the load balancer was losing transactions under heavy load. In highly automated factories, PTP should be considered if not already in use.
• Collection of event specific information – Information from security events and raw logs is collected at a central service point where the information is analyzed and processed for future use by the SIEM solution. Links to the respective raw log items are retained for forensics investigators who require access to this identification information.
• Correlation of security-related events – An essential functionality of SIEM solutions is its capability to correlate security events across an OT/ICS environment. Security event information is collected by the SIEM solution and stored for a long time, so that backtracking can be performed to correlate security events with previous ones. The effect of this functionality can be profound.

Correlation is what allows the massive amount of information to be used in an intelligent way and spot possible attacks and security breaches. Without it, the same event occurring multiple times simultaneously will look just like a repeating event. With correlation, this behavior looks like an attack. To illustrate, suppose an ID card records an employee is leaving and 30 minutes later, that user’s login username and password is logged in a server room. Without correlation, this would look like two security events allowed within security controls. With correlation, you can connect the dots to see that a login occurring after an ID card is used to leave would indicate a security breach.

Premium Grade—AI/ML Baseline with Anomaly detection

By combining real-time data monitoring with orchestration and automated response, AI/ML solutions prove their value when compared to legacy systems and human-intervention driven response times. While these systems still require a lot of tuning today, they are becoming more intelligent and improving SOC efficiencies.

At a Black Hat Europe conference, security research firm CyberX demonstrated how data exfiltration was possible from a supposedly air-gapped ICS network. By delivering a payload of specific ladder logic code into PLCs, the attack was programmed to send out copies of data through encoded radio signals received by AM. As the communication channel is outside the TCP/IP stack, there is no encryption to safeguard the data once it is captured. In this example, AI/ML can be used to create a baseline of normal/standard behavior and monitor traffic and configurations to compare against the baseline. This baseline can include network traffic, equipment settings, and even the source code of PLCs. With continuous heartbeat checks, the algorithm can detect when the system deviates from the baseline and immediately alert security staff of the change.

The biggest advantage of implanting an AI solution for OT/ICS cybersecurity is its real-time response and orchestration. AI tools don’t need to wait for security staff to make a decision. They don’t see a black and white picture of firewall rules which often miss malware traffic flying under the radar, masquerading as “normal” network signals. Machine algorithms can detect abnormal data exchanges and immediately respond to the threat long before a SOC resource would be alerted. Some AI offerings can even monitor devices that don’t communicate over TCP/IP, creating powerful visibility into non-networked equipment.

One tool to protect industrial control systems is Cyberbit’s ScadaShield, a layered solution to provide full stack ICS network detection, visibility, smart analytics, forensics and response. ScadaShield continuously monitors and detects across the entire attack surface for both IT and OT components and can combine with SOC automation to trigger workflows to accelerate root cause identification and mitigation.

A Common Hybrid IT/OT Deployment

A hybrid approach is most common when installing OT/ICS security controls. Unlike modern IT security where users and devices are on one side and services and applications are on the other side, and all the security controls and management are in the middle. OT security requires tools and firewalling at the edge in the facility.

In Figure 10 below, on the left-hand side shows both the OT and the IT systems and the associated security required at the edge site. After this, going to the applications on the right-hand side, whether those are in a private data center or public cloud via IaaS or SaaS, they both go through the SASE security controls in the middle.

Figure 10: Hybrid IT & OT Architecture

The SASE security controls include:

• URL Filtering – Via DNS and/or Web proxy, controlling which sites a user has access
• IPS/IDS – Intrusion detection and prevention by decrypting TLS traffic, inspecting it, and either alerting or blocking suspicious traffic.
• Layer 7 Application Visibility and Control – Data inspection and classification and masking and/or blocking if the data is confidential or private.
• CASB – Cloud Access Security Broker and leveraging the cloud access APIs to ensure standard and approved authentication and authorization.
• Malware – Monitoring signatures, reputation, and network traffic patterns to identify and block malware.
• Isolation – Executables including malware sandbox to ensure a suspicious file is safe before passing it on.
• SIEM – Logging and monitoring of all network sessions and classifying any types of suspicious traffic and passing it to a SOC.

These cloud-based security controls can be updated and managed in near real time. Today, the Secure Cloud Gateways from companies such as zScaler, Palo Alto Prisma, or Netskope only inspect traffic in the cloud and do not have edge nor OT security capabilities. If they want to expand their market, they will have to develop this through acquisitions or home-grown solutions.

Sample of OT Network Discovery, Monitoring, and Management Solutions

One of the themes we have in many of our consulting engagements is to define “what good looks like” as a starting point to assess where you are and where you want to go. A good OT/ICS solution has the following features:

• Industry and country-specific security compliance checking.
• OT/ICS protocol support for over 200 vendor proprietary protocols.
• Passive and active asset discovery and identification.
• Vulnerability risk management based on OS version, password security, protocol validation.
• OT/ICS specific threat detection.
• APIs to import and export to other systems such as ServiceNow, Cisco ISE and SIEMs such as AlienVault or Microsoft Sentinel.
• Security analytics including building a standard operations baseline and detecting anomalies in connectivity, thresholds, and/or protocols (This includes AI/ML capabilities).
• Manageability including a great user interface to visually see and parse the connectivity at a site and between sites.
• A great database to organize the data and quickly search and find specific information.
• Support for and integration with the expanding array of IoT devices and gateways.

Below is a sample of some of the more common OT/ICS specific tools. This sampling has both small vendors focused purely on the OT space along with some large IT vendors getting into the OT market. There are over 30 vendors in this space and this market is growing from $5.5B in 2022 at a CAGR of 21%.

Dragos

Dragos was founded in 2016 with headquarters in Hanover Maryland. With over $300M in funding and 400-plus employees, Dragos provides OT security solutions throughout the world and across many different industry verticals. Their product, The Dragos Platform, provides the full suite of tools required to implement the OT security framework as outlined in this research paper.

Dragos is known for having a solid OT/ICS incident response team if an enterprise is compromised. Many enterprises will subscribe to this service even if they are not using the Dragos platform.

Furthermore, Dragos has Neighborhood Keeper, which is a collective defense and community-wide visibility solution that enables a more informed industrial defense by sharing threat intelligence across industries and geographic regions. By participating, each organization’s defensive capability is made stronger than what they can achieve on their own. Neighborhood Keeper is a free, opt-in, anonymized information sharing network available to all Dragos Platform customers. It was developed by Dragos in collaboration with the U.S. Department of Energy.

Dragos leads the market in utilities and critical infrastructure. Their solutions are optimized for emerging applications like the Industrial Internet of Things (IIoT), enabling their clients in power and water utilities, energy, and manufacturing industries to establish a resilient and adaptable security posture.  Dragos integration into IT security and operations is still maturing.

Fortinet

Fortinet was founded in 2000 by Ken Xie, the visionary founder and former president and CEO of NetScreen, and has its headquarters in Sunnyvale, California. Fortinet has over 10,000 employees, operates in 160 countries, and is a publicly traded company with a market capitalization of 50 billion dollars.

Fortinet has been selling security hardware and software with the goal of providing an end-to-end networking and security solution including ruggedized industrial ethernet switches and firewalls. As part of this vision, Fortinet has been expanding their OT security capabilities including new FortiDeceptor product, which aims to bring threat deception capabilities to OT environments. FortiGuard also provides signature feeds specific for OT, though this does require an additional software license.

Fortinet’s strength in OT/ICS security stems from its market-leading integration breadth with IT and OT technology vendors. This includes integrations with OT security vendors and control system vendors. Buyers with many Fortinet products can manage them all relatively easily. With the wide range of Fortinet security controls, it’s possible to build a Zero Trust IT/OT converged network. FortiSIEM and FortiAI helps enterprises provide the detection in large complex environments.

An enterprise looking for a single network and security vendor for both IT and OT networks globally should consider Fortinet across all market verticals.

Microsoft

Microsoft was founded in 1975 and its current headquarters are in Redmond, Washington.  Microsoft operates in 190 countries, has over 200,000 employees, and is a publicly traded company with market capitalization around 2 trillion dollars.

Microsoft acquired CyberX in 2020 and rebranded it as “Azure Defender for IoT” and has been busy integrating it into the Microsoft security reference architecture and products, including Azure Sentinel. Microsoft sees the value of bringing IT security controls into the OT environment and intends to integrate Microsoft 365 Defender (XDR) and Azure Sentinel (SIEM/SOAR) with Azure Defender for IoT.

How Microsoft will implement Azure Defender for IoT without impacting OT systems, or how this solution will support non-Microsoft operating systems and non-standard protocols is something yet to be seen. Azure Defender for IoT has strong asset discovery and identification. The product provides good visualizations of customer OT assets.

Azure Defender for IoT is suitable for security and risk buyers around the world already invested in the Microsoft security ecosystem and who do not have a lot of legacy OT systems. Being one of the top IT companies in the world, Microsoft can do whatever they choose to focus on. It is unknown how big a focus OT security will be.

Nozomi Networks

Nozomi Networks was founded in 2013 with headquarters in San Francisco, California. They have approximately 250 employees and have raised about $200M. With 11 offices through the world and many large partners, Nozomi serves all industries across the globe.

Guardian is the name of the Nozomi sensors that are used to collect all the inventory and network traffic and associated meta-data that is then sent to their Vantage cloud platform for analysis, visibility, detection, and response to incidents. Nozomi Networks’ industrial cybersecurity solutions are designed to protect mission-critical environments across all industries and sectors, including OT and IoT physical devices and processes.

Nozomi excels at proprietary OT protocol support with accurate analysis of over hundreds of OT, IoT and IT protocols. They are also leading with AI-driven insights and root cause analysis providing actionable intelligence to accelerate response or guide remediation efforts. They have the unique ability to write ad hoc queries to create alerts when assets or systems are not functioning correctly.

Nozomi has been working to integrate seamlessly with IT systems SIEM and SOC systems for closing operational visibility and security gaps. Their platform also provides rigorous forensic analysis for incident response and greater insights into how a threat got into the environment and the extent of damage done. Nozomi is also working hard to be able to make initial deployment quick and easy. Nozomi has many customers worldwide including utilities and manufacturing companies.

Verve Industrial Protection

Verve Industrial Protection, founded in 1994 with headquarters in Chicago, Illinois, is one of the older OT/ICS security companies operating. They have a little more than 100 employees and are a privately held company. The bulk of their customers are in North America, and they serve all industries.

Verve Industrial Protection has ensured reliable and secure industrial control systems for 25 years. Their principal offering, the Verve Security Center, is a vendor-agnostic OT endpoint management platform that provides IT-OT asset inventory, vulnerability management, and the ability to remediate threats and vulnerabilities from its orchestration platform. Verve’s Design-4-Defense professional services support clients in ensuring their OT environments are designed and operated in a secure manner.

This solution is 100% software based, and as such, does not require additional hardware on site. It integrates with any existing IT tools and uses agentless technology to provide visibility to PLCs and safety systems. One of its differentiators is the integration with many disaster recovery vendors, which would help a customer restore operations much faster than other vendors.

Verve Industrial Protection is still maturing with its integrations with secure remote access and LAN micro-segmentation. Verve Industrial Protection is good for many industrial organizations, including utilities and manufacturers in North America.

Next Steps

While OT security can seem overwhelming, like with IT security, start with the basics of good people, processes, information, and technology. Attributes that reflect on the maturity of an organization’s security include:

• Good Physical Security – Good physical security usually translates into good IT security. Physical security should be an ingrained part of an organization’s culture and should always be top of mind.
• Well Documented – All assets, systems, applications, and processes are well documented and routinely updated.
• Thorough Risk Registry – All known vulnerabilities are documented and the associated risk to the enterprise is quantified and reviewed monthly.
• DevOps Based – Security is continuously assessed and improved. Implementing good IT security is a journey and it needs to be regularly tuned.
• Tools & Automation –The cybersecurity landscape is changing too quickly to be able to manage all alerts, changes, and updates manually.

Starting at the bottom left in building a complete inventory of all OT systems and their current vulnerabilities is an important first step. This will then highlight the risks in the environment and justify spending the time and money to further move up and over on the OT security framework.

For many enterprises that have not invested in their LAN/WLAN in the past decade, this will require and local network refresh which can cost millions of dollars. Not only in new Ethernet switches and wireless access points, but also in fiber to support 10 & 100Gbps uplinks driven by more video traffic such as surveillance video and spanning all OT traffic to ensure no malware is not in the environment.

The final step and investment will be in AI/ML tools to be able to wade through the data lake of logs and information to identify, classify, and rectify any known security anomalies. With real-time signature updates, new malware attacks can be shut down quickly. AI/ML will also be used by cybercriminals and nation states, so like with other areas of security, enterprises will have to invest in the people, processes, information, and technology to stay one step ahead.

Conclusion

Digital business that requires interconnected systems, such as those within critical infrastructure, healthcare, and manufacturing industries, greatly increases cyber risk for a variety of enterprises. Given the geopolitical climate, governments around the world are warning that increased cyber-attacks are likely on key infrastructure and economic assets.

OT/ICS security is an emerging technology category that is growing in importance in response to many high-profile breaches in utilities, hospitals, and large manufacturers in the past few years. As a result, enterprises are prioritizing OT/ICS security and bringing in specific vendors to help them in their journey.

OT/ICS security platforms are being built and deployed with an initial focus on asset discovery, visibility, and network topology. New features are being added that include threat intelligence, vulnerability management, risk scoring and secure remote access. Most enterprises start their OT/ICS security journey by making a full discovery of all the assets they have and identifying the corresponding risk each asset creates.

While OT/ICS is less visible than IT security at most organizations, it is no less important to the economy and to people’s everyday lives. After all, OT systems control the critical infrastructure and machines businesses and people depend upon.

The investments required to follow through with the OT architecture recommended in this research report are significant, especially on the network side of the facility and particularly if the LAN/WLAN network has not been refreshed within the past decade. This is why TechVision recommends a phased approach so that enterprises can provide the right level of investment and stagger the upgrades across many years.

As OT/ICS security becomes more prevalent, using standard IT protocols, operating systems, and security controls and management will be the norm. The traditional niche OT security market emphasizes products focused on legacy industrial systems. The market is evolving rapidly as new tools become more sophisticated, but it will take the industry 7-10 years to catch up. Between government regulations and cybersecurity risks, enterprises will have to invest in OT security, which is why the expected growth of this market is 21% per year.

In coming years, OT/ICS security will be fully integrated organizationally along with common IT security practices. Expect the leading next generation firewall vendors to incorporate OT security into their solutions and market the value of an integrated IT/OT solution versus the stand-alone tools in place today.

 

About TechVision

World-class research requires world-class consulting analysts, and our team is just that. Gaining value from research also means having access to research. All TechVision Research licenses are enterprise licenses; this means everyone that needs access to content can have access to content. We know major technology initiatives involve many different skills across an organization and limiting content to a few can compromise the effectiveness of the team and the success of the initiative. Our research leverages our team’s in-depth knowledge as well as their real-world consulting experience. We combine great analyst skills with real world client experiences to provide a deep and balanced perspective.

TechVision Consulting builds off our research with specific projects to help organizations better understand, architect, select, build, and deploy infrastructure technologies. Our well-rounded experience and strong analytical skills help us separate the “hype” from the reality. This provides organizations with a deeper understanding of the full scope of vendor capabilities, product life cycles, and a basis for making more informed decisions. We also support vendors in areas such as product and strategy reviews and assessments, requirement analysis, target market assessment, technology trend analysis, go-to-market plan assessment, and gap analysis.

TechVision Updates will provide regular updates on the latest developments with respect to the issues addressed in this report.

 

About the Author

Sorell Slaymaker has 30 years of experience designing, building, securing, and operating IP networks and the communication services that run across them. His mission is to help make communication easier, cheaper and more secure since he believes that the more we communicate, the better we are. Prior to joining TechVision Research, Sorell was an Evangelist for 128 Technology which is a routing and security software company.  Prior to that, Sorell was a Gartner analyst covering enterprise networking, security, and communications.

Sorell is an IT Architect with a focus on network, security, and communications architecture.  He specializes in IT Architecture – Network Architecture, SIP Trunking, Contact Centers, Unified Communications, and Security Architecture.

 

Appendix 1

Additional Resources

• NIST SP 800-82r3 – Guide to Operational Technology Security

U.S. Government OT Mandate Summary

On July 28, 2021, President Biden issued a National Security Memorandum on Improving Cybersecurity for Critical Infrastructure Control Systems. This memorandum formally announced the Industrial Control Systems (ICS) Cybersecurity Initiative and outlines U.S. policy to safeguard our critical infrastructure, with a particular focus on the cybersecurity and resilience of systems supporting the functions of government and the private sector so vital that their disruption would have a debilitating effect on our national or economic security or the public health and safety of the American people.

Technologies that support organizational adoption of practices outlined in established ICS frameworks and standards, such as CIS CSC, ISA/IEC 62443, NIST SP 800-53, and NIST SP 800-82. 1

Security Features

Critical infrastructure organizations should consider whether ICS/OT cybersecurity monitoring

technologies have sufficient security features, such as:

• Support for multi-factor authentication (MFA), especially hardware-based MFA or other phishing-resistant methods.
• Cryptographic protection of data in transit, in use, and at rest (e.g., leverage NIST FIPS 140-3 approved cryptology to protect the data where feasible).
• Prohibition of universal default passwords for initial installation.
• Controls to minimize exposure of ports, protocols, or services to the network that are not necessary for the successful function of the technology.
• Guaranteed communication standard references for communications protocols and use different cryptological keys for different devices.

[1] https://www.ssh.com/academy/operational-technology-breaches

[2] www.nozominetworks.com/downloads/US/Nozomi-Networks-Protocol-Support-List.pdf

[3]  https://subscription.packtpub.com/book/networking-&-servers/9781788395151/1/ch01lvl1sec10/the-purdue-model-for-industrial-control-systems

[4] https://gomindsight.com/insights/blog/network-access-controls-why-you-need/