The intrusion detection system (IDS) jumped from research labs and universities into being a standard information security safeguard in just several years.
Together with firewalls and vulnerability scanners, intrusion detection is one of the pillars of modern cybersecurity. This article will look at certain critical shortcomings of IDS technology and analyze possible future trends to overcome them.
While the IDS field is still in motion, several classes of products have formed. Most IDS products fall into network IDS (NIDS), host IDS (HIDS) and hybrid products. To summarize, a host IDS needs to be deployed on each protected machine (server or workstation). It analyzes data local to that machine such as system log files, audit trails and file system changes, and sometimes processes and system calls. HIDS alerts the administrator in case a violation of the preset rules occurs. Host IDS might use pattern matching in the observed audit trails or generate a normal behavior profile and then compare current events with this profile.
Network IDS, on the other hand, is deployed on a separate device (computer or appliance called an IDS sensor) and monitors the entire subnet for network attacks against machines connected to it, using a database of attack signatures or a set of algorithms to detect anomalies in network traffic. Alerting and attacks analysis might be handled by a different machine that collects the information from several sensors.
Hybrid IDS is somewhat harder to define. Sometimes a hybrid IDS correlates the host events or vulnerabilities with network traffic, or only monitors the network local to the host, or combines attack signatures with some means of anomaly detection.
In light of these different characteristics, IDS possesses distinct advantages and disadvantages both from technology and management viewpoints. Network IDS almost always uses a sniffer that grabs all network traffic passing through the wire, runs the collected packets through the analysis engine and then reacts accordingly (records the packets, alerts the security professional or initiates some countermeasures such as terminating the connection or modifying the firewall rules to block the traffic).
An analysis engine looks at individual packet protocol flags, source and destination addresses, and application protocol payload such as email messages or web requests. In addition, analysis might be applied to the whole TCP connection rather than individual packets or even include correlation of the connection to those occurring earlier or elsewhere on the network. The attack signatures, typically supplied by a vendor or developed in-house, consist of strings (such as "CMD.EXE" in case of many Microsoft IIS web attacks) or network packet parameters (such as size for the 'Ping of Death' attack). The advantages of network IDS include an ability to protect the whole network with a single IDS machine with no impact on the network architecture and hosts, possible high security or even invisibility of the IDS itself, and simplicity of signature updates.
Host IDS technology might include special software that runs on each system or even a kernel-level driver to track system calls. HIDS is better at detecting actual attacks rather than potentially dangerous traffic, can detect non-network based attacks (such as from a system console) and attacks on specific applications. It is also unaffected by factors that plague network IDS, such as switched networks and encryption. However, HIDS has to be deployed on all computers, sometimes modifies the applications and operating system and can be disabled by an attacker after successful penetration.
It appears that signature-based network IDS is the most widely deployed type of intrusion detection. Simplified management and the availability of inexpensive IDS appliances together with dominance of network-based attacks are believed to be the primary reasons for that. However, simple network attack signature detection can be extremely misleading, ineffective and cause more trouble than benefits under certain circumstances. We will now outline these circumstances and then provide some possible safeguards.
First, the arrival of switched networks where information is only sent to the host that needs it rather than to all machines on the classic shared Ethernet, dealt a blow to IDS technology. NIDS now has to be connected to a switch itself. In addition, since the speed of networks is increasing, the IDS has to handle higher and higher loads. While listening on the switch port, the IDS receives the combined network traffic up to maximum network throughput, times the number of switch ports. Moreover, since IDSs are getting smarter (i.e. wanting to perform more in-depth analysis) at the same time, their ability to keep processing traffic on wire speed is jeopardized. IDS is inherently 'fail-open': unlike firewalls, if IDS cannot keep up, the communication will pass it uninspected. Second, the popularity of VPNs and encryption makes network sniffing and analysis impossible. Third, NIDS by definition cannot know of the impact of the attacks on the actual attacked host. Moreover, it cannot know whether the network communication will be accepted by the host.
Since the critical paper of T. Ptacek and T. Newsham (https://secinf.net/info/ids/idspaper/idspaper.html), even more limitations of NIDS have become clear. The authors specify a huge number of problems with network IDS detection and even with some proposed solutions to those problems. "Insertion, Evasion, and Denial of Service: Eluding Network Intrusion Detection" was released in 1998 and many of the easily exploitable scenarios are still not addressed by IDS vendors.
Fragmentation was one of the better known IDS weaknesses - for years, hacker tools to route attacks through the fragmenting host (fragrouter by Dug Song, 1999) existed, to the chagrin of IDS vendors. Sending TCP packets out of order is also used to fool intrusion detection. Web application attack tool Whisker can also operate in several IDS avoidance modes and is able to fool many intrusion detection systems. However, vendors are potentially able to fix most of those limitations.
However, some recent developments might truly bring an end to NIDS. Why try to sneak by the IDS if one can just render it ineffective? Such attacks are sometimes called IDS denial-of-service (DoS) or blinding. Several attack generators that can go through the IDS rules and generate thousands of attacks of the precise type that an IDS is set to detect have emerged recently. IDS testing tools such as snot (https://www.sec33.com/sniph/), stick (https://www.eurocompton.net/stick/projects8.html) and sneeze (https://snort.sourceforge.net/sneeze-1.0.tar) can trigger hundreds of IDS alarms per second (!). If these fail to make the IDS start skipping attacks, than they will make the IDS analysts watching the screens start doing so. One tool author reports that his program crashed one commercial IDS in two seconds and made the Linux-based IDS snort drop packets. The scary thing is that the smarter the IDS is, the faster it will die of resource starvation, since the software has to do more processing of each packet.
What are the countermeasures to this IDS attack? In general, there are none: IDS has to detect valid attacks that are occurring. However, it is believed that several methods to mitigate such behavior exist. Rate limiting will only help against the simplest attack generators - better tools can send randomized attack packets that still trigger the IDS, but have different properties otherwise. A better idea is to make an IDS more aware of the network context to determine whether the attack might have an impact. For example, sending a buffer overflow string before establishing a TCP/IP connection can be safely dropped, since it will have no effect on the target. This is actually implemented by some vendors. Unfortunately, the attack generator can get away with only valid attacks and still flood the sensor. In this case, the only win of the defending side is knowing the identity of the packet source - established TCP connections cannot be spoofed.
To conclude, we just briefly touched upon the limitations of network IDS. The obvious conclusion is that NIDS is not a 'shoot-and-forget' technology yet. Before deployment, it is wise to evaluate available IDS solutions by their reaction to anti-IDS attacks. After deploying the IDS, an organization should be aware of its limitations and possess the necessary expertise to run the intrusion detection effectively.
Anton Chuvakin, Ph.D., is a senior security analyst with netForensics (www.netforensics.com), a security information management company that provides real-time forensics software solutions.
