Product Details

Special report: Intrusion prevention

The NSS Group has conducted a six-month investigation into host and network intrusion prevention systems. Bob Walder shares some of their results

The inadequacies inherent in current defenses have driven the development of a new breed of security products known as Intrusion Prevention Systems (IPS). The term has provoked controversy, because some firewall and intrusion detection system (IDS) vendors think it has been hijacked to be used as a marketing term, rather than as a description for any kind of new technology.

While it is true that firewalls, routers, IDS devices and even AV gateways have intrusion prevention technology included in some form, we believe that there are sufficient grounds to create a new market sector for true IPS.

These systems are proactive defense mechanisms designed to detect malicious packets within normal network traffic (something that the current breed of firewalls do not actually do, for example) and stop intrusions dead, blocking the offending traffic automatically before it does any damage rather than simply raising an alert as (or after) the malicious payload has been delivered. Within the IPS market place, there are two main categories of product - Host IPS and Network IPS.

Host IPS

As with Host IDS systems, the Host IPS (HIPS) relies on the protection of agents installed directly on the system. It binds closely with the OS kernel and services, monitoring and intercepting system calls to the kernel or APIs in order to prevent attacks as well as log them.

It may also monitor data streams and the environment specific to a particular application (file locations and registry settings for a web server, for example) in order to protect that application from generic attacks for which no "signature" yet exists.

One potential disadvantage with this approach is that, given the necessarily tight integration with the host operating system (OS), future OS upgrades could cause problems.

Since a HIPS agent intercepts all requests to the system it protects, it has certain prerequisites - it must be very reliable, must not negatively impact performance, and must not block legitimate traffic. Any HIPS that does not meet these minimum requirements should never be installed in a host, no matter how effectively it blocks attacks.

Network IPS

The Network IPS (NIPS) combines features of a standard IDS, an IPS and a firewall, and is sometimes known as an in-line IDS or gateway IDS (GIDS). The next-generation firewall "the deep inspection firewall" exhibits a similar feature set, though we do not believe that the deep inspection firewall is ready for mainstream deployment just yet.

As with a typical firewall, the NIPS has at least two network interfaces, one designated as internal and one designated as external. As packets appear at either interface they are passed to the detection engine, at which point the IPS device functions much as any IDS would in determining whether or not the packet being examined poses a threat.

However, if it should detect a malicious packet, in addition to raising an alert, it will discard the packet and mark that flow as bad. As the remaining packets making up that particular TCP session arrive at the IPS device, they are discarded immediately.

Legitimate packets are passed through to the second interface and on to their intended destination. A useful side effect of some NIPS products is that as a matter of course - in fact as part of the initial detection process - they will provide "packet scrubbing" functionality to remove protocol inconsistencies resulting from varying interpretations of the TCP/IP specification (or intentional packet manipulation).

Thus, any fragmented packets, out-of-order packets, or packets with overlapping IP fragments will be re-ordered and "cleaned up" before being passed to the destination host, and illegal packets can be dropped completely.

One thing to watch for - don't let the "reactive" IDS vendors lead you to believe that they have intrusion prevention capabilities just because they can send tcp reset commands or re-configure a firewall when they detect an attack (a worrying implication we have noticed in some recent IDS marketing literature).

The problem here is that unless the attacker is operating on a 2400-baud modem, the likelihood is that by the time the IDS has detected the offending packet, raised an alert, and transmitted the TCP resets - and especially by the time the two ends of the connection have received the reset packets and acted on them (or the firewall or router has had time to activate new rules to block the remainder of the flow) - the payload of the exploit has long since been delivered? game over! Our guess is that there are not that many crackers using 2400-baud modems these days?

A true IPS device, however, is sitting in-line - all the packets have to pass through it. Therefore, as soon as a suspicious packet has been detected - and before it is passed to the internal interface and on to the protected network, it can be dropped. Not only that, but now that that flow has been flagged as suspicious, all subsequent packets that are part of that session can also be dropped with very little additional processing. Oh, and for good measure, some products are also capable of sending tcp resets or ICMP Unreachable messages to the attacking host.

Implementation challenges

There are a number of challenges to the implementation of an IPS device that do not have to be faced when deploying passive-mode IDS products. These challenges all stem from the fact that the IPS device is designed to work in-line, presenting a potential choke point and single point of failure.

If a passive IDS fails, the worst that can happen is that some attempted attacks may go undetected. If an in-line device fails, however, it can seriously impact the performance of the network. Perhaps latency rises to unacceptable values, or perhaps the device fails closed, in which case you have a self-inflicted denial-of-service (DoS) condition on your hands.

On the bright side, there will be no attacks getting through. But that will be of little consolation if none of your customers can reach your e-commerce site.

Even if the IPS device does not fail altogether, it still has the potential to act as a bottleneck, increasing latency and reducing throughput as it struggles to keep up with a Gigabit or more of network traffic. Devices using off-the-shelf hardware will certainly struggle to keep up with a heavily loaded Gigabit network, especially if there is a substantial signature set loaded, and this could be a major concern for both the network administrator - who could see his carefully crafted network response times go through the roof when a poorly designed IPS device is placed in-line - as well as the security administrator, who will have to fight tooth and nail to have the network administrator allow him to place this unknown quantity amongst his high performance routers and switches.

As an integral element of the network fabric, the NIPS device must perform much like a network switch. It must meet stringent network performance and reliability requirements as a prerequisite to deployment, since very few customers are willing to sacrifice network performance and reliability for security. A NIPS that slows down traffic, stops good traffic, or crashes the network is of little use.

Dropped packets are also an issue, since if even one of those dropped packets is one of those used in the exploit data stream it is possible that the entire exploit could be missed. Most high-end IPS vendors will get around this problem by using custom hardware, populated with advanced FPGAs and ASICs - indeed, it is necessary to design the product to operate as much as a switch as an Intrusion Detection and Prevention (IDP) device.

It is very difficult for a security administrator to characterize the traffic on his network with a high degree of accuracy. What is the average bandwidth? What are the peaks? Is the traffic mainly one protocol or a mix? What is the average packet size and level of new connections established every second? These are both critical parameters that can have detrimental effects on some IDS/IPS engines. If your IPS hardware is operating "on the edge", all of these are questions that need to be answered as accurately as possible to prevent performance degradation.

Another potential problem is the false positive. The bane of the security administrator's life, the false positive appears when an exploit signature is not crafted carefully enough, so that legitimate traffic can cause it to fire accidentally. While merely annoying in a passive IDS device (consuming time and effort on the part of the security administrator) the results can be far more serious and far-reaching in an in-line IPS appliance.

Once again, the result is a self-inflicted DoS condition, as the IPS device first drops the 'offending' packet, and then potentially blocks the entire data flow from the suspected hacker. If the traffic that triggered the false positive alert was part of a customer order, you know the customer will not wait long as his entire session is torn down and all subsequent attempts to reconnect to your e-commerce site (if he bothers to try again) are blocked by the well-meaning IPS.

A potential problem with any Gigabit IPS/IDS product is, by its very nature and capabilities, the amount of alert data it is likely to generate. On such a busy network, how many alerts will be generated in one working day? Or even one hour? Even with relatively low alert rates of ten per second, there will be 36,000 alerts every hour. That is 864,000 alerts each and every day. The ability to tune the signature set accurately is essential in order to keep the number of alerts to an absolute minimum.

Once the alerts have been raised, however, it becomes essential to be able to process them effectively. Advanced alert handling and forensic analysis capabilities - including detailed exploit information and the ability to examine packet contents and data streams - can make or break a Gigabit IDS/IPS product.

Of course, one point in favor of IPS when compared with IDS is that because it is designed to prevent the attacks rather than just detect and log them, the burden of examining and investigating the alerts ? and especially the problem of rectifying damage done by successful exploits - is reduced considerably.

Testing methodology

The NSS Group has conducted the first comprehensive Intrusion Prevention System (IPS) test of its kind. This exhaustive research offers a complete perspective of the capabilities, maturity and suitability of the products tested for their special requirements. This test was established because IPS products are being actively deployed as a new layer in defense-in-depth security architectures.

As part of its extensive IPS test methodology (see www.nss.co.uk for the detailed methodology) the NSS Group subjects each product to a brutal battery of tests that verify the stability and performance of each IPS tested, determine the accuracy of its security coverage, and ensure that the device will not block legitimate traffic.

To assess the complex matrix of IPS performance and security requirements, the NSS Group has designed and created a specialized lab environment that is able to exercise every facet of an IPS product. The test suite contains over 750 individual tests that evaluate IPS products in three main areas:

- Performance and reliability;

- Security accuracy;

- Usability.

This thorough review should give readers a complete perspective of the capabilities, maturity and suitability of the products tested for their particular needs.

Performance

Any IPS is expected to be reliable. In other words, an IPS is not supposed to crash, it should never block legitimate traffic, and it should not unduly affect network or host system performance.

Detection/blocking performance under load

This group of tests verifies that the IPS does not adversely impact legitimate traffic, even when new Transmission Control Protocol (tcp) connections are being created rapidly. We also verify that the sensor is capable of detecting and blocking exploits when subjected to increasing loads of background traffic, up to the maximum bandwidth supported as claimed by the vendor. An IPS that misses attacks under load is dangerous because it can be evaded. An IPS that adversely affects legitimate background traffic will not stay in-line for long.

All tests are repeated with 250Mbps, 500Mbps, 750Mbps and 1000Mbps of background traffic (or up to the maximum rated throughput of the device in 25 percent increments should this be less than 1Gbps). The test is conducted with User Datagram Protocol (udp), http, and mixed-protocol traffic. The test has packet rates up to 1.48 million packets per second and connection rates up to 20,000 connections per second.

Latency and user response times

This group of tests determines the effect the IPS sensor has on the traffic passing through it under various load conditions. High packet latency will lower tcp throughput. High application latency will create a negative user experience.

Bi-directional network latency of udp packets is measured under three test conditions: with no load, with 500 Mbps of http traffic (or half the rated load of the device if this is less than 1Gbps), and while the device is under a heavy SYN flood attack (up to ten percent of the rated throughput of the sensor).

Spirent Avalanche and Reflector devices are used to generate http sessions through the device in order to gauge how any increases in latency will impact the user experience in terms of failed connections and increased web response times. This 'application latency' is measured with no background load and also while the device is under attack.

Stability and reliability

Our stability and reliability tests verify the stability of the IPS device under a variety of extreme conditions. Long-term stability is critical for an in-line IPS device,because failure can produce inconvenient network outages.

In the first part of this test, we expose the external interface of the sensor to a constant stream of millions of attacks over an extended period of time (eight hours). The device is configured to block and alert, and therefore, this test provides an indication of the effectiveness of both the blocking and alert handling mechanisms. The device is expected to remain operational and stable throughout this test, blocking 100 percent of recognizable exploits, raising an alert for each, and passing through 100 percent of legitimate traffic.

In the second part of the test we stress the protocol stack of the device being tested by exposing it to traffic from the IP Stack Integrity Checker (ISIC) test tool for eight hours (both management and sensing interfaces). The device is expected to remain operational and capable of detecting and blocking exploits throughout the test in order to pass.

Security effectiveness, detection accuracy and breadth

This group of tests verifies the IPS will not block legitimate traffic (accuracy) and is capable of detecting and blocking a wide range of common exploits (breadth). Although breadth is very important, accuracy is critical because an IPS that blocks legitimate traffic will not remain in-line for long.

While it is not possible to validate completely the entire signature set of any IPS, this test demonstrates how accurately the IPS detects and blocks a wide range of common exploits, port scans, and denial-of-service (DoS) attempts.

This test is repeated twice. The first time, this test is run with blocking disabled on the IPS in order to determine which attacks are detected and how accurately they are detected (attack recognition rating). The second time, it is run with blocking enabled in order to determine which attacks are blocked successfully regardless of how they are detected or what alerts are raised (attack blocking rating).

Resistance to evasion techniques

These tests verify that the IPS is capable of detecting and blocking basic exploits when subjected to varying common evasion techniques.

An IPS that cannot detect attacks subjected to these 'script kiddie' evasion techniques is easily circumvented. These tests consist of packet fragmentation and stream segmentation (i.e. fragroute); URL obfuscation (i.e. Whisker); and miscellaneous evasion techniques (a mixed set of protocol or exploit-specific evasion techniques). Stateful operation The aim of this section is to be able to determine whether the IPS is capable of monitoring stateful sessions established through the device at various traffic loads without either losing state or incorrectly inferring state.

An IPS that does not maintain tcp session state can flood the management console with false-positive alerts. Although this should not directly impact the IPS blocking function, it can make it very hard to perform forensic analysis of the attacks.

In addition, if the default condition of the sensor is to block all traffic for which it does not believe there is a current connection in place, then an inability to maintain state under extreme conditions could result in the sensor blocking legitimate traffic by mistake - obviously an undesirable result.

We test whether the sensor is capable of preserving state across increasing numbers of open connections, as well as continuing to detect and block new exploits while not blocking legitimate traffic when the state tables are filled. Various numbers of tcp sessions from 10,000 to 1,000,000 are tested.