Product Group Tests

Intrusion prevention (2006)

Group Summary

We rate the Reflex IPS 100 as our Best Buy in the large appliance group.The device is low cost,demonstrates high performance and flexibility,and provides first-rate intrusion protection.Our Recommended award in this category goes to the InstaGate PRO for its flexibility and extensibility. In the small system category,we rate Ally ip100 as Best Buy for its portability, very low cost, and its ability to provide quick and effective protection on small to medium-sized networks.

Scroll To Full Group Summary Below

Click for a side by side comparison of products
Click for a side by side comparison of products

Full Group Summary

In the past two years, since our last SC group test of intrusion prevention systems (IPSs), they have become more effective, more widely distributed and more complicated to deploy. What’s more, the more complicated systems are consistent with today’s more complex networks.

For an IPS to be effective, it needs a proper installation. This can be a daunting task, so be sure you include on your deployment team the best experts on your network that you can find.

As you plan for deployment, you need to remember that the more complex (configurable) the IPS, the more opportunities you have to make errors. If you intend to depend on it to protect you, that can be a serious problem. Also, the more detail and customisation is required when writing policies, the more likely errors are.

However, although we were, generally pleased with this batch, one area that disappointed us was the lack of dependable, comprehensive protection. While all the products performed better than their peers two years ago, about half were unable to prevent our more aggressive attacks. All were good at blocking simple attacks, such as port scans and vulnerability sweeps, but when we unleashed our big guns, several buckled under the strain – a fundamental flaw for this type of product.

Another area of disappointment was support. While all vendors offer support of some type, many ask you to purchase it. In its most extreme example, this even extended to access to the vendor’s support website. For a class of product where more than half the vendors offered us our own, personal onsite support engineer (and one even recommended in its manual that you use an onsite support engineer to deploy its product), we think that customer support should be free, at least for the first year while the bugs ring out of the implementation.

This group was also full of surprises. In a field where a midrange product can cost around £15,000, the real standout was a product that measured about eight inches long, looked like a square orange tube and cost £500. It was the only product we tested that performed flawlessly in all areas. So we selected two Best Buys: one in the large appliance category and the other for products that work well in small enterprises.

Before we tested this group, we configured an appropriate test bed – an interesting challenge, because some products were in-line, some had multiple sensors, and some were self-contained. The architecture for IPSs is varied and usually reflects the complexity of the enterprise in which it is to be used. Multi-sensor products fit well with large, distributed enterprises, for example.

Once the product was in its test bed, we configured it to its default settings and attempted to see it and its sensors over our isolated test network. Network-facing sensors should not be bound to an IP address in order to keep them safe from attacks intended to disable them. Address scans should not reveal the presence of any sensor.

Our next task was the soft scans. These were comprehensive vulnerability scans using a NetClarity Auditor Enterprise 4.1 vulnerability scanner. This is the vulnerability assessment workhorse in our lab and it gives us a comprehensive picture of a target’s vulnerabilities. We scanned both the IPS (usually just the console if the sensor is stealthed) and the target network being protected.

Our final test used Core Impact 5.1. This let us configure specific penetrations based upon exploits that we believed would get past the IPS. First, we ran a general penetration test on both the IPS and the target. Finally, we ran our suite of IPS evasion tests and tried to bypass the IPS. About half the time we succeeded. Core Impact is very powerful and our evasion tests include such capabilities as packet fragmenting.

We ended our tests with mixed emotions. First, the improvement over the past two years has been remarkable. Two years ago, some products simply did not work, and were easy to penetrate because they were based on unhardened Linux OSs.

Today, many products had purpose-built operating environments and they could not be penetrated using our tool sets.

On the other hand, these tools are very complicated, and follow the current trend of requiring complexity to support today’s more complex networks – a trend seen in almost every product group test this year.

We wish that some vendor would recognise that complexity in the tool is not necessary, even if the enterprise is complicated. Like many things in life, simplicity is better. Some IPS products could certainly use some designed in from the start.

All Products In This Group Test