We looked at six IPS appliances, some of which this magazine has looked at from different perspectives in the recent past. We selected a range of products that cover the gamut from expensive devices from well-known vendors to less expensive, less well-known products. Surprisingly, perhaps, some of the less expensive products performed as well as, and in some cases better than, the more expensive ones. The major differences were in the ease of setup and configuration.
Because some products had only recently received rigorous performance testing, we focused on user-related functionality such as ease of setup, features, documentation, technical support, administrative user interface and general security of the product.
We were surprised that many of the products we tested were very difficult to install and configure. This is recognized by the product vendors, because in most cases we were offered an on-site support engineer during our testing, an offer we universally declined.
We developed a standardized matrix of questions we wanted to answer about each product. Many of these questions were answered during the installation and test phases and some related to such things as completeness of documentation and adequacy of technical support.
In the "ease of setup" category we were concerned about such things as a defined procedure; quality of installation documentation; level of scripting versus the need to enter command-line commands; a procedure to verify that the installation had proceeded correctly; length of the process; and presence of an installation log. We also evaluated the hardware and software completeness.
For example, as an appliance, all hardware and software required for implementation should have been included and shipped with the product. Sadly, that was not always the case. In more than one instance we were not provided with the appropriate web administration clients without contacting technical support for an overnight shipment or a key for download. In one case, the product required that the buyer purchase a copy of Microsoft SQL Server as a back-end database.
Once the product was installed, we ran vulnerability scans against our test-bed network and evaluated the product's response both in monitor (sniffer) mode and in in-line mode. Additionally, we ran the scans against the product itself to determine its level of self-protection. Finally, we evaluated the time and effort required to install the product and the presence of default accounts.
We contacted each of the technical support numbers provided by the product vendors. While we expected to need to fabricate problems to evaluate technical support response, we actually found there were more than enough genuine difficulties to test technical support organizations completely.
Some of the factors we considered in our evaluation of technical support were the presence of a toll-free number, ease with which we were able to talk directly to a support engineer, 24-7 availability of support, type of support available (live engineer, voicemail, email, web, and so on) and wait time. We also looked at web-based support help such as FAQs, support archives with solutions to typical problems, user groups and mail lists.
Arguably, the most important feature of an IPS is the user interface. We looked for an easy-to-use graphical interface that offered enough flexibility to create appropriate rules whether they were standard within the product or not. If a scripting language was available, how easy was it to use and how well documented was it?
We evaluated the availability and nature of online help, as well as such important features as session logging, preservation of menu structure and assumed level of user knowledge.
Here we were concerned with the level of security preserved by the appliance. For example, in some cases it was easy to reboot the appliance into single-user Unix mode and defeat completely any internal security on the device.
We were interested in such other security functions as default file protection privileges; ability to override or change the root password for the underlying operating environment without knowing the password in the first place (that is, the ability to boot into Unix maintenance mode); and security of the management console.
In general, we were disappointed with the level of maturity in the IPSs we tested. They were difficult to configure and verifying the correct configuration was often ambiguous. Products we expected to be complete as delivered often were not and we even received one engineering model instead of the requested production version.
