Many years ago, in the days when commercial vulnerability assessment (VA) products were just coming on the market, I developed an approach to VA that included a broad-based assessment followed by penetration testing to validate the results of the VA scan. Today, that is the standard practice among penetration test experts.
Just as the general methodologies for vulnerability assessment have evolved, so have the tools. The VA tools in this review are as diverse as the landscapes in which they are used.
There are three types of VA tools. First are scanners, which give little beyond listing vulnerabilities, their relative importance and suggested remedies. These are very useful, because they can be used easily, mostly automatically, and offer a good ongoing quality assessment. The downside is their limited functionality compared to other tools we tested. These tools, however, such as Nessus/NeWT and Saint are very good value and have a definite place in your testing arsenal.
The second type of tool is the full-featured appliance, which not only perform vulnerability scans, but correlate results to regulatory compliance, patch management and a host of other reporting functions. These can be pricey, but are the right answer for many organizations – if nothing else, they address the critical issue of compliance. We were extremely impressed with these appliances.
Finally, we have the (currently unique) tool that does just what experienced pentesters do: scan and follow up with penetration attempts. This tool, Core Impact, behaves exactly as one would expect a hacker to behave. It scans for vulnerabilities and then attempts to penetrate. Saint Corporation will soon introduce a competing product.
To help decide which of these three types of tools you need, look at expected outcomes and testing methodology. Organizations with significant risks in core areas – such as banks with online banking systems – need to pull out all the big guns to ensure that they are safe and in compliance. For these organizations, a combination of a tool that maps scan results against compliance issues and outputs a clear report, and a tool that attempts penetration makes sense.
For smaller organizations with limited tester and financial resources, a scanner might be enough.
Organizations that want to simplify patch management should look at products that offer patch management tied directly to the scan results.
Reporting in all these products was excellent relative to the results we expected the particular tool to produce. For example, the simple scanners produce detailed technical reports and are very good for engineers tasked with remediating security holes. The fully-featured products provide a variety of reports, from a simple graphical executive report to highly detailed engineering outputs.
Generally, the current crop of VA products offers a good range of capabilities for just about any organization, regardless of size.
Our test lab was configured to contain a set of targets with pre-defined vulnerabilities. These targets were of a variety of types and platforms. We used two Windows computers (Win2K and XPPro) and a Linux computer as platforms for software products, and a test rack for setting-up appliances. Targets were of two types: a honeynet based upon HoneyD running in a RedHat environment, and a few discrete targets that included a Sun Solaris 8.0 computer, two Linux machines and a Windows 2000 workstation.
We would like to acknowledge Nick Michaluk, a technology student at Eastern Michigan University, for contribution of his superb live CD HoneyD implementation called HoneyEMU.
We ran the products against this test bed and, where the product was designed to discover the network, we allowed it to try. In a couple of cases, the device was unable to detect the presence of the honeynet without being set up manually. All of the targets and products were given static addresses. We monitored the tests using Snort in packet capture mode plus intrusion alerting.
Virtually all scanners, with one exception, detected between 100 and 120 vulnerabilities, and we concluded that this was not the primary measure of a good product. In all, we graded the products in over 50 categories.
We selected two best buys, one in the pure scanner category and one in the full-featured compliance management category. Because Core Impact stands out as the only penetration tool in the group, we single it out for special recognition as strongly recommended for users who need this sort of power.
