This month we looked at vulnerability assessment and penetration test tools. The leading difference between last year’s tests and this year’s is that this year we saw more hybrid products that offered both vulnerability scanning and penetration testing. We also reviewed a passive scanner for the first time and saw a lot more attention to meeting regulatory requirements, especially in the payment card industry.
As always, we had a nice bunch of products that included appliances and software-only. We had one product that is strictly a penetration testing tool and, as we had some other products that included vulnerability assessment and penetration testing, this forced us to break the group up into three sub-groups. These groups are vulnerability assessment, penetration testing and hybrid (both).
Our general observation is that appliances are becoming the platform du jour for this product group. Generally, the appliances offered more features than software-only products and the reporting was more robust. Additionally, we are seeing more products based on the open source version of Nessus. We included Nessus in this review because it is one of the most popular vulnerability scanners available. The differentiators for the products using Nessus as a core platform are ease of use and the available extensions, such as advanced reporting, user interface, dashboard, etc.
Finally, another important trend is distributed scanners reporting to a central console that acts both as a management console and results correlator. These systems usually offer sophisticated management, correlation and reporting capabilities, as well as add-on services, such as patch notification, trouble ticketing and remediation assistance.
Hybrid products are not always the best bet for all organizations. There are reasons to scan only, reasons to buy a solid penetration testing tool, and reasons to buy a hybrid tool that scans and penetrates. We see straight vulnerability testing tools as appropriate for ongoing testing. Regular scanning reports now can be integrated into multipurpose devices, SIM/SEM products and the new category of security risk management (SRM) products to give a more complete view of the network’s security posture. We recommend that all organizations use some sort of regular vulnerability scanning.
Penetration is appropriate as an add-in to scanning. Scanners will tell you where the holes are likely to be, but penetration testing tools can attempt to exploit those potential holes to let you know for certain. If you have neither, a hybrid is a good bet. Moreover, vulnerability testing is a bit difficult in the sense that scanners often find false positives. Some products recognize a possible false positive and tag it for you. The most reliable approach is to use two scanners or a scanner and a penetration tool.
Penetration by itself is not reliable unless the penetration tool performs a preliminary scan to decide where to test. Core Impact, for example, uses this technique if you run all of the scripted test sequences, but it is a penetration tool and cannot be considered to be a scanner. Scanning simply is part of its overall technique to identify possible holes.
How we tested
Our vulnerability test bed consisted of examples of several operating environments, both patched and unpatched. Our victim suite included MS Windows XP and 2000, both as shipped and with current service packs and updates, Solaris, and two versions of Linux. We took vulnerabilities detected from the scanners and we measured the ability to exploit holes for the penetration tools.
We also ran penetration scripts when available. This technique runs a suite of penetration tests based on such things as operating environment, version and patch levels and open services.
From the functional perspective we looked for ease of use in a production environment where tests must be run by few people in little time. We liked the penetration tools to prove they penetrated by placing a shell agent on the victim or placing files.
We evaluated the number and type of reports, whether custom reports are possible and how the product presents its findings on a dashboard or other standard output. We especially favored those products that take advantage of the common vulnerabilities and exposures (CVE) to define vulnerabilities unambiguously.
In general, we found this year’s batch of tools to be a significant improvement over last year’s and we were impressed with their utility, ease of use and comprehensive reporting.
