Product Group Tests

Application vulnerability assessment 2007

Group Summary

SPI Dynamics Assessment Management Platform (AMP) is a very solid offering which builds on the foundation of WebInspect, adding enterprise use and role management. We rate AMP as our Best Buy, despite its price, for its strong enterprise management of web vulnerabilities.

Scroll To Full Group Summary Below

Click for a side by side comparison of products
Click for a side by side comparison of products

Full Group Summary

Port 80 is often called the highway into networks. Port 443 for secure sockets layer (SSL) is often referred to as the UFBP — universal firewall bypass port. Today, many legacy applications are either web-enabled or in the process of becoming web-enabled. Consequently, these applications — which were never designed to be used in this fashion — are now being exposed in new ways to larger and larger user communities, as well as attacking communities with increasing sophistication. In many cases, the process of web-enabling an application exposes critical assets, such as large databases with personal client information.

Even some non-legacy applications that were designed from inception to run as web-enabled applications can contain significant security vulnerabilities. There often is a disconnect between web programmers, auditors and information security staff that allows these web applications to bypass many system development life cycle controls, such as code reviews and security testing. This is not to say that all web coding is bad; however, there is a potential where mistakes can be made and critical data or even system control can be lost.

These issues have been brought to the forefront by the PCI-DSS (Payment Card Industry — Data Security Standard) in many organizations. In the standard, section 6.3.1 requires "Testing of all security patches and system and software configuration changes before deployment."

This is typically part of a comprehensive system development life cycle and often the term vulnerability assessment is applied to this testing. Further in the PCI-DSS standard in section 6.5, several web-based vulnerabilities are listed to be tested by the application provider.

To mitigate these risks and also for compliance with industry best practice standards, application vulnerability assessment must be performed. This type of assessment is different from the more common network vulnerability assessment because of the need for a greater understanding of web-based vulnerabilities. For example, the most commonly used network vulnerability assessment utility, Nessus, checks for XSS or cross-site scripting errors. However, it does not check the hundreds of different permutations of the XSS attacks. In order to scan for these dynamic attacks — such as XSS or SQL injection — a utility with greater understanding of the application environment is necessary.

The utilities in this group tested for either web-based vulnerabilities or vulnerabilities inside of an SQL database. All of these products had the additional intelligence to scan beyond the depth that a traditional network vulnerability assessment utility could.

Products in this review broke down into one of two categories. The first category assessed the web application itself, while the other category of utilities tested the database manually. Pricing in this category ranged as much as the overall function. With products that started below $1,000 to products which began at over $36,000. The range was truly surprising.

How we tested

We tested the applications by installing the utility on a Windows XP professional machine with an AMD 64-bit 4.0 Ghz processor,

1 GB of RAM and 100 GB hard drive. Next, we ran the utility against a small php base website with several small vulnerabilities. The website used custom error pages, which can throw off many of the spider features of application scanners by re-directing all bad web requests back to the site’s home page. This is a common first step in securing many web servers and is deployed by most major organizations. For a utility in the review to interpret the results correctly, the crawler had to distinguish between the returned custom error page of 302 — page moved as opposed to a 200 message for page found. Not all scanners were able to make this distinction.

All products were scored on ease of use, number of pages discovered, if vulnerabilities were sorted by class of vulnerability, an ability to report false positives to the manufacturer, the number of false positives found, the time the scan took to complete, the number of vulnerabilities uncovered, the types of reports offered, if remediation steps were included with the report, and if the product uninstalled cleanly. The key criteria for each product can be found in the overview matrix.

- Mike Stephenson contributed to this Group Test.

All Products In This Group Test