This year, when we looked at vulnerability assessment tools, we found something very interesting: Vendors are starting to decide where their products fit in the marketplace. On the surface, that sounds like a no-brainer, but, as it happens, there have been some subtle - and a few not so subtle - evolutions in the world of vulnerability assessment. We saw most of them this month.
One important question that is beginning to appear is: "What do we really mean by vulnerability assessment?" As I mentioned in the opening column this month, there is an emerging trend toward vulnerability management and I predict that we will begin to see the vulnerability assessment category - pure-play vulnerability assessment - give way to vulnerability management. I think that this is upon us for a couple of reasons.
First, regulatory requirements are forcing ever more detailed reporting. Second, managing such things as patching, remediation, etc., is becoming more and more difficult, especially for very large enterprises. Since both of these issues address vulnerabilities directly, we are beginning to see the emergence of novel ways to manage vulnerabilities and the resulting follow-on.
This question actually raises several more. For example, does vulnerability assessment include penetration testing? If it does, what is the role of each and how do they work together? That may sound like an easy one to answer - and, in my view it is - but apparently it's not. Only three of our products this month include both capabilities natively (one more uses a Metasploit plug-in), but many of the rest don't do a good job of null scanning (scanning without any credentials). But more on that one in a moment.
Depending on your vulnerability testing philosophy - and I like the term "vulnerability testing" for its comprehensiveness - you probably want to identify a broad range of vulnerabilities first and then test them for exploitability. That is fine for simple, one-off vulnerabilities, but that is not the way of today's world. Today the buzz terms are "chained exploits" or "blended exploits."
These introduce a much more difficult environment because part of the "chain" or the "blend" includes malware, such as trojans, bots and, especially, root kits. These are likely to operate from within the enterprise, so the concept of reachability becomes far more complicated. Now we care as much about how the root kit got into the enterprise as we do about what it can do once it is there. That means that vulnerability testing must work in concert with threat management and must be more comprehensive.
That introduces a new challenge: false positives (and negatives). I have seen web application vulnerability scanners that deliver upwards of 10,000 vulnerabilities on a modest web farm simply because they record every vulnerability from every angle. So a single vulnerability buried three levels deep on a web application shows up multiple times because there are multiple ways to access that page. This makes remediation quite tricky.
Back to the issue of pen testing. In my view, this is the only way to test thoroughly, but it cannot be used in isolation because it is not, by its nature, comprehensive. Vulnerability assessment is comprehensive - too comprehensive in some cases, where it reports numerous false positives or redundant results. Using pen testing as a quality process on top of vulnerability assessment is a very good approach but, with a couple of exceptions, that means using multiple tools.
Before you say that pen testing really isn't necessary, let's revisit the issue of null scanning. A null scan is the old way we used to run scanners. These scanners simply look at those devices they can touch and the vulnerabilities generally are in the interface to the outside, in the case of some applications, or the communications stack on the platform. Today, those vulnerabilities are getting close to being under control (except in web applications, of course). So how does the tester dig deeper?
The answer is to pretend to be an authorized user. I have a problem with this because that is not the way attackers attack (unless, of course, the attacker has inside information that allows them to pass through authentication). So, in my view, there are real limitations on vulnerability scanners that depend on having credentials before they can be effective.
Vendors actually seem to recognize this limitation because most pure play scanner vendors asked us specifically if we were going to limit our testing to null scans. One vendor actually refused to participate if we were going to limit our testing. This, in my view, is an admission of inadequacy. I expect some push-back on this from our vendors and I will be glad to respond in our letter section. Pen testing can help address this problem by exploiting vulnerabilities that bypass controls.
So, obviously, all of this points to a single "how to buy" caveat: Select your vulnerability testing tool(s) to meet your particular testing requirements and assess those requirements as holistically as you can. Our testing did use both null scans and credentialed scans, penetration testing when we could, and was conducted against our target field of a variety of operating environments with known vulnerabilities.
One important question that is beginning to appear is: "What do we really mean by vulnerability assessment?" As I mentioned in the opening column this month, there is an emerging trend toward vulnerability management and I predict that we will begin to see the vulnerability assessment category - pure-play vulnerability assessment - give way to vulnerability management. I think that this is upon us for a couple of reasons.
First, regulatory requirements are forcing ever more detailed reporting. Second, managing such things as patching, remediation, etc., is becoming more and more difficult, especially for very large enterprises. Since both of these issues address vulnerabilities directly, we are beginning to see the emergence of novel ways to manage vulnerabilities and the resulting follow-on.
This question actually raises several more. For example, does vulnerability assessment include penetration testing? If it does, what is the role of each and how do they work together? That may sound like an easy one to answer - and, in my view it is - but apparently it's not. Only three of our products this month include both capabilities natively (one more uses a Metasploit plug-in), but many of the rest don't do a good job of null scanning (scanning without any credentials). But more on that one in a moment.
Depending on your vulnerability testing philosophy - and I like the term "vulnerability testing" for its comprehensiveness - you probably want to identify a broad range of vulnerabilities first and then test them for exploitability. That is fine for simple, one-off vulnerabilities, but that is not the way of today's world. Today the buzz terms are "chained exploits" or "blended exploits."
These introduce a much more difficult environment because part of the "chain" or the "blend" includes malware, such as trojans, bots and, especially, root kits. These are likely to operate from within the enterprise, so the concept of reachability becomes far more complicated. Now we care as much about how the root kit got into the enterprise as we do about what it can do once it is there. That means that vulnerability testing must work in concert with threat management and must be more comprehensive.
That introduces a new challenge: false positives (and negatives). I have seen web application vulnerability scanners that deliver upwards of 10,000 vulnerabilities on a modest web farm simply because they record every vulnerability from every angle. So a single vulnerability buried three levels deep on a web application shows up multiple times because there are multiple ways to access that page. This makes remediation quite tricky.
Back to the issue of pen testing. In my view, this is the only way to test thoroughly, but it cannot be used in isolation because it is not, by its nature, comprehensive. Vulnerability assessment is comprehensive - too comprehensive in some cases, where it reports numerous false positives or redundant results. Using pen testing as a quality process on top of vulnerability assessment is a very good approach but, with a couple of exceptions, that means using multiple tools.
Before you say that pen testing really isn't necessary, let's revisit the issue of null scanning. A null scan is the old way we used to run scanners. These scanners simply look at those devices they can touch and the vulnerabilities generally are in the interface to the outside, in the case of some applications, or the communications stack on the platform. Today, those vulnerabilities are getting close to being under control (except in web applications, of course). So how does the tester dig deeper?
The answer is to pretend to be an authorized user. I have a problem with this because that is not the way attackers attack (unless, of course, the attacker has inside information that allows them to pass through authentication). So, in my view, there are real limitations on vulnerability scanners that depend on having credentials before they can be effective.
Vendors actually seem to recognize this limitation because most pure play scanner vendors asked us specifically if we were going to limit our testing to null scans. One vendor actually refused to participate if we were going to limit our testing. This, in my view, is an admission of inadequacy. I expect some push-back on this from our vendors and I will be glad to respond in our letter section. Pen testing can help address this problem by exploiting vulnerabilities that bypass controls.
So, obviously, all of this points to a single "how to buy" caveat: Select your vulnerability testing tool(s) to meet your particular testing requirements and assess those requirements as holistically as you can. Our testing did use both null scans and credentialed scans, penetration testing when we could, and was conducted against our target field of a variety of operating environments with known vulnerabilities.
