This category is my personal favorite because it touches on the types of tools we use in the lab. These tools cover a lot of territory – from vulnerability analysis to forensic tools. Even within a category, such as forensic tools, we see some splits that we have needed to extract as product types in themselves. For example, many, including me, view SIEM as a network forensic tool and, as such, we might put it in the forensics subcategory. Not so here, though. Here we give it its own subcategory.
The general rule for analysis and testing is that the category needs to provide some sort of analysis capability. In that sense, it is value-added for the network. In some cases, such as SIEM, the products in a subcategory sit on the network, quietly monitoring and analyzing, alerting when necessary, saving audit trails and providing some sort of analytical decision support system for human operators.
In some cases, the tool is applied on a case-by-case basis. Whatever the subcategory, we are looking for analysis or, simply, testing – sometimes both.
As with our other categories, this one is populated by old friends and new discoveries. We saw pretty much the same challenges here that we saw in the other categories, which surprised us a bit. I rather expected that this category would be unique and, perhaps, somewhat isolated from the run-of-the-mill challenges while confronting challenges of its own. There were a few unique ones.
First, complexities are an important challenge to testing and analysis. The reasons seem obvious: Complexity takes more eff ort to analyze and complexity is, well, complicated. This means that what works well and easily in a less complicated environment is much more difficult to apply, and results are harder to analyze in a large complex enterprise.
One way to have this is to distribute sensors the way an IDS/IPS does. At least one of our products does that. Another way is to preprocess the data before analyzing it. While that is a staple of some manual analysis techniques, it is much less common in the tools we looked at here.
At the end of the day, these tools have as their primary purpose telling us things we don't know about the enterprise, but need to know. And the companies that bring them to us are doing a very creative job of figuring out ways of doing exactly that.