Risk and policy management have, traditionally, been treated as separate categories by us. However, as the years have passed, we have noted a convergence between these two types of products. This year that became quite apparent. We saw fewer pure-play risk or policy management tools. The two have converged such that managing policy is taken in the context of managing risk. To us that makes perfect sense. We looked for several things this year. First we wanted to see an automated workflow system that walked the user through the analysis and reporting process. We also wanted to see auto-discovery of assets. Finally, we wanted to see some sort of ticketing system, with internal or third-party integration.

We saw three general types of products this month. There were products that attacked the risk/policy management challenges with all guns blazing. They covered every imaginable function needed to determine risk within the enterprise and then applied policy changes to effect remediation.

The next group was far more specialized. These tools focused on some specific aspect of the enterprise, usually firewall or internet-working device management. These tools applied policies to the management and tuning of firewalls, routers, switches and the like.

Finally, we had a few tools that focused on marrying policy to practice but required the administrator to read the reports and make the changes. Those were fewer than last year but they appeared to stand still in the face of far more sophisticated tools. Still, we do not offer “shoot-outs” or “bake-offs,” so, as usual, we rate these products on their own merits in context with what they promise. We also look at industry expectations and factor that into our evaluation. In this case we were looking closely at functionality.

Another trend that we saw beginning to emerge last year was the application of next-generation techniques, such as machine learning and application of external threat feeds. In this regard we were a bit disappointed. The use of external threat feeds was not as prevalent as we would have expected or liked.

Risk is a combination of threats, vulnerabilities and impacts. You add in remediation – that implements countermeasures to reduce the impact by reducing threats or remediating vulnerabilities – and you have a complete risk picture. There is a tendency to characterize risk one dimensionally, as threats or vulnerabilities, or, in some cases, impacts (the risk is that the database will be corrupted – no reference to the threat that corrupted it or the vulnerability that allowed the threat to be successful). Unfortunately, a few – very few, thankfully – of these products still maintain that one-dimensional focus.

Addressing the use of threat feeds, we really don’t see how you can get a credible risk picture without taking the global threatscape into account. If you don’t plan to do that manually – you shouldn’t, of course – you need to automate consumption of threat feeds. The optimal way to do that is with STIX, but we saw almost none of that this time.

The other area where we were disappointed was asset discovery. With the advent of the virtual environment, servers and other devices spring up like weeds. That by itself poses risks. Having virtual servers that were spun up for some purpose and not decommissioned when the purpose was over is a very common problem. That problem exposes these obsolete servers – that may or may not have been well secured as can be the case of development or test platforms – to threats against their inherent vulnerabilities (unused or weakly passworded accounts, for example), especially if the server is internet-facing. Asset discovery should unmask these hidden servers, but performing discovery manually is a non-starter with most organizations, especially large ones. Ten-thousand servers are pretty hard to enumerate manually. And, with virtualization, when you are finished your enumeration won’t be accurate anyway.

We strongly believe that some form of auto-discovery is an absolute requirement for this type of product. That could be inherent in the product or it could be through integration with a vulnerability assessment too. If the tool uses integration, though, the discovery needs to be integrated tightly enough that it can provide data for the risk assessment. Unfortunately, not even good integrations usually allow you to weight the assets based on criticality, sensitivity or some other criteria. Without that weighting, you simply do not have a credible way to measure risk.

Finally, we are suspicious of any risk measuring system that attempts to quantify risk beyond a comparison with other assets being risk-rated. Regardless of what you might think, there is not a really credible way to quantify cyber risk. And, we contend, it doesn’t really matter. Cyber risk is only one element of the total risk picture of the organization, so if you do not take cyber risk in that context you will miss a huge piece of the picture. That said, there are several accepted ways to look at the overall risk to the organization. Cyber risk needs to play into that.

But the bottom line – however you measure risk – is that it is measured against compliance with policy and there must be a way to mitigate or remediate those identified risks and, in a closed loop, circle back around to test and validate your remediation or mitigation.