Critical Infrastructure Security, Incident Response, Network Security, TDR, Vulnerability Management

Measuring success: Performance metrics

Gauging performance does not necessarily result in enhanced security, but quantifying risk is still vital, reports Stephen Lawton.

These days, it seems everything has a metric. Some questions with quantitative results can be answered easily – such as how a network is performing – and key performance indicators (KPI) can be useful in that environment, says Gartner research director Anton Chuvakin. 

KPIs are a strategy to measure the effectiveness of an enterprise, a division or individual employees. However, there are other questions around basic data security that cannot be so easily assuaged. For instance, a company that spends a lot of money on security might identify attacks, but that does not mean it is well protected, Chuvakin says. 

It could mean that even with a substantial security budget, the organization's security team is not identifying and defending against the attacks. Likewise, a company with a small security budget might have an effective security plan in place that protects its corporate assets. 

Clearly, Chuvakin says, a security budget alone is no real indicator of how well an entity can protect its intellectual property. He maintains that because each company has its own security profile, there can be no standard set of KPIs to determine security.

A whitepaper produced by U.K.-based Iris Accountancy Solutions, a software and services company, says KPIs vary by department, so it is important for each to outline and work to its own set of measurements. The paper describes them as objectives or goals that can be used to measure the performance of each department. 

KPIs generally have five qualities: they are specific, measureable, achievable, realistic and timely, according to Iris. They need to quantify issues that will make a difference, and are thus focused on areas for which the department or employee can have an impact. Creating a goal that is outside the scope of an individual or department is unachievable, and should not be considered a KPI.

As with so many aspects of managing information security, administrating data overload can be an issue, the whitepaper explains. Understanding what needs to be analyzed and putting that data into a comprehensible format is critical. Because Iris has a large number of financial services clients, a popular format is the spreadsheet, which can incorporate both current and historical data.

This leads to the inevitable question: Can one create KPIs for security? The short answer is: yes and no, Chuvakin says. If one sets their standards too high, they will never reach them. Too many variables and an ever-changing security landscape mean that KPIs that are too general cannot be met.

On the other hand, if the KPIs are set at the tactical, instead of a more strategic, level, it is possible to meet those levels. The result, however, might not meet the real security needs for the network, he says. Today, setting network security KPIs is an inexact science with too many variables, Chuvakin says. “There is no silver bullet.”

With a given amount of money allotted for security, a company may spend that on specialty products based on price, he says. However, if a company takes a more holistic view of its security profile, they might find more efficient ways to protect their network – with the budget being built to meet the network's needs rather than those of a preset spending plan.

It all comes down to determining a company's risk profile, experts agree. Once a company understands its
risks, it can start to compile the indicators to determine where potential vulnerabilities lie.

Typically, an organization's measurable security level is related to the amount of negative incidents detected, such as the number of viruses inoculated or number of breaches halted, says Shawn Chaput, executive consultant at Privity Systems, a security consultancy in Vancouver, British Columbia. 

“This approach tends to exclude the nasty stuff, such as unidentified viruses, like Flame, which was in the wild for [an estimated] two years before it was discovered,” he says. “The problem with this approach is that it tends to reward organizations for not having attributes that respond to its security controls.” 

Chaput says many companies do not know that they already have been compromised because they do not have the tools or expertise to recognize what is happening on their networks. “Valuable security measurements should likely be tied to the typical C-I-A triad: confidentiality, integrity and availability,” he says. “Whatever systems are in place to protect those elements should have metrics which are measured and benchmarked. This isn't as easy as you'd expect.”

A component to determining what is a sufficient security KPI entails defining what is and what is not a threat, Chaput says. Not every vulnerability deserves the same level of risk mitigation, just as every piece of data has its own value. 

“Ultimately, an organization needs to identify its critical assets and apply security controls to those systems commensurate with the risk,” he says. “[For example,] if an organization's revenue solely relies on an e-commerce system, that e-commerce system would be mission critical relative to the internal SharePoint site.”

This also means that vulnerabilities that may affect that system should be treated with higher urgency than the same flaws that may affect the SharePoint site, he adds. “Effectively, this goes back to risk-ranking methodologies with risk as a function of impact – in this case, related to criticality; probability – how likely is this to happen; and some other elements depending on the risk-ranking method one chooses,” he says. 

KPIs need to be tied to the strategic objectives and the key risks and challenges of any business, says Bernard Marr, a U.K.-based management consultant and author. “It is then important to identify the questions and information needs in relation to each of the strategic objectives and challenges.”

Once the questions are clear, he says, indicators can be developed to help answer them. “Like with most strategic objectives, they are unique to any given business and, therefore, the KPIs have to be uniquely designed to answer the questions of any given business – same for security,” Marr says.

Torsten George, vice president of worldwide marketing and products at Agiliance, a risk management vendor, agrees. “A vulnerability in and of itself does not represent any risk,” he says. A risk requires a threat plus a vulnerability. “If the threat is not capable of exploiting the vulnerability or cannot reach the vulnerability, there is no risk,” he says. Once an organization has determined that the combination of threats and vulnerabilities poses a risk for the business, then it must decide how to mitigate the risk. 

Get daily email updates

SC Media's daily must-read of the most current and pressing daily news

By clicking the Subscribe button below, you agree to SC Media Terms and Conditions and Privacy Policy.