Compliance Management

The hard sell: which key performance indicators to use in reports

When security pros begin to look for industry-wide standards for reporting metrics, they generally find a simple answer — there isn't one. The metrics that security professionals use to define key performance indicators are usually as unique as the companies for which they work. When framing security needs, goals and accomplishments of an organization, security officers keep the uniqueness of a company in mind, including business goals, risk management strategies and maturity of the security structure, says Khalid Kark, senior analyst at Forrester Research.

"I started off a year and a half ago working on security metrics and trying to figure out if I could narrow down the metrics to ones that everyone can use across the board," says Kark. "And what I quickly found is that there is no uniformity. The metrics depend on the size of a company and the maturity of the security organization. There are lots and lots of variations of what might work well in a particular instance."

"One truth across the board, however, is that companies can't just throw money at the problem. Effective C-level executives are familiar with their own company's risk management strategy — including compliance concerns, assets and types of threats it faces from the outside world and within its own cubicles. Companies using repeatable statistics to show improvement in targeted areas — or the rise of new threats — have superior metrics to those reflecting money spent, says Max Caceras, director of product management at Core Security.

"I would say that there is not a general consensus on what type of metrics to show. Some folks will report operational metrics, which don't necessarily report how security has improved, but just say that this and that were deployed this month," he says. "They don't really mean anything in terms of security. Some organizations are further down the line, and they understand that measurement is the key thing in terms of reporting metrics."

Testing for results

One way to measure the effectiveness of a corporate security structure is to make testing a routine. To ensure security, some executives take a page from the book of military planners by creating and running real-time war-games exercises simulating mass attacks on their networks. No matter what size an organization is, or what industry vertical it resides in, penetration testing is an effective method of reporting what strides a company has made and where additional attention needs to be paid, says Karim Toubba, vice president of marketing and business development at Red Seal.

"Most CSOs say that the only way they can rest at night is to say, ‘I have vulnerabilities and I have issues.' I constantly measure those vulnerabilities and know which risks will be acceptable to the business and what I have to get my team to fix as a priority," says Toubba. "It's a combination of technology and policy. One of the things that we do is to start to give some business context to the vulnerability, for instance, business values for the host."

Many organizations test for easy targets in their system beyond software vulnerabilities and areas of non-compliance. Checking the security awareness of employees is another method used by corporations to check just how secure their infrastructure — including their workforce — really is. Attackers, shown in study after study to be increasingly profit-driven, are certainly mindful that it may be easier to glean valuable information from an employee rather than a network, says Kris Kendall, principal engineer, Mandiant.

"Another thing is to build the security awareness of your people. The low-hanging fruit for a hacker has gone from, ‘I'm going to attack routers and servers,' to ‘I'm going to attack your people,'" Kendall says. "One service that we see a fair amount of is social engineering exercises. You actually get a percentage — 25 percent of your people were willing to give up their password over the phone. Generally, you do phishing and phone calls in those types of exercises. It's super easy to use as a metric."

Measuring compliance

Boardroom executives and CEOs are also well aware of who Sarbanes-Oxley and other compliance regulations were written to influence: themselves. With SOX threatening executives with hefty fines or jail time for non-compliance, corporate officers want to know how much work has to be done to gain compliance, says Malcolm Palmer, director of product management at Cybertrust, which was acquired by Verizon Business in July.

"In terms of specific metrics, the key is whether they have a security plan and are they checking the implementation of that plan," says Palmer. "They're basically checking that baseline set of controls and doing that over an entire enterprise. That's very important because it lets people know how they stand up against the baseline."

Executives also want to ensure that security staffers have an eye on the future when ensuring company compliance, knowing that likely it is only a matter of time until Congress — or legislatures in states where the company operates — pass legislation demanding adherence to new standards. Demonstrable metrics showing adherence to existing control frameworks are a step toward future compliance goals, says Kark.

"I've been talking to a lot of CSOs who are in the process of going through individual compliance programs, or have instituted programs to get compliant, and they realize that tomorrow another regulation is going to come along. They say, ‘Let's not worry as much about individual regulations.

Let's worry about having a good control framework, and let's develop that framework and then worry about regulations to comply with,'" he says.

The key language

When reporting key performance indicators, IT professionals can make the best case for their budgets by placing the statistics in terms that the board pays the most attention to.

Security professionals should be careful to tie their reports to overall company strategy and show exactly how security investment — and the progress provided by it — affects the company's bottom line. Showing where past incidents have cost money can be a helpful measuring stick, says Kendall.

"The best indicators are the number of real incidents that have cost us money, or to say, for example, ‘We had zero incidents last year that cost us money.' So you have to be able to measure the real things first, the number of confirmed incidents, virus outbreaks, how much data we lost. Those are the easiest metrics to measure, but the problem is that the goal is to make sure none of those happen at all," he says.

Making the case

"One of the comments I have about the process is that the metrics are meaningless unless you have well-thought-out goals and stakeholders, and I would always couch my metrics in terms of assets and the policies guarding those assets. But if you don't know what your assets are, you can't protect that," Kendall says.

Put into simple terms, security professionals charged with reporting to the corporate heads are at their best when they speak the board's language — money, says Palmer.

"You really want to deal with direct cost. You want to be able to show X transactions per second and per hour, and attach security to how they allow you to put that into your business plan and the savings that were done by implementing that process," he says.


Operational metrics

Collect data automatically: Use tools that collect granular data but output that information to a central location. There, CSOs and CISOs have all the information needed at their fingertips.

Use repeatable measurements: Don't report on anomaly cases. Metrics are most useful when they measure occurrences over a period of time to analyze behavior and predict future patterns.

Collect data automatically: Use tools that collect granular data but output that information to a central location. There, CSOs and CISOs have all the information needed at their fingertips.

Use repeatable measurements: Don't report on anomaly cases. Metrics are most useful when they measure occurrences over a period of time to analyze behavior and predict future patterns.

Keep in mind unintended consequences: Ensure that newly instituted metrics will not influence user behavior by swaying them to use easy fixes for more favorable statistics.

Define thresholds: IT staff and C-level employees should set percentage-based thresholds for the deployment of solutions and other security strategies, such as 70 percent of machines in a corporation should receive spam filtering software. If those numbers are violated, management should immediately be notified.

Create metric ownership: For each metric, a staff member should be assigned to collect the information, analyze it and report on it to a superior. That employee should periodically check on the data.

Make sure the metric has an audience: Ensure that for each piece of information collected, an employee has been assigned to keep track of and analyze that data. Collecting metrics on information that is ignored for long periods of time is a waste.

Source: "Are we secure yet?," by Khalid Kark, with Laurie Orlov, Paul Stamp, Samuel Bright, Forrester Research

Get daily email updates

SC Media's daily must-read of the most current and pressing daily news

By clicking the Subscribe button below, you agree to SC Media Terms and Conditions and Privacy Policy.