MITRE Corporation headquarters in McLean, Virginia. (Antony-22, CC BY-SA 4.0 https://creativecommons.org/licenses/by-sa/4.0, via Wikimedia Commons)

A recent study of 10 organizations found that, on average, rules and policies tied to security information and event management solutions, or SIEM, cover only 16 percent of the tactics and techniques listed in the MITRE ATT&CK framework.

Often considered a core component of security operations, SIEM solutions aggregate log data from various network devices and services and analyze them to detect threats.

Meanwhile, the MITRE ATT&CK framework is considered one of the preeminent global repositories of attack methodologies used by major threat actors. One might therefore believe that SIEMs and the MITRE ATT&CK framework would align more often than not. But that’s not the case according to a new report from CardinalOps, which says the company’s findings demonstrate the “truly poor efficacy of the average SIEM deployment.”

For the study, CardinalOps looked at the SIEM practices of 10 of its customers, all but one of which are multi-billion dollar multinational corporations. SIEM technologies vary among these customers, including solutions from vendors such as Splunk, IBM, Qradar and SumoLogic.

In the company’s corresponding report, author Yair Manor, chief technology officer of CardinalOps, states that SIEMs’ coverage of the ATT&CK framework remains, “in practice, far below what organizations expect and far below what the SIEM and detection tools can provide. This results in a chasm between the security SIEM users “assume they have and the actual security they get in practice.”

This is not the first research-based report focusing on such challenges. A September 2020 McAfee and the UC Berkeley report revealed that 45% of polled organizations said they suffered from lack of interoperability with security products while using ATT&CK. Additionally, 43% said they have experienced difficulty mapping event data to known tactics and techniques, while 36% said they receive too many false positives from their SIEMs.

The report also said some organizations do not use the ATT&CK framework because it does not prioritize any adversary techniques, and no weights are assigned.

Experts say much of the responsibility of making sure that SIEMs reap maximum benefits from the ATT&CK framework falls on users’ ability to understand their own environments and prioritize which threats pose the greatest risk, so they can create rules that best protect themselves. But that requires time and effort.

“The most challenging problem for an organization is its desire to use any toolset, application, system, or framework as an instant panacea that’s going to solve all of its problems,” said Kim Jones, an information security expert and professor of practice at Arizona State University. “Taking an existing detection posture and attempting to drop a framework on top of without doing your own analysis and prioritization or evaluating your tool appropriateness for the work is shortcutting the effort.”

Jones suspects that unreasonably high expectations are being placed on SIEMs “to make [up] for failures or gaps in detection engine configurations” and the inability of organizations to optimally design their defenses around threat intelligence they receive. “Blaming the tools or the framework for that problem is disingenuous,” he added. “In my mind it’s like blaming the screw for being faulty and cheap because the head stripped when in reality the screw failed because you used a pair of pliers instead of a screwdriver to install it.”

Using a framework in ways that weren’t intended?

Ryan Kovar, distinguished security strategist at Splunk, said he’s not surprised that so little of the ATT&CK framework is covered by most SIEMs’ policies. But, then again, integrating with SIEMs was “not originally what MITRE ATT&CK was intended for.”

“Many people don’t realize that the MITRE ATT&CK framework was not initially designed to solve SIEM problems,” Kovar said. Rather, it is a “cognitive thought model” that is “designed to help threat intelligence experts methodically map adversary behavior to empirical findings.”

Indeed, some TTPs in the framework can’t even be addressed via a SIEM solution, said Kovar. For instance, under the framework’s reconnaissance and resource development categories, there are 16 techniques “that are almost impossible to write SIEM alerts for.”

Mapping to the framework is often not even SIEM buyers’ main motivation, Kovar noted. Instead, “they are looking for tools that can scale, collect disparate data, allow them to alert on known bad things, and then review their data for information when they are attacked in novel methods, like SolarWinds.” Still, by gaining a clearer understanding of the framework’s true purpose, users can leverage it through SIEMs, he added. 

CardinalOps advisor Anton Chuvakin, a security solution strategy professional at Google Cloud, and former research vice president and distinguished analyst at Gartner, agreed that a SIEM “is not meant to cover the entire ATT&CK [framework] – as it contains a fair bit of deep endpoint attack indicators that may not end up in logs,” and an endpoint detection and response solution is needed for those.

Even so, there is a large disparity between SIEMs rules covering the entire framework, and covering a mere 16%, leaving 84% of malicious techniques disregarded.

In an interview, Manor at CardinalOps told SC Media that the struggle for organizations is not in adapting rules that effectively match the framework’s contents. It’s that there are simply not enough rules and policies instituted in the first place.

Manor offered several theories behind why this is the case – and chief among them is companies’ lack of visibility into the efficacy and comprehensiveness of their malicious TTP coverage.

“Additionally, the complexity of managing and operating the SIEM often creates a glass ceiling, limiting the coverage that can be achieved,” he said. And thirdly, “with the ever-evolving IT landscape and threat landscape, security engineers are often unaware of what needs to be done to address the latest use cases and threats.”

Adam Pennington, MITRE ATT&CK lead at the non-profit Mitre Corporation, said another issue facing SIEM users is that detections of known ATT&CK techniques requires additional investment beyond just SIEM technology.

“Some of the biggest challenges we see are implementing sensors to gather the appropriate data sources, and bringing them all together before the organization handles analyzing the data sources into ATT&CK techniques,” said Pennington. “Increasing these data sources and the analytics around them will naturally increase the ATT&CK techniques addressed.”

Risk assessment and threat prioritization are key

Experts said there are steps organizations can take to make sure SIEMs are getting the most out of the MITRE ATT&CK framework.

For starters, Jones said companies must take an “objective, unbiased look at [their] protection posture.” To do this, they must first identify the telltale indicators they would need to look out for in order to detect TTPs, and then determine the devices, systems and applications needed to spot them. Then, organizations need to ascertain if these toolsets are able to send alerts into the SIEM when these detections happen. “I would contend that folks who are attempting to implement MITRE ATT&CK might not be doing this rigorous analysis,” said Jones.

Next comes threat prioritization, which requires an understanding of which SIEM rules will prove most crucial to protecting your network’s assets, so you can custom-build your policies according your own environment’s biggest risks.

Pennington at the Mitre Corporation said the CardinalOps report’s finding that SIEMs cover on average only 16 percent of the framework is “lower than what we’ve generally seen in our own experience.” But even so, he acknowledged that no SIEM can cover 100 percent of the known threats out in the wild. For that reason, threat prioritization is necessary.

“We’ve recommended against focusing on complete coverage of ATT&CK in the past, and continue to do so,” said Pennington.”We also recommended that organizations chose a way to prioritize what pieces of ATT&CK to implement as they’re getting started rather than attempting to implement everything all at once.”

There are attack scenarios that are more or less probable or impactful for an organization, explained Jones. “Focusing on those scenarios and building from there is an appropriate risk-balanced approach to implementation. I’d rather know that I can detect and alert on 99.999% of the TTPs associated with my most probable or impactful scenarios than measure against the totality of the framework.”

“You know who is out to get you better than anybody,” said Kovar. “Sure, there are commodity threats that affect everybody but beyond that, the end user informed with their threat intel sources must do the ranking.”

Kovar recommended risk-based alerting, by which SIEMs will alert SOC analysts to a potential threat, but only if the anomalous event matches multiple SIEM rules, which makes it a high-risk incident – one that’s unlikely to be a false positive. “Users end up having many different rules for firing and detecting, but the risk analysis framework provides the ability to identify actions that raise the risk profile of individuals or assets, significantly reducing alert fatigue,” said Kovar.

But while the primary responsibility lies heavily on the user organization, vendor partners can also play a role in improving SIEM’s coverage of the ATT&CK framework.

“Splunk has said in the past that vendors can still do better at integrating MITRE ATT&CK into their tools,” said Kovar. “I believe that every software vendor’s goal should be to make the jobs of their customers easier… As an industry, we should work to have better coverage across the matrix but recognize that we will never have a checkbox on every technique.”

Pennington said Mitre is also taking steps to lower the ATT&CK framework’s burden on SIEMs, noting the creation of and ongoing improvements to the organization’s Cyber Analytics Repository, “which has analytics that organizations can use in their SIEM to detect ATT&CK techniques.”

“We are also currently working on improving data sources in ATT&CK to better describe the information that organizations need to gather for a given technique,” Pennington continued. “Leveraging these resources can help various parties increase the percent of MITRE ATT&CK-listed techniques that are covered.”

Indeed, Kovar at Splunk said Mitre is “taking strides to help organizations operationalize the data,” through such initiatives as MITRE Engenuity – a foundation that collaborates with private industry to accelerate innovation. One of Engenuity’s offering is ATT&CK Evaluations, which assesses vendors’ ability to defend against known adversary techniques, and openly publishes the results for industry end users to review.

Pennington said Mitre is also trying to help users with their threat prioritization but “working to better understand which techniques are most commonly used by adversaries.” To that end, the Mitre Corporation initiated a pilot intel-gathering program called ATT&CK Sitings, through which members of the ATT&CK user community can report sightings of techniques to each other.

Still, the last word on prioritization must come from the user organization.

“Prioritization is unique to each organization,” said Pennington. “We’ve tried to create strategies for people to have different methods for prioritizing ATT&CK techniques, but we’ll never be able to tell you what technique is most important to your organization.”