The MITRE Engenuity ATT&CK evaluations are a transparent, yearly assessment of leading enterprise endpoint-protection solutions as tested against known threats. The level of detail provided by their results not only demonstrates the efficiency of endpoint solutions but provides any defending team with deep knowledge of how to protect their own organization according to the MITRE ATT&CK framework.
How Engenuity Evals are different from traditional antivirus reviews
The underlying model for most evaluations of security products is the antivirus review. But generally, an AV review will tell you only whether a product stopped a threat, or perhaps whether the threat was blocked instead of neutralized.
Such reviews may be useful for consumer antivirus products that defend home PCs against internet-based threats, but enterprise endpoint-protection products require more detailed evaluations.
Antivirus reviews "may potentially help evaluate a protection product, like a traditional AV from a traditional AV vendor," said Shyue Hong Chuang, product manager for Cisco Secure Endpoint. "But when it comes to the stuff that got past, what did your product tell me? It's the MITRE evaluation, it's the AV-Comparatives EPR [Endpoint Prevention and Response] test that gives a bit more visibility (across the attack kill chain)."
The MITRE evaluation Chuang refers to is the MITRE Engenuity ATT&CK evaluations, or Evals for short, which MITRE has run almost every year since 2018. The Evals document every step in the kill chain of a well-known, real-life, sophisticated attack against a Microsoft Azure instance protected by one of the endpoint security products being tested.
For example, in the latest round of Evals, conducted in late 2021 with results released in March 2022, 30 different security vendors submitted their products for testing, including Cisco, CrowdStrike, McAfee, Microsoft and Symantec.
Each product faced two well-known adversaries: first, the Wizard Spider criminal group that has used the BazarLoader, Conti, Emotet, Ryuk and Trickbot malware against enterprise targets; and second, the Russian state-sponsored Sandworm group, notorious for attacks upon the Ukrainian energy sector as well as the NotPetya wiper malware attack in 2017.
The advantages gained from more visibility
Because the MITRE ATT&CK framework is well understood among security practitioners, the level of detail provided by the Engenuity evaluation results is a treasure trove of information about how each tested endpoint product fares at each step of the kill chain. MITRE posts the results publicly and freely, and while the documentation can be a bit hard to decipher, there's no better way for organizations considering an endpoint solution to assess how well a product may be suited for them.
"Defenders use Evals to make better informed decisions on leveraging the products that secure their networks," states the MITRE Engenuity ATT&CK evaluations website. "Each vendor evaluation is independently assessed on their unique approach to threat detection. Evaluation rounds are not a competitive analysis; they do not showcase scores, rankings, or ratings and are transparent and openly published."
Dr. Joel Fulton, co-founder and CEO of Lucidum, an asset discovery company, pointed out that the MITRE ATT&CK framework also helps CISOs better communicate their needs to executives.
"Most CISOs will ask for investments and increases in budget to respond to either current events or longstanding security concerns, but they don't have sufficient data points to support the ask," Fulton told CyberRisk Alliance. "By using the MITRE ATT&CK framework as a guide for these conversations, CISOs will be able to effectively explain the severity of threats and the actions to mitigate them while allowing CIOs to be active participants."
How the Engenuity results can help protect your organization
But it's not only those enterprises looking for new endpoint-protection software that can benefit from the MITRE Engenuity ATT&CK results. Because the evaluation results are so granular, skilled defense teams can use them to pinpoint weaknesses in their own security posture and adjust their strategies accordingly.
"Here is a true-to-form attack in sequence with the kill chain, the way that Sandworm or Wizard Spider actually facilitated these opportunities," said Adam Tomeo, senior product marketing manager for Cisco Secure Endpoint. "At this point, regardless of where you can potentially stop it on the kill chain, you can leverage each one of these sub-steps to help strengthen your security posture in your organization."
Both these threat actors are still very active, as are the attackers in the previous rounds of MITRE Engenuity ATT&CK evaluations, which include the Carbanak and Fin7 criminal gangs and the Russian state-sponsored Cozy Bear or APT 29, the latter believed to be behind the devastating SolarWinds supply-chain compromise of 2020.
"By viewing the MITRE ATT&CK framework as a 'board game' or checklist, security teams can thoroughly understand where their vulnerabilities lie and take the appropriate action to prevent attacks," said Fulton.
To Eric Howard, lead technical engineer for Cisco Secure Endpoint, the MITRE Engenuity ATT&CK evaluations provide "the ability to have a common language between both those that know how to test an environment and those that are tasked with defending against the things that are thrown at an environment."
"Red and blue teams can speak the same language," Howard added, "reversing the power of the Babel effect so that we can get to the same goal."
