Not only must security pros contend with ever-increasing attacks to their networks, they also must finagle the tool sets guarding their systems to make certain settings are as they should be, reports Greg Masters.
No wonder alert fatigue has become an unwelcome part of the mitigation process. With red lights constantly triggered – could be a legitimate intrusion, could be a false positive – IT security administrators, charged with keeping the data flowing without malware or any other pollutant getting into the operation, face a formidable obstacle.
At the top of the list is desensitization – with so many bells and whistles going off, how is one supposed to remain alert to what is truly necessary? The point is, security personnel can grow wary of the notices their equipment is throwing back at them, as the alarms go off so frequently that the humans monitoring the systems can only handle so much. In other words, so many of the notifications are set off by minor infractions that they lose urgency.
The results of an April 2017 study, "A Day in the Life of a Cyber Security Pro," an Enterprise Management Associates info brief written by David Monahan for Bay Dynamics, illustrates the challenge: Respondents identified that they have to deal with a large number of vulnerabilities in their organizations. On average, 10 vulnerabilities exist per system. In fact, nearly three-quarters of security teams stated they were overwhelmed by the volume of vulnerability maintenance work assigned to them.
And, when security teams were queried about contending with threat alerts, an even bigger percentage (79%) said they were overwhelmed by the volume.
One issue is that alerting systems, such as security incident and event management (SIEM) systems, often don't come equipped with the data required for security pros to make informed decisions, the EMA study found. "This creates a situation where too many alerts are created, with the highest priority then requiring additional work by analysts to make a proper reprioritization."
Those queried for the survey said they have to manually reprioritize over half of the threat alerts they receive. Obviously, this creates more work and adds considerably to the stress factor, the report said.
Kevin Reid (left), VP of national security and CIO at KeyLogic Systems, says alert fatigue is like the boy who cried wolf: "If there are too many similar alerts that end up being empty threats, eventually IT security teams will just ignore the warnings," he says.
A great example of alert fatigue resulting in a widespread attack is the Target breach, he points out. "Leading up to this attack, the security team was consistently seeing the same, empty malware alerts, so they grew numb to the notifications and ignored the warning when there was a real intrusion."
Another challenge Reid sees is the bulk of data, which, he predicts, is only going to continue increasing. "Even if an alert isn't received, security teams get a report of abnormalities within their system that need to be analyzed. However, as the amount of data grows, so do these reports, making them more difficult to examine for threats.
Security teams can have all the best tools available, but if they aren't being implemented correctly, their networks are still at risk, Reid says.
Alert fatigue is the threshold at which it becomes too difficult for a security analyst to recognize the important alerts from the stream of everything that they receive, says Maxine Holt, principal analyst at the Information Security Forum (ISF). "Analysts must review each alert to decide if it really is suspicious or another ‘false positive,' when they all appear similar at first."
When magnified by multiple systems/software delivering alerts, it quickly becomes apparent that there are so many alerts it is difficult to see the ‘true positives' among the false ones, Holt says. "Furthermore, some alerts require aggregation before the combination can be confirmed as a ‘true positive' and potential business impact assessed."
When a security analyst suspects a true positive there are a number of processes to follow to initiate an information security incident, Holt adds. "Some analysts are concerned about making mistakes if they report too many alerts, leading to a potentially costly mistake if a true positive isn't dealt with in a timely manner."
For Dan Lohrmann, CSO and chief strategist at Security Mentor, alert fatigue challenges can be grouped into the traditional buckets of people, process and technology: The people part of the pie involves the long hours doing the same role and functions, he says. Along with that comes improperly trained staff with not enough experience or not knowing how to use tools.
The team issue begins with weak partnerships, where communication is lacking to/from/between all levels of management. Caffeine or drugs are sometimes used to help overcome tiredness or provide more attention, but these ‘solutions' can sometimes lead to health issues, Lohrmann (left) says.
It all factors into becoming desensitized to important alerts causing an increase in response times or missing items.
With process issues, Lohrmann points to an improper distribution of workload and/or alerts, an improper categorization of specific alerts types, and problems with alert levels or classifications, such as too many high level alerts.
He says that the notification and escalation process is flawed. Proper help may not be available when needed or management may be called in too often – thus causing issues when real problems emerge.
As far as the technology piece, Lohrmann points to wrong tools (old legacy, too many alerts, not enough alerts, etc.), multiple tools that don't work together, and not a wide or specific enough view of data, threats, etc. That is, lacking is a national or global perspective that comes with Information Sharing & Analysis Centers (ISACs) and other global data trend information.
Tools today produce too many false positives or incidents for the system analysts to review, Lohrmann says. We use technology to process as many of these as possible without human intervention – such as looking at system logs and network alerts being reported to management consoles.
The problem for Sam McLane, head of security engineering at Arctic Wolf, is that many security operations operate under the 'work harder not smarter' mode, which, he says, is unsustainable.
"When you have too many alerts, the solution is not to work harder to get through them all. Eventually, your staff will be desensitized, and their diligence will wane."
Lenny Zeltser (left), vice president of products at Minerva, points out that the global shortage of IT security personnel results in many teams tasked with handling the alerts being understaffed and overworked. "This factor, combined with the overwhelming number of alerts that need to be handled on an ongoing basis, creates an imbalance. In turn, many important alerts go unnoticed or are disregarded even though they could be the indicator of an actual attack."
While most shops have tools to sense threats and alert security professionals – these alerts typically lack meaningful context to understand the impact or potential impact, says Druce MacFarlane, vice president of products and marketing for Bricata. In other words, he says, security is deluged by alerts that are often technically true, but largely irrelevant.
"This requires IT security to investigate such alerts, but the volume and vectors have grown beyond the finite resource of most organizations. Consequently, some alerts start to slip and go uninvestigated."
The Sony breach of 2015 demonstrated this challenge, MacFarlane points out. "While the tools were able to identify the malicious activity, those alerts were lost in a sea of 40,000 other alerts that same month. With a limited security staff, some malicious activity went uninvestigated until the inevitable happen."
A good security team will want to collect as much information as possible about the systems they protect, says says Chris Simpson, academic program director, BS Cybersecurity, National University, School of Engineering and Computing. However, he adds, this is a double-edged sword as they collect more data than they require so more resources are needed to understand the data.
"Alerting is used to bring important information to the attention of a security team,"Simpson says. "Alert fatigue can occur when a system generates so many alerts that the operator can't prioritize or respond to all of the alerts. For example, an alert can be generated if a user has a failed login attempt, and when many of the alerts are false positive this causes the security team to miss valid alerts."
A recent survey conducted by the Cloud Security Alliance highlights the large number of alerts that organizations deal with, Simpson says. The survey noted that 2.7 billion events were generated by the average enterprise using cloud services. Of these events, 2,542 on average were anomalous of which 23 were actual threats. The survey also noted that 32% of the respondents ignored alerts due to the large number of false positives.
The Target data breach is an example of alert fatigue that allowed a data breach to go undetected, Simpson explains. "Target had the right technology in place and received valid alerts that malware was inside their system. However, because the system was new and they had received excessive alerts they were unable to properly handle the alerts. This affected millions of customers, cost Target millions of dollars and lowered consumer confidence.
Many assume the biggest and only challenge IT security personnel face in dealing with alert fatigue is the overlooked threats that gets by among the sea of alerts, says May Wang (right), CTO and co-founder, ZingBox. "Unfortunately, that is not the only damaging result of alert fatigue. Due to the sheer volume of alerts, many IT staff are forced to define their own unique criteria of what's worth the time to investigate and what is not. Organization's exposure to specific threats can vary greatly from hour to hour based on the shift of the IT staff. It can also vary greatly across organizations even when they employ the same security solutions. The inconsistent security coverage resulting from this practice can often pose a bigger risk than few threats that may get overlooked."
Unfortunately, alert fatigue will not go away any time soon, Wang says, adding that many organizations have come to expect false positives as a sign of comprehensive security coverage during proof of concept. "When presented with X number of threats, detection of anything less than X number of threats is frowned upon. However, detection of more than X number of threats, as long as the specific threats are detected, is often considered a successful evaluation. Some security vendors are leveraging this unfortunate misconception and very much focus on 'lighting it up' during product evaluations."