By their nature, advanced persistent threats (APT) are not easily identified. They are engineered to lurk surreptitiously beneath the thresholds set in security devices to send an alert. The annual Verizon Data Breach Investigations Report notes that 66 percent of breaches remained undiscovered for months or more – which is a lot of time for an intruder to search an environment for a target.

While slow-moving over time, APTs still move faster than many internal detection methods can handle. In fact, the Verizon report notes that 69 percent of breaches it studied in 2012 were discovered by external parties.

How can attacks that span such a long period of time continue to escape our detection?

The most significant reason that threats go undetected can be summarized in two words: insufficient data. Many organizations simply are not collecting, storing and analyzing easily available data that could alert them to the presence of an APT or provide earlier and deeper insight into an APT's source and progress.

This may seem ironic, given that a typical enterprise is already “drowning in data.” Existing infrastructure such as servers, routers, firewalls and intrusion detection systems (IDS) are producing more data than security analysts can consume. But is it the right data? Can it be analyzed? And can it be acted upon?

With the right answers to these questions, organizations can reveal the “big picture” and begin to defend against APTs. Let's take a closer look at some of the best practices and approaches that can help.

First, is it the right data? Getting the right data involves tapping into all sources and retaining that data over time. The more sources of data tapped, the greater the chance of finding relevant information. APTs attempt to enter an organization through many different paths, so the net must be cast as widely as possible. Any event with a time stamp – like a system log-in or file transfer – can potentially turn out to be evidence of an attack. This is why data must come from a large variety of sources including server logs, firewall and IDS/IPS logs, and flow data indicating network connections.

Furthermore, enterprises need to collect and save that data over a long timeframe not so they can necessarily find the one incident that will serve as a “smoking gun” to identify or stop an APT. Instead, organizations need this data so they can analyze adversary behavior across time and data sources to identify patterns that simply cannot be revealed with single point-in-time approaches. Having the appropriate storage technologies to support the retention and analysis of massive volumes of data is paramount. For example, traditional relational database management system (RDBMS) technology can be a poor match for the retention of significant volumes of time-stamped event data and the ability to conduct real-time, detailed analysis.

Which begs the question, can the data be analyzed?  Without the tools to sift through large amounts of security and infrastructure data, look for patterns and bring together seemingly disparate incidents, the clues this data contains may never be revealed. Siloed organizational structures can be the enemy of effective counter-attack strategies.

What's needed is an approach that centralizes event data management. When significant data points from all collected data sources are brought together in a central location, trained security personnel can more easily look across multiple sources in search of trends, anomalies or other valuable insights. Technologies that let security analysts use this data to profile attackers will start to reveal the big picture, including the attack vectors exploited, the targeted applications, how the attacks are obscured and what the attackers are looking for. With this data you can assess how compromised nodes are managed by a command-and-control network and locate where the compromised data is sent.

Finally, can the data be acted upon? With tools that enable detailed analysis and an approach that includes automation and real-time threat intelligence, security analysts can now act upon this data to mitigate the risk to the organization. The Verizon report notes that even though APTs can remain undetected for months, 69 percent of the attackers were sending out enterprise data within hours or minutes after initial compromise, or even seconds in some cases. To decrease time to respond, it becomes necessary to utilize a full spectrum of threat remediation methods to automate the investigation of threats and their mitigation.

Further, because attacks are constantly changing, defenders need to evolve so they can keep up. Security analysts should continue to educate themselves on the latest threats and also keep their security tools updated with the latest detection and mitigation capabilities. Although many enterprises are reluctant to share threat intelligence outside of their borders, this type of open communication benefits everyone in defending against new threats.

APTs are a serious and mounting threat, but the disciplines of capturing, storing and analyzing event log data, profiling the attackers, and automating threat remediation, can help mitigate the impact of APTs and perhaps even prevent them.