Incident response has become a more complex art, says Rusty Agee, the city of Charlotte’s security leader. Karen Epper Hoffman reports.
By most accounts, last year’s Democratic National Convention (DNC) was a rousing success. And at least some small part of that is owing to Rusty Agee’s progressive approach to incident response (IR).
Not a politician or an event organizer, Agee is instead the information security engineer for the city of Charlotte, N.C., where the convention was held in early September 2012. A high-profile national forum, the DNC undoubtedly would have been a major target for hackers of all stripes, and yet the event went through with nary a major reported cyber security breach. With support from the city’s IR vendor, Agee says that if a major attack had occurred, the IT team was ready.
“We haven’t had any significant incidents to speak of for quite some time,” Agee says, conceding that the city still deals with the routine infected machines and malware outbreaks. “When I was first doing security, we all worried about someone hacking into the network. But over the last few years, the industry as a whole has come to realize that you have the threat of [people] trying to hack in, and it’s a lot easier for users on the network to make mistakes…It’s caused us all to be a lot more proactive.”
Agee began working for the city of Charlotte’s network team in 1999 as a contractor before moving in-house and then to the security side in 2007. Since then, he has seen incident response evolve greatly. In his current position, he is responsible for maintaining the busy city’s network of 6,500 users across more than 100 locations, including fire stations, police satellite buildings, utilities, solid waste facilities and the transportation and engineering departments. One of his major decisions as the city’s top information security engineer came in 2010, when he decided to replace Charlotte’s outdated incident management system with a more up-to-date security information and event management (SIEM) system.
The old IR system was not only going into end of life, but while it was efficient at collecting logs, it was not easy to get the data out of it, Agee says. With the implementation of the new platform in early 2011, Agee and his team are now able to generate and collect logs and analyze data from multiple sources to obtain a better picture of what behavior is normal and what is suspicious. “Now we can drill down with a couple of clicks,” he says, adding that the new system offers an enhanced view of the network from before and after intrusion, and fits in well with the new role-based security that the city has implemented.
Manageability, control and ease of use are becoming more important selling points to IR technology, as organizations increasingly recognize that the threat of a breach is more than a threat – it’s an inevitability. In the face of some bruising incursions, industry observers point out that companies and government agencies are finally realizing that it’s just a matter of time before their number comes up. And that is impacting the way they handle incident response from top to bottom.
“[Cyber attacks] have always been a reality,” says Tom Cross, director of security research for Lancope, an Alpharetta, Ga.-based security and network performance monitoring company. “But one thing that has changed is the appreciation for the sophistication of certain kinds of attacks.”
More targeted attacks, perpetrated by organized criminals – often outside the jurisdiction of federal or state authorities – have amped up the need for better, more systematic and thorough incident response, Cross says. “Incident response has become more popular in the past few years as a consequence of the more sophisticated targeted attacks people are facing that just weren’t there a few years ago,” he says. “They know that their perimeter defenses are just not up to some of these attacks.”
Major data breaches have not only become a weekly, if not daily, topic for the headlines, they are increasingly happening to some of the best-funded and most tech-savvy players across industries – including ones that are operating highly proficient networks. For instance, global consumer electronics firm Sony incurred widely reported back-to-back data breaches in April and May 2011 when hackers stole names, addresses and credit card data from as many as 77 million user accounts on the company’s popular PlayStation Network. RSA Security also was hit by an advanced persistent threat attack in March 2011, where thieves nabbed information related to the computer and network security company’s SecurID two-factor authentication products. And, earlier this year, daily deal site LivingSocial announced that it too had been compromised – with as many as 50 million customers’ records potentially exposed in the attack.
“The message increasingly is: ‘It’s not a matter of if, but when,’” says Christopher Pogue, director at SpiderLabs, the advanced security team for Trustwave, a Chicago-based information security company. “Companies can no longer defend the fortress at every level. We need to help companies prepare for the eventuality of a breach, mitigate the financial risks, the losses to customers.”
According to Verizon’s “2013 Data Breach Investigations Report,” two-thirds of breaches reported in 2012 took months or more to discover—potentially because organizations have been so focused on keeping hackers and criminals out that many do not realize they are already in. Peter Tran, senior director for the advanced cyber defense practice at RSA, the security division of EMC, based in Hopkinton, Mass., says that based on his 15 years in IT security, he has seen adversaries embed themselves in organizations’ systems for as long as seven years without detection. Also, according to the Verizon report, in roughly seven out of 10 cases, breaches are discovered by external parties – in most cases unrelated parties, such as internet service providers and intelligence organizations that track bad actors.
“There has been a lot of investment in the preventative-based technology, but now the threat is starting to change, and we’re seeing very advanced and highly targeted attacks and malware,” says John Vecchi, vice president of product strategy for the advanced threat protection group at Solera Networks, a South Jordan, Utah-based Big Data security analytics company purchased in May by Blue Coat Systems. “This new breed of attack can slice through these security fortresses like a hot knife through butter.”
The prevalence, pervasiveness and perniciousness of such attacks has fueled a recent shift in thinking, according to Vecchi: Organizations need to be prepared for the inevitable and be effective at responding to incidents in a way that mitigates loss and damage and offers insights into how to prevent similar breaches in the future. This is informing and influencing how incident response is handled from well before an incident occurs to well after it is contained.
Chris Petersen, CTO and co-founder of LogRhythm, a Boulder, Colo.-based SIEM provider (which is working with the city of Charlotte), says that he too has seen a defining shift in recent years – from people being focused on being breached and concerned with compliance to a place where awareness of potential attack has reached the executive board level. “The wheels have come off,” Petersen says. Players in the industry have recognized that their sense of security is gone. “The underlying forces are making them realize that they are vulnerable. Most verticals are in the crosshairs of one bad actor or another.”
Enter a new approach to incident response, which is, as Petersen describes it, “not necessarily a single technology, but a combination of technologies, processes and people that enable incident response as a whole.” The people and the process are, he adds, more oriented around collecting all relevant machine and log data in the infrastructure and making it intelligible. Real-time analytics play a critical role, and so easily getting access to information is key. “We need to implement countermeasures and automate our response, and take various measures to mitigate the issue,” Petersen says.
However, Agee says the evolutions in incident response are a “double-edged sword.” On one hand, he says, his users are getting more aware of phishing attacks, malware and more broad-based threats. On the other hand, crooks are finding “a lot more ways to target and attack.” Mobility and social media are creating more potential avenues of attack, he says, adding, “but you can’t just sit everyone down and say you can’t use that anymore.”
Marc Bleicher, senior incident response consultant for Bit9, a Waltham, Mass.-based security vendor, says he is seeing more collaboration within vertical industries about sharing the information regarding attacks. “Open source intelligence is more pervasive…especially with the amount of breaches in the news in the past nine months,” he says. It’s been a gradual evolution over the last three years since the Sony breach, he adds. “We all need to collaborate a lot more, change the past attitude of ‘we can’t let this information get out.’ It’s often the same actors and indicators showing up at different locations.”
Organizations are, in many cases, stepping up and proactively going after hackers with measures that include leaving decoy data or inserting delays into the malicious scripts in order to throw off or slow them down. More companies are enlisting approaches that look closely at network behaviors, not just signature rules-based alerts, says RSA’s Tran, who adds that this is part of a mindset that is shifting to investigate clues.
“They’re not waiting for the fire,” Tran says. “They’re going to see if they have anything smoldering now. They’re looking to detect an attack in motion.”
Ultimately, incident response technology needs to be easier to use – embedded with more automation – to enable a broader pool of talent without requiring years of forensic training, says Anthony Di Bello, strategic partnerships manager for Guidance Software, a Pasadena, Calif.-based digital forensic company. “All organizations are struggling with incident response,” Di Bello says. “Many companies are still lacking the necessary budget and the resources.”
Before and after
It’s not just responding to the attack in motion that has changed, but how organizations prepare before potential attacks and how they handle the post-mortem and clean-up after incidents are discovered. Tran describes it as the three Rs of incident response: readiness, response and resiliency.
Agee, of the city of Charlotte, says he and his group routinely play out breach scenarios so that they are better prepared for the real thing. “We have to have it laid out and have a plan of attack,” says Agee. “It’s certainly important in planning around those incidents – from a monetary and reputational standpoint.”
Lancope’s Cross says that first steps should include having an incident response team in place that knows exactly what to do when a breach is uncovered. But, as basic as this may sound, experts say that many organizations still lack an IR team, especially one with buy-in from top executives. “That IR team needs to go in and investigate what is happening, without being obstructed by the business process center,” Cross says. An IR group should include experts in computer forensics, full-time malware analysts (if the organization is in a sector where that is warranted), and an analyst relations or public relations liaison to address customers, media and even internal business users about breaches, says Cross.
Some companies, when they have a bad incident, may want to consider bringing in third-party contractors to augment the staff so that regular team members do not get burnt out contending with day-to-day work and a big breach, Cross says. And if the IR team plans to interfere with attackers, it must understand the consequences because some hackers, if they are spotted, may be sophisticated enough to “pivot and attack the network in a way that is harder to see.”
Trustwave’s Pogue recommends that organizations have a formal computer IR plan in place. A former U.S. Army officer, Pogue likens such pre-incident planning to the same battle preparedness and training exercises soldiers must undergo. The biggest mistake an organization can make is “not preparing at all, having a pervasive but naive feeling it will not happen to me,” he says.
Other common mistakes?
Follow-through is the most important, Cross says. “Not just understanding what happened, but protecting the network. The corollary to that, as he sees it, is the lack of intelligence sharing across organizations. He admits, however, that threat intelligence collaboration is getting better, especially in verticals like financial services, which is ahead of the curve.
Ultimately, there are only three types of organizations, Pogue points out. “Those that know they have been breached, those that are about to be breached, and those that have been breached and just don’t know it yet.”