Experiencing a system breach that brings down multiple corporate computers is bad enough. Opting to get the affected systems back up and running quickly, without making any attempt to understand the true root of the problem, is usually worse.
A U.S. Department of Defense agency knows this all too well. "It's just an ongoing thing with the Department of Defense (DOD). It is constantly under attack on its network computers around the world," says John Colbert, CEO of Guidance Software.
But in the past six months, the unnamed DOD agency had a significant problem with a hack that had compromised many systems by proliferating throughout the infrastructure more quickly than IT professionals could keep pace with. So it decided to wipe out hundreds of computers, re-baseline them, and then put them back in use.
"It took about a week, and everybody lost their emails, all their work, all their documents and what-not. While these systems were getting re-baselined, [they] were inaccessible," recalls Colbert. "The end result was about two hours after going back online, they were hacked again with the same hack."
The story is emblematic of the unfortunate way most organizations go about recovering from an incident, he adds. Choosing what he calls the patch-and-proceed method of recovering from an incident, which entails re-baselining the system, patching the problem, and then proceeding without ever learning the hack's root, might not cure the trouble.
"An organization is better off trying to find the root of the problem, determining how to actually correct that problem, and then doing some type of a compromise assessment across their network to look for any other systems that are affected," he says.
Failing that, a company might spend weeks or months attempting to recover from the episode, much like the DOD agency as a result of its recent attack.
"The Department of Defense is just as vulnerable as any Fortune 500 company. As for this incident, we support those investigative organizations out there, but we don't actually run those investigations," explains Captain Robert Renko, operations chief for the Defense Computer Forensics Lab, a division of the Defense Cyber Crime Center (DC3), which is part of the Air Force Office of Special Investigations. The Center provides computer forensics support to the DOD's various military criminal investigative organizations, such as the Navy's NCIS or the Air Force's Office of Special Investigations.
Guidance's Colbert says that the DOD agency is "still trying to determine exactly what the intent was, but it was definitely a compromise of the systems." So what is the most effective way to prepare for an attack before it hits?
Getting back up and running fast and collecting data and evidence to both learn from the incident and possibly pursue litigation are far from diametrically opposed goals, especially if you have a response plan in place.
As long as organizations have clear procedures on what is done when a breach occurs, and which employees are directly involved, the two aims can be met, contends Christopher Painter, deputy chief with the Department of Justice's Computer Crime and Intellectual Property Section. And part of achieving these ends means ensuring from the start that companies and their systems operators understand that while they are remediating their systems, they must retain pertinent data in such a way that it can be trusted and used in both internal and, possibly, law enforcement-driven investigations.
"The obvious next step is to think about some best practices. We have some of those [that were developed through the G8 Justice and Home Affairs Ministers' Subgroup on High Tech Crime] and we're working on more of them all the time," says Painter.
But it is not enough simply to draft a response plan and maintain a corporate IT security policy that informs staff of their everyday infosec responsibilities, contends Curtis Tomlinson, manager of investigations for AMD, the California-based chip maker. Corporate security professionals must stay educated about infosec goings-on and maintain connections with peers. Doing this will keep them apprised of trends in the industry and, sometimes, offer a heads-up to various new attack methods.
"It's not enough now to develop a policy that looks sound, [then] put it in place and leave it there. It's something that has to evolve over time, depending on what's occurring in the immediate environment," he explains, noting that such continuous development applies to both overall IT security policies and investigative procedures.
Procedures and policies are living, he says, and should avoid being peppered with vague directives. Planning and people are critical; without them, the first reaction will always be to fix the problem immediately and/or take the system off-line, which might lead to evidence being compromised. Network administrators, infosec professionals, those who know about forensics practices, public relations specialists and corporate lawyers all play roles when a hack occurs and need to know their parts. The last position companies want to be in is only planning a response while under attack.
"A lot of it is preparation. When a problem is first noticed, the first thing a sysadmin wants to do is log onto the machine and see what the problem is, and when they do that, they impact on very important evidence," says Tomlinson. "That's why there needs to be a protocol in place so that when something is noticed that's amiss, someone who has the tools and the knowledge is the first one to react. And the response plan needs to be really specific. When we started looking at our plans several years ago, all the right steps were in place, but the specifics of how to perform each step weren't there."
Calling in the cavalry
One action that often gets short shrift is notifying law enforcement. Companies are reluctant to call in the cavalry when a significant breach occurs for a number of reasons.
"The studies suggest concerns about reporting still exist and a lot of organizations do not report," says Jennifer Granick, of Stanford Law School's Center for Internet and Society. But the truth is that getting official investigators involved is usually beneficial for organizations and their internet peers, and is crucial if organizations wish to seek criminal or civil action.
"If you're thinking about a prosecution, you probably want to get the FBI involved at an early stage, or your local police," says Granick.
DC3's Renko believes all public and private sector entities should seriously consider contacting law enforcement early enough to follow a still hot trail of evidence. But, again, it is crucial to have a plan before even getting to this step.
"By the time the intrusion occurs, if you don't have a policy in place it's too late. So plan in advance what the policy's going to be – that's step one. And then you have to determine what your final goal is. Are you prepared to push for prosecution? Is it corporate image that's most important or financial solvency of the company?" says DC3's Renko. "But on the flipside of that, you also have to think of what the ramifications would be if you don't contact law enforcement."
Despite some numbers indicating otherwise, companies do seem to be coming around to the idea of contacting law enforcement, believes Guidance's Colbert. A unified effort among organizations to fight cybercrime is growing, as is a more communicative approach among government bodies. The DOD is definitely taking measures to increase communications among its agencies, although the government seems always to be "leading the road on responding to hacks and incidents."
"I think there are still problems in the private sector with organizations sharing that information, but we are increasingly seeing organizations turning around on their opinions and realizing that communications with their competitors and with law enforcement is really beneficial to them," explains Colbert.
Reporting is also getting a bit more play than before, because organizations are getting better at identifying cyber attacks through technology and better incident response processes, says Kris Lovejoy, chief technologist and vice-president of technology and services for Consul Risk Management.
"This fact, coupled with the new teeth defined in such regulations as Sarbanes-Oxley, not to mention case law, has led to an increase in the numbers of cyber attacks reported," she says.
"Of course, this trend needs to be assessed within the context of counterbalanced variables," she continues, "the increase in the number and complexity of complex cyber attacks versus the improvement in protective and detective controls."
And what about going public with news of a hack? Lovejoy tends to view the question of reporting cyber attacks from the perspective of a shareholder. He wonders at what point would he prefer to see a company take news of an incident to the masses.
"And my answer is this: Assuming that the company has implemented an operational risk framework, I would want it to report any attack that resulted from a material deficiency in the control framework that had been established," she explains.
"In layman's terms, a cyber attack is not simply a security issue, it's a business issue. An attack that results in a cost which rises above the acceptable bar must be reported to the shareholders, if not the public."
No panacea exists
Even when policies covering incident recovery, investigative best practices and end-user procedures are planned and rehearsed alongside the deployment of countless preventive security tools, "then misuse or abuse of business critical information is inevitable," says Consul's Lovejoy.
"The key is being able to distinguish real events from noise and take corrective action. Companies should therefore offer their IT security professionals a combination of a technology infrastructure that enables security staff to more rapidly detect and react to suspicious activity before it causes problems (using log management as the base), as well as more institutional training and practice on execution of the corporate incident response policy," she says.
Plus, companies must be mindful that hacks rarely prove successful on their own merit – they typically take advantage of a corporation's vulnerabilities. To put it simply, hackers exploit system holes and take advantage of corporate-wide ineptitude.
"I was always surprised by the fact that it's not really the latest and most technological exploits that are the cause of an attack," says Renko. "Really, it's poor security practices, people not loading the latest patches, and things like that causing the greatest percentage of intrusions. No computer is 100 percent secure... You take as many safeguards as possible, but it's risk assessment every time you connect to a network out on the internet."