Jones, an investigative manager for a large U.S. financial services company (who asked to remain anonymous because of the sensitive nature of his business), relies on Cybertrust, Herndon, Va., to help him identify, forensically examine, and remediate database compromises intended to steal the information on credit cards' magnetic strips processed by his company's systems. The data-stealing breaches don't occur on his firm's systems. They actually take place on those of the retailers, restaurants and other merchants that accept credit cards as payment for goods or services.

Jones is responsible for deciphering where and how the break-ins occur. He then figures out how to stop them.

The most common type of breach uncovered by Cybertrust's security forensics experts, he says, allows criminals to gain access to files used for diagnostics purposes in a merchant's credit card processing software. These files retain, on hard drives for 30 to 90 days, the data found on a credit card's magnetic strip. When problems arise, developers use them to debug the credit card processing software applications

This practice is in direct violation of the Payment Card Industry (PCI) standards, according to Jones. Still, it "happens a lot," often leading to unauthorized purchases on consumers' cards, he says.

"We use Cybertrust to investigate the systems at the site of the suspected compromise," says Jones. During this process, the forensics experts "determine what happened, how many accounts were affected, which accounts were actually affected, and remediate the problem."

During these investigations, Cybertrust's forensics engineers look for "common points of illegal purchases," Jones says. "Most of the time, this points to a merchant or chain of merchants. In some cases, however, it takes more investigation to determine a common third-party to find a common point within the data stream, and that's typically a third-party processor."

Jones says "it is important for us to have Cybertrust's people working on this every day, looking at data compromises not only from my business, but other businesses."

Jones certainly isn't alone in this regard, says Michael Gavin, a senior analyst with Forrester Research. "It takes a bit of an expert to run forensics tools and know what you're doing when investigating computer-related crimes," he says.

Some of the enterprise-class computer forensics tools available include CA's eTrust Network Forensics, Guidance Software's EnCase Enterprise Forensic Edition, AccessData's FTK, NetWitness's NetWitness, and the open source Helix. They are all extremely complicated applications that require considerable training and expertise to use properly, according to Matthew Shannon, a principal with Agile Risk Management, a forensics and litigation support services provider based in Tampa, Fla.

Looking to outsource

But it often doesn't make sense to hire and keep forensics experts on staff full-time, notes Shannon. Consequently, most companies find a consulting firm they can partner with for their forensics investigations, Gavin says. PricewaterhouseCoopers and Ernest & Young, are among those Gavin points to. Others include Agile, Kroll Ontrack, Mandiant, and Neohapsis.

Netwitness is one such "hired gun" that is called in regularly to help pin down some form of malicious behavior. Nick Lantuh, the company's vice president of strategic development, recalls one instance when information "leaks" were plaguing a Fortune company. "We were brought in to find out who was releasing information in a blog."

In another, Netwitness was called in to determine how a zero-day virus attack penetrated and took down a 75,000-node network. "By looking at network traffic, we were able to identify the virus in 10 minutes," he says.

An additional reason many enterprises outsource their forensics work is because they want an unbiased third-party investigation firm to work for them — someone without a personal interest in putting dirt on the person they're investigating, says Erik Thompson, computer forensics specialist with Chicago-based Neohapsis. This can be important when civil or criminal litigation might result from the investigation, he says.

Keeping it inside

Still, many enterprises do keep computer forensics in-house. Just how many is open to discussion, says Forrester's Gavin.

"Companies are buying more forensics software than they used to, but it's very difficult to get real numbers," he says. "Most enterprises are very hush-hush about it — they're afraid of giving anything away to potential criminals."

The usual security-related suspects, including compliance regulations, such as Sarbanes-Oxley (SOX) and the Health Insurance Portability and Accountability Act (HIPAA), are driving these deployments, many experts say. Yet computer forensics-related deployments that are supposed to help address such issues are anything but straightforward for most enterprises, says Alan Brill, a senior managing director with Eden Prairie, Minn.-based Kroll Ontrack.

"It's not like going to the store and buying a fire extinguisher," he says. "These are not simple packages. They have a lot of options, and part of the skill is knowing which option to use under which circumstances."

Deploying and maintaining an enterprise-class forensics application in-house is something that requires training and a certain amount of experience, Brill says. "And, after a certain amount of time, you have to update the software, and you have to understand how the update affects how you use the software."

Still, even big firms find organizing an internal response program can be troublesome, says Kevin Mandia, president and CEO of Mandiant, a security consulting firm with offices in Washington, D.C. and New York City. Most notably, when an incident such as a security breach that requires forensics investigation occurs, executives within a company often don't know who the in-house experts are or how to contact them, he says.

"I've seen companies with 100,000 employees, and only eight experts in computer forensics and incident response," he says. "Most organizations are challenged with how to get these people involved as soon as possible."

In deploying a forensics capability, therefore, it is vital to get every department within an organization involved in the process. When determining the policies forensics investigation experts must follow when scrutinizing what appears to be an incident like fraud or the theft of intellectual property, the legal and human resources departments, as well as information technology and security groups must be engaged.

These company initiated policies are the critical component of a well thought-out forensics initiative, according to Tim Leehealey, an executive vice president with Guidance Software, which has its world headquarters in Pasadena, Calif. "They must state clearly who is involved in an incident response, the scope of the investigation, when it should be escalated, and who to escalate it to," he says.

When enterprises deploy forensics systems, they're apt to make similar mistakes, according to Leehealey. "Not getting all the parties involved is a huge common issue," he says.

Policies are key

"A forensics investigation is a unique IT issue that cuts across business lines in an organization," he adds. "It's a knee-jerk reaction to let IT pick out a forensics application and buy it. It doesn't make sense to give IT an uber tool and let them do what they want with it," he says.

An enterprise needs to protect the rights of the people being investigated, he adds, and you also need to make sure the investigation is conducted with the right scope, including discovery.

Many forensics investigators fail to think before they act, notes Neohapsis' Thompson. Turning on and investigating a suspected hard drive without proper forensics tools is a common mistake.

In addition, many enterprises wait too long to begin an investigation into a security event, says Sam Curry, a vice president in CA's security management group. "If you think you have a problem, it's already time to bring an expert in," he says.

Jim Carr is an Aptos, Calif.-based freelance business and technology writer.


Incident response

By assessing your security program from enterprise, technical and compliance perspectives, you can develop and implement more effective policies and procedures. You need to devote appropriate resources to your program while fostering security awareness throughout your organization.

In addition, it is critical to develop a strategy and supporting tactics for responding to security incidents — including investigating and analyzing them after they occur. Only through effective forensics can you properly secure, recover and authenticate data for use in criminal and internal investigations, as well as civil litigation.

Create well-defined policies and procedures for handling security incidents, including how they will be reported and resolved.

Engage your legal department and also define clear roles for business unit leads as well as IT management and staff.

Establish a clear approach to forensics — the process of collecting, preserving, analyzing and presenting computer-based evidence, which typically includes log data from all network, security and systems hardware involved in the event, external application data and IT resources that may have been compromised, and testimony from the individuals responsible for management and alerts with respect to the enterprise's security infrastructure.

Compare your policies and procedures to industry best practices to ensure they are bullet-proof and map to your business objectives. Consult the following organizations for their policy recommendations: International Information Systems Security Certification Consortium the Information Systems Audit and Control Association, or the SANS Security Project.

Consider the pros and cons of using third-party services or forging an internal forensics/response team

— Brian Anderson, director of product management, SunGard Availability Services.


Do's and Don'ts

Do ensure that senior management buys into your incident response initiative. Lack of management buy-in to comprehensive incident-response and computer forensics polices ultimately leads to failure, according to Harlan Carvey, a regional manager for X-Force Emergency Response Services at Internet Security Systems, Atlanta.

Do align IT's security objectives with the line-of-business priorities. IT personnel should realize the business impact of shutting down an ecommerce server they suspect has been compromised.

Do ensure that every employee reads, understands, and signs a security waiver that acknowledges your organization's authority to read employee email, track internet usage, or monitor access to various network resources for forensics purposes.

Do set up a standardized chain of custody after collecting forensics evidence from a computer. Failure to do so can make evidence inadmissible in civil or criminal proceedings.

Do keep thorough documentation on each forensics investigation.

Do clearly define and publish within the company who is responsible for what within your incident-response team. This can enable non-technical employees to readily report potential breaches or suspect activity.

Don't turn on a computer you suspect has been compromised without using proper forensics tools, such as Write-Blocker, to ensure you won't damage potential evidence by changing file attributes or overwrite deleted files.

Don't develop your incident-response policies in a vacuum. "Work with your legal, human resources, and PR teams as well as executive management to determine what your critical issues are, what data is classified or most sensitive, and the various sources of information," says Matthew Shannon of Tampa, Fla.-based Agile Risk Management.

Don't wait until you must respond to a security incident to attempt to learn how to use your forensics solution. Practice using the application to not only become familiar with it but to see what problems arise, says Eric Eifert, executive director in the cyber defense division of ManTech, which has its corporate headquarters in Fairfax, Va.

Don't think of forensics as mere troubleshooting — discovering how a hacker got into a system, then trying to fix the problem. That can change a system and destroy a proper chain of events or timelines, eliminating the opportunity to prosecute the offender. — Jim Carr