Security information and event management (SIEM) tools have frustrated many – yet they are here to stay, reports Beth Schultz.

Depending on who you ask, security information and event management (SIEM) is either a boon or boondoggle, or maybe a little bit of both.

Done well, SIEM produces undeniable benefits – the ability to react in real time to threats and to meet compliance mandates rank at the top among them. However, the problem, many say, is that SIEM solutions can be difficult to tune to full effectiveness. In fact, sometimes the effort can be so overwhelming that companies leave their SIEM appliances in a corner collecting dust rather than deal with the time and effort required to make them useful.

 “As much as I love SIEM – it's my favorite technology – I know there are a lot of people who are frustrated by it,” says Anton Chuvakin, who worked within the SIEM vendor community for nearly 10 years and now heads SIEM consultancy Security Warrior Consulting. “This includes people who think they've been deceived by vendors and those who think they've been promised something that they haven't been given. That's a very powerful sentiment about SIEM.”

But still, Chuvakin adds that SIEM's promise of combining security technologies together for better visibility across everything going on is an important goal. “We almost have to have this, and the fact that we cannot have it yet doesn't stop me.”

From threat monitor to compliance checker

SIEM tools evolved out of the intrusion detection system (IDS) and intrusion prevention system (IPS) disciplines, upping the ante with the notion of real-time threat monitoring.Early SIEM tools – whether software or appliance – collected data from security devices, such as a firewall, IDS or IPsec, and searched for patterns indicative of threats. They used rules-based correlation to speed problem recognition.

Historically, SIEM represented the be-all and end-all of security management. It was the purview of only the largest enterprises – those with deep pockets and big IT security staffs manning multi-console security operations centers.

“The belief was that in order to be a serious security shop, you had to have a SIEM in place,” says Richard Bejtlich (left), principal technologist and director of incident response at General Electric Co. “A lot of companies fell into this trap. ‘OK, the next phase of our maturity is we have to have one of these things,' they said.”

While Bejtlich says reality hasn't borne out the absolute necessity of SIEM, enterprises of all sizes flock to the technology today. Many enterprises, no matter the size, are drawn initially to SIEM for compliance rather than its original intent: threat monitoring.

Indeed, when new compliance mandates hit the enterprise – stemming from initiatives such as the Health Insurance Portability and Accountability Act (HIPAA), the Payment Card Industry (PCI) Data Security Standard and the Sarbanes-Oxley Act – SIEM took on a new life. By feeding network device event logs into a SIEM, enterprises created a central repository for the logs they'd need for meeting compliance mandates. They could analyze and run reports on the logs, as well as archive them. Suddenly, SIEM data grew to colossal proportions at many companies.

SIEM on fire

Now, SIEM has reached the mainstream. “We're in the broad adoption phase,” says Mark Nicolett, a vice president at Gartner Research who has been following the technology's development since its earliest days. In 2009, Gartner fielded 40 percent more calls from clients with funded SIEM projects than it did in 2008, which had seen higher SIEM-related call volumes than the year prior, too. “Investment in SIEM projects carried right through the economic depression,” he says.

SIEM has become a hugely popular security management technology, agrees Charles Kolodgy, research director of security products at IDC. In fact, SIEM is driving overall growth in the security and vulnerability management technology sector with a compound annual growth rate of 16 percent for the five-year period ending in 2013, he says. Led by ArcSight, EMC, Symantec, Attachmate and Q1 Labs, SIEM revenue for 2008 reached $663 million, IDC's research shows. That figure is expected to reach $873 million in 2010 and $1.4 billion in 2013.

But still, many industry watchers say SIEM evolution has been disappointing overall, and that in many instances the technology falls far short of its potential.

One wary SIEM watcher is Joel Snyder, senior partner with Opus One, an IT consulting firm. “Fundamentally, I haven't seen a huge evolution in the technology inside a SIEM” he says. “There has been change, so it's not as if we've sat still. But really innovative ideas, like using reputation services, haven't permeated throughout the marketplace. We see that sort of thing in one or two products, but more or less that's it. So it's been disappointing.”

SIEM specialist Chuvakin agrees. “Basically, a lot of the technology is pretty much what was built by the early SIEM vendors in the late 1990s,” he says. “There were a lot of ambitions put forth and a number of promises about what SIEM would do, ultimately providing a single pane of glass for viewing all security things across the corporation. But that hasn't really happened.”

The problem lies in the very nature of what SIEM tools aim to do. “You've got this massive pile of logs and the SIEM tools have to give you useful, actionable data from that,” says Snyder. “The hardest thing is the rule writing. While SIEM companies have done a great job of providing rules or giving the ability to write rules, they haven't taken a quantum leap. It's easier, but fundamentally the same.”

GE's Bejtlich echoes these concerns: “SIEM provides a smart system that can take data from all our different systems and make sense of it and tell us what's really going on. Only problem is, if you're starting with data that you couldn't operationalize – meaning, turn into an action – then it's pretty ambitious, to be polite, to think you could have a system that could make sense of that stuff.”

Roll up your sleeves

All this is to say that SIEM requires hard work – a lot of it. “It's certainly not something you can buy, install and forget about,” says Johannes Ullrich, chief research officer with the SANS Internet Storm Center.

Nor can you turn to an easy-to-use manual for guidance, he adds. “How you configure SIEM depends on your network, and then you need to make continual adjustments, going through the network device by device to figure out how best to collect and correlate event logs.”

Jeff Dalton, technical operations officer for Regulus Group, a nationwide payment services provider in Napa, Calif., adds that when it comes to SIEM, he doesn't think there's such a thing as 100 percent deployed. After all, he says, “every time we add a new device to our network, we've got to add it into this wonderful world.”

Dalton, for one, approaches SIEM with caution. He's been driven to SIEM with the goal of more efficiently and effectively handling the 1,700 special audit requests his company must handle yearly. “These requests could easily be one or 100 questions, but they are all mainly asking, ‘Are your roles and controls surrounding governance adequate?'” he says.

For now, PCI remediation is his focus, for which he turned to Q1 Labs' QRadar SIEM tool. He's feeding logs from about 350 devices into the tool, roughly 40 percent of his production environment, and checking to make sure the reports he can pull from there pass PCI, the SAS 70 auditing standard and other requirements. “We're not going any further until we validate that the reports are coming through and doing what they're supposed to do,” Dalton says.

So far, so good, and, he says, the benefits have been both organizational and personal. “I used to take at least an hour-and-a-half to two hours a week to make sure we could pass a quarterly audit for a specific customer, and now I'm down to about 20 minutes per week. That's saved me quite a bit of time, almost 60 percent, which is really important,” Dalton says. “We don't have a whole lot of IT staff, and being an outsourcer we're trying to stay lean and mean in order to be competitive.”

Size matters

While smaller and mid-sized enterprises might have just a single person keeping tabs on the SIEM data, a large organization might devote a team or more of security analysts to implement and then watch over SIEM. Such is the case at BNY Mellon, a leading asset management and securities services company based in New York. Created out of mergers and acquisitions, BNY Mellon has inherited multiple SIEM tools and has since consolidating on a single platform, says Daniel Conroy, the firm's managing director of information security.

“We have teams of analysts that look for internal and external threats, and a team that does access privileges for database monitoring. Plus, automated alerts trigger off anything else that happens to be outside the environment for those teams. For example, our firewall managers might look at the logs sometimes if they're trying to diagnose a problem,” Conroy says. “We use the SIEM across the organization for four or five different reasons – from debug to analysis to investigation.”

A global operation, BNY Mellon is feeding data into its SIEM from upwards of 70,000 devices. The SIEM tool handles some one million events per day, so far with aplomb. That's a pleasingly different experience than what Conroy says he has had with previous products. “We crushed other SIEMs with a lot less traffic than that,” Conroy says.

And one million events per day is nothing, comparatively. Mark Evans, information security manager for Salt Lake County Information Services, in Salt Lake City, reports having had 600 million events in his SIEM database after only two months of use. While Evans has begun the finetuning that will help reduce that volume, he says he isn't worried about it. “I know I've got a long way to go before this thing is fully implemented,” he says.

While other products he tested couldn't keep up with that volume, the tool he selected, NitroSecurity's NitroView Enterprise Security Manager, is doing so without a problem. “We can jump around in it like lightning,” he says.

Plus, Evans already has cut the database in half just by turning off 95 percent of the signatures it had been feeding into the tool from its Websense web traffic monitors. “We left the juicy ones in, like who tried to download an .exe file or visit a keylogger site, and turned off the rest. Now we're only getting a few thousand signatures a day rather than a few million,” he says. And, he adds, deleting the 300 million unnecessary signatures from the database only took five or so minutes.”

Not all SIEM users experienced such ease of use. Brian Shellock, senior security analyst at Lehigh Valley Health Network, a large hospital system serving eastern Pennsylvania,  says a sizable event-per-day volume flattened his first SIEM tool, essentially turning it into a security and compliance nonentity, he says.

“We were getting more than 100 million events per day from just half of our systems, and it couldn't handle the volume. It would chug along, freeze up and miss 22 hours of the day. It wasn't catching or storing the data,” Shellock explains. “Plus, running even a short report would take hours.”

For the past two years, however, Shellock has been using Prism Microsystem's EventTracker SIEM tool without any problems at the health care organization based in Allentown, Pa. “It doesn't miss a beat,” he says.

This SIEM tool has changed the way Shellock approaches security and compliance management, he says. “It essentially gives me the eyes and ears going through the events, so I can be out doing other things, getting more information, figuring things out, setting more alerts and correlating behind the scenes.”

Previously he had been limited by the ineffectiveness of that earlier SIEM tool, which he had brought in-house in large part to address HIPAA compliance mandates. “I spent all my time collecting events just so I could say, ‘Well, I'm obeying HIPAA and storing my logs.' That's great, but what does it all mean? All this information was there, but I wasn't getting anything out of it.”

Easing into SIEM

For many companies, the sense of ease they're expecting following a SIEM implementation never does materialize. This can especially be the case at smaller companies, which often rely on the SIEM tool vendor or other external experts for implementation help. Subsequently, they often feel swamped by event notifications.

Tom Franciosi, CIO at Covenant Dove, a national nursing home provider based in Memphis, Tenn., knows how critical, yet overwhelming, the initial SIEM experience can be. Franciosi joined Covenant Dove three years ago to help centralize IT operations as the company undertook an aggressive growth strategy.

“Given that health care is one of the most popular targets for nefarious hackers, I knew we needed some means of gaining real-time visibility about how the network was behaving, or not, and that was SIEM,” he says.

Having evaluated SIEM products for use at a previous employer, Franciosi selected TriGeo Network Security's TriGeo Security Information Management product and set it up to collect information from almost 1,000 devices.

“When we first installed SIEM, our security guy did feel the danger of having too much information coming at him, so we tuned it down a bit – the first exposure to this type of product can be overwhelming,” Franciosi says. “But once you're used to it and you know what you're looking for, like anything it becomes more manageable.”

But this is not always the case. “One of the biggest problems with SIEM implementations is that most companies hire a consulting company for deployment,” says Nathan LaFollette, CEO of iNet|Detect, a security consulting firm in Columbus, Ohio. “The consulting company will install it, throw every device at it and tell the owner, ‘Hey, you've got all this great data here.' And the owner says, ‘Oh, this is so valuable, all this visibility is great.'”

Problem is, the next day the contractor leaves the company to its own devices. And the IT security professional is stuck trying to make business sense of 15 million alerts a day. “That can be a daunting task for any company,” says LaFollette.

“Most customers at this point weigh the cost of sifting through millions of events per day and tweaking the false-positives in hopes that at the end of the tunnel they will see business risk measurability,” he says. Some decide the effort won't pay off or, perhaps, that they simply can't devote the necessary personnel hours to the ongoing task.

Those who decide to proceed with the fine-tuning ought to be prepared to spend six to 12 months tweaking the tool before seeing any business risk value, LaFollette says. “Those who don't will continue to use the SIEM technology as a glorified syslog server that cost way too much.”

Taking the fast track

GE's Bejtlich says he's talked to many peers who essentially do just that. “They use it sort of as a giant bucket. They dump everything into the bucket and use it to do simple things, like searching. That's not SIEM. Correlation is missing.”

Still, this works for some. “What's nice about this approach is the amount of information you need to do something useful upfront is low, as opposed to the SIEM idea which involves deep knowledge of your environment,” Bejtlich says.”

It's a fast-track approach, Bejtlich adds. “You put everything in one place and then search it once you know what to look for. I call that ‘retrospective security analysis.'”

True SIEM implementations are far more time-consuming, even more so than LaFollette's six months-to-a-year projection, especially for large enterprises. “Anybody who tells you they've got a successful SIEM deployment has generally taken two, three or maybe even four years to get there,” Bejtlich says. “What you're doing is using the knowledge of your own environment that you've built over the years and you're encoding that into a tool. That's the key lesson from the whole SIEM idea – it ends up being more about what you learn about your environment and less about the tool.”

In and of itself, that's not a bad thing, Bejtlich adds. “Anything you can learn about your environment is definitely worthwhile. The question is, Are you willing to spend six figures, potentially seven, for a premiere product to get to that? Or is there another way?”

At GE, the security team uses SIEM as yet another source of indicators rather than as a universal system for pooling events and tracking incidents. “We put a select feed of data into SIEM and have the SIEM correlation engine tell us, as Chris Matthews would say, ‘Tell me something I didn't know,'” says Bejtlich, referring to the host of MSNBC's Hardball with Chris Matthews.

The SIEM reveals to the GE security team something it didn't know by culling through and normalizing the log data, sending an alert when it comes up with something fishy. The team treats that alert just as it would any other data received from its systems, Bejtlich says.

“We have a dozen or so generic ways to find activity, and SIEM is just one of them. That's a quick way to SIEM, taking a period of months instead of years to implement – and it's flexible, realistic and doesn't cost as much money as when using SIEM for everything,” he adds.

While Bejtlich declines to name the SIEM tool in use at GE, he notes that the company looked for a product that met its needs, but intentionally stayed away from the premiere wares. “We didn't feel the money for a Cadillac was justified by the extra features we would have gotten. We were more interested in the correlation engine piece,” he says, noting that GE has been using its tool in a meaningful way for more than a year, having installed it in the fall of 2008.

SIEM of the future

A critically important part of the story, Bejtlich says, is enterprise log management. “If you get that as part of your SIEM, that's great. If you integrate a standalone product into your SIEM, that's great, too. I can't recommend this highly enough,” he says. “You've got to have a repository that allows quick searches and accommodates a huge variety of data types.”

When GE began its SIEM project, log management wasn't a common request. That meant that integrating the company's standalone log manager was a bit of chore, Bejtlich says. If he had to make the SIEM decision now, a top question he'd ask potential vendors would be whether they offer log management functionality as part of their SIEM and, if not, with which log management systems they integrate.

Integration with an ever-increasing array of network devices is a common theme among SIEM providers and users – as well it should be, says Gartner's Nicolett. “We have companies that are three-plus years into this, and they're still expanding the scope. That's a sensible way to tackle this,” he says.

Sustaining the effort makes the difference between good and great, he adds. “It's a basic decision that a company makes. If it is just interested in solving the compliance issue, it basically stops expanding the leverage of the technology and doesn't get the full benefit from it – and mostly it's a security benefit. Security benefits accrue for companies that keep moving forward with it after the initial install.”

And companies should not expect to reap cost-savings out of those efforts, Nicolett notes. It's not possible to justify the technology on potential cost reductions. What he tells clients instead is this: “If you don't have the technology in place, then you're flying blind and you don't know what's happening in your environment. This technology represents work that has not yet been done. You install the technology, and if you're using it properly, you'll discover issues that were unknown to you. When those issues are discovered, you'll have to do some project work to resolve them. So this is not about cost-reduction. It's an improve-your-security-capabilities type of technology.”

As Bejtlich says, “Before SIEM, we had terabytes of logs available, but we weren't really actively doing anything with them. Once we put in SIEM, not only did those become available for searching, but SIEM goes through and finds things we didn't know were there – and that's what it's all about. There's just no way a human can do that.”

[sidebar 1]

What's next for the SIEM market?

Expectations surrounding security information and event management (SIEM) technology have always been grand. After all, what enterprise IT security executive could resist the thought of gaining visibility across all security domains – application, database and network?

As much frustration as this ambitious SIEM vision has fostered, however, industry watchers and participants still expect big things of the technology in the future.

“The future SIEM will have to be able to find, in real time, the needle-in-the-haystack security event out of a pool of hundreds of thousands of events per second and, at the same time, be able to run a query over terabytes of data and deliver results in a matter of seconds or minutes, not hours,” says Jon Oltsik, principal analyst with Enterprise Strategy Group, based in Milford, Mass.

That's a tall order, indeed, he says. But several trends inside and outside the SIEM arena will ease the way to that inevitably, he adds. For one, the intelligence associated with event filtering and correlation from multiple data sources is continuing to improve. Ultimately, Oltsik says, the data collection and storage processing part of SIEM will go away and instead be handled by a log management tier.

Secondly, SIEM tools will become better and better at doing complex queries for investigations, forensics and analysis, he says.

The BNY Mellon, a leading asset management and securities services company based in New York, could use just such improvements, says Daniel Conroy, managing director of information technology at the firm. “You're always going to have a certain noise level across the network, but no SIEM platform today really provides an understanding of that activity and its spikes,” he says.

What he hopes for in all SIEM technologies is trending and behavior analysis. Say, for example, a database administrator logs in at 4 p.m. till 4 p.m. the next day, and all of a sudden at 2 a.m., activity under his account begins happening.

“The technology should be able to detect that as unusual based on trending of people with that sort of data linked to the IP address and user account information,” says Conroy. “That's where we're going to have to be in two or three years. We need that type of behavior analysis built into these toolsets.”

For now, security analysts must scour log data and search for indicators. “But these are mathematical patterns that should be built into the toolsets,” Conroy adds.

With the continual improvements in processing, such goals should be attainable, Oltsik says. “It's certainly possible, but it will take focus. Some of the legacy SIEM folks may not be able to manage through the changes as we move from data collection and all-in-one packaging for security and compliance to really deep event detection and analysis skills.”

– Beth Schultz

[sidebar 2]

SIEM: Enough?

For many SIEM clients, the Verizon “2010 Data breach investigations report” brought good news, showing a reduction of nine percent in attacks by outside agents. The bad news was that insider attacks were up a whopping 48 percent! The report also indicated that attacks by hackers were down 24 percent while data breaches involving privilege misuse were up 26 percent.

What does it all mean? It means that SIEM isn't enough to protect your data, your privacy, your money or your business. Good SIEM systems can block outsider attacks effectively, but rarely can tell you who the attacker was – something that's very important to know when it is an insider doing the attacking. When the SIEM system is tightly integrated with your IAM (identity and access management) system, however, not only can you discover the breaches much more quickly, block any “holes” but also identify who is behind the attacks and take appropriate action. For example:

Recently, a Canada Revenue Agency tax collector used her privileged access to view thousands of records to find high-income citizens whom she could later hit up for a business she ran on the side.

One company required three signatures for large payouts, which they could prevent embezzlement. One manager got around this by maintaining the accounts of departed subordinates then using their electronic signatures for the “independent approvals” needed to okay expenditures she created. By the time this was discovered, she had stolen $11 million.

A man working for an airline's customer service department reported false complaints, using his personal bank account as a beneficiary. Then he authorized payments. The employee also reopened old cases against the airline and replaced the original account by his own.

All three might have eventually been stopped by a good SIEM system. But if that SIEM was tightly coupled with a good IAM system, then little or no loss would have occurred. The Canadian tax agent would have been tagged for unusual activity and called in for an explanation. The “puppet master” would have been blocked by the company's de-provisioning system (part of IAM) from accessing the electronic signatures of her subordinates. The airline employee would have been stopped by a Separation of Duties (SOD) policy, part of an access governance module of a good IAM system.

Only an integrated IAM and SIEM system can monitor and correlate all of these activities. Only a system with a real-time, enterprise-wide view can prevent them from occurring. By combining security monitoring with identity management including access management and user provisioning, you can deliver business process automation that provides users with the appropriate resources, validated in real time, to ensure compliance with company policies—eliminating the gaps that have left so many companies at risk. If your solution isn't doing that, it is time to take a second look at the options.

– Chase Jones, solution marketing manager, Novell