Historically, log management and security information and event management (SIEM) have been considered distinct functions and therefore warranted distinct applications. Although the majority of vendors initially developed solutions that focused only on logs or security events, the two functions are inextricably linked. Over time, organizations have come to realize the advantages of a single integrated solution to handle both log and event management, and are now demanding all-in-one systems. The benefits are many and include: greater compliance automation, broader visibility to threats and risks, faster and more effective incident response, improved reporting and a substantial reduction in implementation and administration costs.
SIEM was invented to solve the data overload problem, primarily from intrusion detection systems (IDSs). The idea was to take IDSs, firewall logs and other security data and reduce multiple related logs to a highly correlated single event. Simplification has great value, but in this data reduction process SIEMs discarded a lot of valuable forensic information which was permanently lost. In contrast, log management was developed to collect and store all of the underlying forensic information, and to make this data readily available. Keeping all of the log data in its original, unfiltered, unmodified state preserves evidence and is less likely to be misinterpreted. What evolved were two different types of solutions: log management solutions that collected and stored logs in their original form, and security event management solutions (SEMs) that correlate and simplify the monitoring of security events...
Since logs are the underlying source for generating events, integrating both functions in a unified system that collects and saves all logs, provides a correlated view of events, and allows access to the raw data when it is needed is a natural evolution. For example, by having a single solution, when an IDS alarm goes off, it is possible to then pull all of the raw logs from a web server corresponding to that attack and corroborate what the IDS is saying. This provides the forensic data to truly understand the incident from beginning to end.
Once logs and events are integrated, one of the most effective ways to garner insight into security and audit/compliance issues is to apply advanced analytical techniques to this data. For example, some sophisticated log and event management systems provide a common taxonomy for the classification and description of log messages in a more readable form. Some include a built-in knowledge base that describes actions to take when specific log messages are observed. Others can automate the detection of more sophisticated events by performing statistical and pattern-based analysis of log messages and trends. This capability could detect the same user trying but failing to access sensitive files repeatedly over the course of a day.
The advancements in log and event management are extending into file and endpoint security. Some forward thinking companies have integrated file integrity and endpoint monitoring with log management and SIEM functionality, providing an even greater level of security. By fully integrating log management and SIEM functionality with file and endpoint monitoring, the collective value of all functions grows substantially. For example, security personnel can be notified in near real-time when sensitive files are changed, deleted, etc., and the activities can be traced back to an individual user. These capabilities allow organizations to meet additional regulatory compliance requirements, such as Payment Card Industry Data Security Standard (PCI DSS) 11.5 and 12.9, without purchasing a separate product.
Similarly, if an employee were attempting to move highly sensitive data from his/her laptop to a removable media device, these advanced security capabilities could log the activity in near real-time, report on it, and if the event violated a pre-defined policy, the system could automatically send an alert to the security team. Some organizations may even choose to leverage endpoint monitoring tools to block movement of data to removable media altogether.
Whether an organization is looking at log management for operations, security, compliance, or a combination of all three, the market is clearly moving toward solutions that integrate log data with additional operational and security information. We are only at the beginning of evolution in terms of the meaningful intelligence that can be derived from log and event data. New analysis functionality will allow us to better detect threats from both internal and external sources that organizations cannot see today. In the end, this innovation is being made possible by the need for a comprehensive view into IT security and infrastructure through the consolidation of log and security event data.
Eric Knight is senior knowledge engineer at log and event management vendor LogRhythm. He is a certified ethical hacker and has over 15 years of experience in the field of network security with an emphasis in vulnerability management and enterprise security architectures.
SIEM was invented to solve the data overload problem, primarily from intrusion detection systems (IDSs). The idea was to take IDSs, firewall logs and other security data and reduce multiple related logs to a highly correlated single event. Simplification has great value, but in this data reduction process SIEMs discarded a lot of valuable forensic information which was permanently lost. In contrast, log management was developed to collect and store all of the underlying forensic information, and to make this data readily available. Keeping all of the log data in its original, unfiltered, unmodified state preserves evidence and is less likely to be misinterpreted. What evolved were two different types of solutions: log management solutions that collected and stored logs in their original form, and security event management solutions (SEMs) that correlate and simplify the monitoring of security events...
Since logs are the underlying source for generating events, integrating both functions in a unified system that collects and saves all logs, provides a correlated view of events, and allows access to the raw data when it is needed is a natural evolution. For example, by having a single solution, when an IDS alarm goes off, it is possible to then pull all of the raw logs from a web server corresponding to that attack and corroborate what the IDS is saying. This provides the forensic data to truly understand the incident from beginning to end.
Once logs and events are integrated, one of the most effective ways to garner insight into security and audit/compliance issues is to apply advanced analytical techniques to this data. For example, some sophisticated log and event management systems provide a common taxonomy for the classification and description of log messages in a more readable form. Some include a built-in knowledge base that describes actions to take when specific log messages are observed. Others can automate the detection of more sophisticated events by performing statistical and pattern-based analysis of log messages and trends. This capability could detect the same user trying but failing to access sensitive files repeatedly over the course of a day.
The advancements in log and event management are extending into file and endpoint security. Some forward thinking companies have integrated file integrity and endpoint monitoring with log management and SIEM functionality, providing an even greater level of security. By fully integrating log management and SIEM functionality with file and endpoint monitoring, the collective value of all functions grows substantially. For example, security personnel can be notified in near real-time when sensitive files are changed, deleted, etc., and the activities can be traced back to an individual user. These capabilities allow organizations to meet additional regulatory compliance requirements, such as Payment Card Industry Data Security Standard (PCI DSS) 11.5 and 12.9, without purchasing a separate product.
Similarly, if an employee were attempting to move highly sensitive data from his/her laptop to a removable media device, these advanced security capabilities could log the activity in near real-time, report on it, and if the event violated a pre-defined policy, the system could automatically send an alert to the security team. Some organizations may even choose to leverage endpoint monitoring tools to block movement of data to removable media altogether.
Whether an organization is looking at log management for operations, security, compliance, or a combination of all three, the market is clearly moving toward solutions that integrate log data with additional operational and security information. We are only at the beginning of evolution in terms of the meaningful intelligence that can be derived from log and event data. New analysis functionality will allow us to better detect threats from both internal and external sources that organizations cannot see today. In the end, this innovation is being made possible by the need for a comprehensive view into IT security and infrastructure through the consolidation of log and security event data.
Eric Knight is senior knowledge engineer at log and event management vendor LogRhythm. He is a certified ethical hacker and has over 15 years of experience in the field of network security with an emphasis in vulnerability management and enterprise security architectures.
