Product Group Tests

Managed security services 2007

Group Summary

With no deficiencies in any tested categories and several strengths, Alert Logic Threat Manager was an all around superior offering which earned our Best Buy rating.

Solutionary ActiveGuard is a very good offering which exceeds all of the contracted terms, and provides a valuable service for clients. These components make the Solutionary offering our Recommended product.

Scroll To Full Group Summary Below

Click for a side by side comparison of products
Click for a side by side comparison of products

Full Group Summary

Managed security service providers (MSSPs) often are an enigma for information security professionals. Many MSSPs have been driven out of business as more organizations have looked to in-sourcing as the best option for managing security devices. When outsourcing of security services is discussed, the focus almost immediately moves to the large telephony and internet service providers.

We invited over 25 service providers to take part in this month’s Group Test, including many of the large telecom companies, some major security providers and many smaller organizations that focused exclusively on managed security services. In the end, we ended up with many organizations unable to participate for a myriad of reasons, and we eventually were left with just four providers to test. We question why this would be, given that this is a hotly contested market.

In the end, we concluded that a combination of slow response — mostly in the largest organizations — and a lack of understanding that we were interested in reviewing services, not products, for this Group Test review, were major contributors.

The four providers that remained all had above average offerings. Of the organizations that submitted offerings, each had a unique slant on the managed security space. One provider submitted an offering that included managed forensics, another provider focused on lower costs per managed device, and another provider focused on the ability to manage applications.

In general, when you are looking for a managed security service provider, it is important to understand the needs of your own organization. If you are looking for a provider that provides functionality as first responder or supplemental computer incident response team member, then the Service Level Agreements (SLAs) and SLA compliance are critical.

If your organization is looking for total threat management, including monitoring client side requests that may violate security policy and possibly lead to network compromise, then only co-location offerings can meet those needs. If your organization is looking for overall security, it is important to ask about the technology used by the provider.

If regulatory or audit compliance is the only reason your organization needs to invest in a managed security service provider (although we don’t recommend this approach), then the lowest cost should be a big decision-making element for you.

How we tested    
We tested each of the managed security service providers by requesting a copy of the service level agreements and looking for the guaranteed response times of the SLA. This became the baseline against which we judged the providers. We were testing for four key categories from the MSSPs. These categories were: response time as judged by the SLA; overall security of the offering; detail and speed of logging of events; reporting provided by the service.

To test the providers we used a multiphase approach that began with a port scan using the nmap utility. We used the following command with nmap: nmap - sS - A - O target.

Once the nmap scan was completed, we waited for the low level SLA response time to elapse or until we were contacted by the service provider.

Once the test was complete, we moved to medium level testing with the Nessus utility and the Web Inspect utility from SPIdynamics. These tests provided a level of testing of the network vulnerability (through Nessus) and a targeted application level vulnerability scan (through WebInspect). We then waited for the SLA response time for medium to elapse or until we were contacted by the vendor.

Finally, we tested each protected system with the Core Impact utility from Core Security. This utility allowed us to send directed buffer overflow attacks against the target application. Once again, on completion of the test we waited for SLA time to elapse or until we were contacted by the provider. The nmap scan was considered a low level event, the Nessus and WebInspect scans were medium level events, and the Core Impact testing was considered the high level scanning.

Overall, we concluded from this testing that if you need an MSSP, don’t be dazzled by the big guns in the marketplace. There are some powerful little guys that want your business and can really produce for you.

- Peter Stephenson contributed to this Group Test.

All Products In This Group Test