How are CSOs coping with the growing burden of software patches? Julie Jervus sought the opinions of four practitioners with strong views on the matter
In 1999, security experts at Carnegie Mellon University's CERT Coordination Center reported almost 10,000 incidents of security violations that involved the internet. In just the first six months of 2003, CERT reported a staggering 76,000 incidents.
With Microsoft dominant in over 90 percent of the desktop market, most experts are not surprised that hackers, intrusionists and cyberterrorists are targeting vulnerabilities in Microsoft products. In September, the Computer and Communications Industry Association warned that Microsoft's continued security practices and monopoly threatens the security of the U.S. economy and its critical infrastructure.
"A lot of our larger Microsoft customers don't feel that the status quo can be maintained," says Mark Nicolett, director of security and privacy at Gartner. "There will be a backlash. Many companies are trying to understand the notion of introducing diversity into their environment and most are now looking at how to patch faster."
While viruses like Blaster, Sven, and SQL Slammer have become household names, patch management has become a routine fire drill for IT departments. Here, chief security experts in five companies across the United States share their experiences, frustrations and approaches in the ongoing struggle to keep their systems secure. For Bruce Peck, information security manager at St. Vincent Hospital in Indianapolis, patch management is rapidly becoming a matter of life or death. St Vincent is a large hospital with about 4,000 desktops and 300 servers, most using Microsoft products. Peck feels Microsoft has them over a barrel and says he's become numb to all the vulnerabilities and exploits.
"At first everyone was angry about the constant parade of viruses and patches," he says. "There were immediate calls to go to Linux and get rid of Microsoft, but in reality we just can't do that. We have to rely on Microsoft to make the situation better and react better ourselves to patching and remediation."
Many of their systems have paper and pencil back-up, says Peck, but he has become increasingly concerned about the number of Windows-based medical devices that run on the network and have IP addresses. "These are devices that calculate and measure drug doses for chemotherapy patients, for instance," says Peck. "I'm concerned a worm could negatively impact them."
Although the recent Blaster worm did very little damage, last year SQL Slammer disrupted the network for 24 hours, damaging files and creating denials of service. "If the network goes down, it may not have a direct affect, but could have an indirect affect on patients' health when the information doctors need is not available," says Peck.
Peck says that the hospital's IT groups have become better at patching and now rely on several third party tools. Patching desktops is the major problem, since the hospital runs 24/7 and the workstations are in constant use.
In addition, Peck says that it's almost impossible to maintain a test environment that can emulate the complexity of their systems and avoid disabling some of those applications. "No one has calculated what this is costing," says Peck. "But it's having a toll. With the latest threat, a lot of people had to drop what they were doing and other projects suffered. I hope that in the future Microsoft could make its products stronger."
David Jordan is frustrated with Microsoft because of the number of interruptions to his work plan, and the associated costs. In fact, he jokes, he's already considered billing Microsoft for all his team's integration testing. Jordan is the chief information security officer for Arlington County, Virginia, home to the Pentagon and around 3,000 county government desktops and servers.
"We're pretty much a Microsoft shop, and will be totally reliant on them by 2004," said Jordan. "We're moving over to only Microsoft on servers, and will be using Outlook. I'm not looking forward to that - I call it 'Lookout!' Most people don't write viruses for anything but Outlook." Nevertheless, Jordan is confident that his multi-faceted approach to security is the main reason why the county's computer systems have 30 months of no downtime.
As well as a virus shield, anti-spam measures and "surgical IP blocking," Jordan is spreading the word to his constituents about the importance of IT security. "Most viruses that have been detected came from employees bringing in discs," says Jordan. "ISPs rarely advertise the risks of full-time connection to the internet, so we've launched an educational program. Our employees are now alert for things that don't look right, which is just as important as expensive software."
Although Jordan feels Microsoft never built their software products with integrity in mind, he also blames the internet's architects for many of today's security issues, especially the allocation of IP addresses. "The internet came from academia," he says. "It's all open source - a cool system, but when it's time to tie it down, it's very difficult."
Jordan's security team is small and has to move quickly when a patch is released. "Every time there's a fire drill, work stops," he says. "The people in this team have big hat racks, and we know we're spending too much time doing these patching pushes, when we could be doing things we want to do. We're pretty much automated now, but it's frustrating that we're paying for a software license and then paying for a machine to patch it."
Gartner's Nicolett says many customers are complaining about having to shoulder the expense and disruption of installing patches. "Patch management is a software distribution problem," says Nicolett. "Microsoft has added patch management support to SMS, but you have to pay for it. They also provide free tools to automate the process and patch analysis tools, but the free tools are not as good as a full-blown software distribution product and lack fine grain release control."
Adam Hansen is the Lead Information Security Engineer at Sonnenschein law firm, responsible for all aspects of IT security for more than 2,000 workstations and servers in ten locations throughout the United States.
His main concern is that the time between identifying vulnerabilities and exploits is getting compressed and companies are not getting enough notice to keep up. "It's not necessarily Micro- soft's fault," says Hansen. "Whenever you have a platform that's complex and a number of people with a vendetta you can expect this."
"The people that write these viruses are getting smarter; they share and collaborate better than corporate America," he adds. Hansen believes Microsoft is doing everything humanly possible, short of doing the patching themselves. "There's no excuse for not getting a patch management system," says Hansen, whose company experienced a 24-hour outage after a non-standardized machine in one of its offices was hit by a virus. "Experiencing an outage as a result of a lack of patching will end up costing you more. The answer is to work smarter, not harder."
After evaluating several products last March, Hansen introduced a patch management system, firewall and intrusion detection software. "The first step is to identify and standardize your systems," said Hansen. "It may require a shift in corporate culture but we have to get better at managing assets on the LAN. I'm constantly thinking up scenarios and solutions and I know that if I've thought of it, someone else has, too."
Not all companies can afford a comprehensive security management system, however. Scott King is a software R&D engineer with HighTower Software, a small, young, private company that develops information monitoring and management software applications. Based in Aliso Viejo, California, the company out-sources a significant percentage of IT support for its corporate infrastructure.
However, King's extensive experience in software development and security means that he and his team are the ones frequently called on to examine the company's computer security issues and give advice. "A lot of the IT outsource companies are not security conscious," he says. "They won't recommend patch management systems, or even automatic updates."
HighTower has to patch its own systems and, with 100 workstations and 20 servers to protect, King says that at least two days are wasted each time. His main frustrations are with the number of patches and the testing necessary to ensure nothing else breaks because of the patch, especially third party applications. "I just installed a new system
on a Windows 2000 server," says King. "There are at least 12 patches or service packs for the operating system alone. Then you have to prepare for something breaking because it was built to run on Windows at a certain service level. I've seen it happen."
Although King believes that many of Microsoft products were written insecurely, he has also seen security problems with Linux and Solaris systems. "The major reason Microsoft products are not up to snuff with a lot of other software is because it didn't have enough people looking at it from a security standpoint," says King. "Microsoft uses closed source code so security researchers don't have access to see what's written insecurely."
William Woloszyn believes that things have gotten a little better with the last few products Microsoft has rolled out, but is still frustrated at having to allocate precious resources to keep up with the long list of patches. Woloszyn is the director of privacy and security at INTEGRIS Health, a not-for-profit health care system in Oklahoma.
INTEGRIS employs around 9,000 workers and its 3,800 workstations, scattered in over a dozen locations, all use Microsoft products. "We've been affected by security flaws in the past," says Woloszyn. "But we've been able to put together a proactive containment plan. The problem is that not all systems are equal and not all can have the same patch applied and co-exist with other applications and services. It's one thing to issue an alert, another to look at it, test it and make sure we're not creating a disaster by rolling out the patch. That's where the industry has a lot of frustration."
Woloszyn is currently looking at third party patch management products but has yet to commit to any one solution. "Right now we have to do threat analysis, look at the impact and how it compares in importance with everything else on our plate. We can roll out a patch in a matter of hours, but the question is whether we want to shift the resources."
Woloszyn has seen the industry "beat up on Microsoft quite a bit" and shares his colleagues' frustrations at feeling like a captive audience. "Personally, I believe Microsoft gets hit on because of its size," says Woloszyn. "Intrusionists are looking for the biggest bang for their buck. A virus is a virus, whether it hits a Microsoft server or one of the others. What's secure today isn't necessarily secure tomorrow. It's never boring."