The widespread impact of the SQL Slammer worm in January 2003 highlighted an issue that most security professionals see as a basic fact - IT system security is not based on a single installation, but on an ongoing management of risks and vulnerabilities.
The Slammer worm, through sheer volume of traffic, managed to shut down various commercial services and internet servers, but was based on a vulnerability addressed in a hotfix issued by Microsoft six months previously. This software patch was freely available, but was not widely installed, so an issue that should have been easily avoided became a worldwide concern.
In this particular case many IT administrators pointed to the fact that installation of the original patch was a complex process, and that this was one reason why it was not widely deployed. Other administrators were simply not aware of the widespread use of the SQL engine on the systems under their control - particularly on desktop and client machines - and so did not anticipate vulnerability in their systems.
It would be easy to point the blame at administrators in this case, and some are probably guilty of poor systems management. However, the sheer scope of the damage indicates the complexity of the issue of patch management and hotfix control.
Most IT systems, even IT security systems, are inherently insecure by the time they're installed on customer sites. There are two main reasons for this - sloppy programming (leaving software open to buffer overflow and input validation vulnerabilities) and the simple fact that as systems and software change, vulnerabilities arise which were not previously anticipated or tested. This second area also includes the more malicious approach - where a new product or release is scanned for vulnerabilities, with a view to exploiting these vulnerabilities in a manner to benefit the attacker.
Vulnerabilities discovered in this way are often publicized through email lists and web forums, and may then be used by other attackers to identify and exploit exposed systems. In many cases vulnerabilities are disclosed to vendors initially, and only publicized after the vendor has had sufficient time to issue a patch. Some vendors have argued that this process increases the likelihood of attacks, as they often do not get sufficient time to produce and test the update, but many security professionals feel that this form of disclosure helps ensure that major risks are identified quickly, and patches issued promptly.
It seems unlikely that these issues are going to go away. Many larger applications are written by teams of software developers in different geographical locations, and to put it simply, mistakes happen. Despite the wealth of testing that often takes place it is practically impossible to test all combinations of hardware, operating system and software, so co-existence problems can arise. These issues are not confined to the world of Microsoft Windows - all commercial applications, regardless of operating system, suffer from the same inherent issues. Different programming techniques can mitigate the risk of code error, but interoperability testing continues to challenge the most efficient development teams.
Whatever the cause, the insecurities are normally addressed through a software update. These are known as patches, updates, hotfixes, service packs, or in the Check Point firewall community - feature packs. Check Point, like many other vendors, issues new features and functions in the same software bundle as the repair code used to address vulnerabilities and programming bugs. Such updates are normally available only to customers who have purchased an annual support subscription, which can cause difficulties for anyone who feels that bugfixes to 'faulty' software should be provided free of charge.
As we saw in the SQL Slammer example, issuing or making available a patch, and having that patch installed on all vulnerable systems, are two completely different things, and different vendors take different approaches to ensuring that the updates are applied. These range from the passive (providing a notice of the patch and the issues it addresses on the company website or subscription mailing list), the active (forcing a pop-up window to appear within the software to highlight the availability of a new update), and the integrated (where the responsibility for updating the software is handed over to the vendor, and updates are pushed to the client as and when available).
Obviously the integrated approach offers the easiest solution for the IT administrator. However, taking this approach raises some questions:
- Do you trust the vendor to provide stable updates to what essentially could be viewed as 'faulty' software in the first place?
- Do you trust external personnel to manage what is one of the key components of your security policy - IT systems management?
- Do you trust that the vendor has carried out sufficient testing on the update - is it safe to download and deploy patches and hotfixes without first testing them in your own lab?
- Do you trust that the organization pushing updates to your systems is who they say they are, and not a malicious third-party masquerading as your software vendor?
Generally the patches or updates issued do exactly what they're supposed to do - they address a vulnerability or coding error in the software. Occasionally, however, the patches introduce an additional vulnerability, or cause other problems. We have recorded several instances where one of the major anti-virus software vendors (not one of ours!) has issued a new anti-virus engine which subsequently crashed the systems on which it was installed. In some cases the update was carried out automatically, overnight, and the damage was only discovered a number of hours later.
The ease of automatic deployment from the vendor and the advantages of applying an urgent update as soon as it becomes available, must therefore be offset against the potential risk of system failure. Patch updates, like any new software deployment, should follow the corporate change control procedure.
As with many components of a corporate security policy, patch management should be subjected to a risk analysis to identify the most appropriate method of managing updates to software applications throughout the organization.
Some of the tools and services available from large vendors are outlined below. There are other steps that IT administrators can take to improve the efficiency of the patch management process:
- Be aware of the available updates and patches, and the issues they address. This can generally be accomplished by visiting the vendor web sites regularly, or by subscribing to mailing lists or user forums.
- Manage the update process. Where possible, download the update and test in a lab or non-critical environment before deploying to all affected systems.
- Use tools where appropriate. In all but the smallest organizations managing the updates to all systems (and you have to include ALL systems) can be difficult. Various tools are available to help in the deployment and in the auditing after deployment, to ensure that the update process has been successful.
There's an old adage that best sums up the process: "Prevention is better than cure". It's generally a lot easier to patch a system than to clean-up after a virus outbreak or systems failure.
The Tools
Check Point SmartUpdate allows products and patches on Check Point and integrated OPSEC products to be managed from a single interface. Firewalls can be upgraded, and applications installed and removed without having to physically connect to each machine. Updates are downloaded centrally, and stored in a repository, from where they can be pushed out as needed. A single screen shows the current status of all managed products.
Check Point SmartDefense offers a subscription service which allows firewalls to be configured to automatically block new attacks by type and class, and ensures that the system configuration is kept up-to-date with emerging risks.
Trend Micro Control Manager is a central management and reporting tool for all Trend Micro anti-virus applications. Updates can be downloaded to a single machine from the Trend website, and pushed out to clients as necessary. The updates can be automatically or manually deployed. The administration screen gives a view of the current status of all managed applications, making it easy to identify out-of-date systems.
Trend Micro Outbreak Protection Services integrate with TMCM (above) to minimize the risk from new and emerging computer viruses. Virus profiles are issued prior to the formal release of the virus pattern update, and these profiles can be used to quarantine any suspicious content. On release of the pattern update the quarantined information can be scanned, and if deemed 'safe' released.
St. Bernard Software UpdateExpert offers centralized management for Microsoft hotfixes and service packs. The administration console scans servers and client machines for product versions, identifies and provides information on any required patches, and allows these updates to be downloaded and deployed in a controlled fashion.
All of the above systems offer a variety of reporting functions, and allow an automated or a manually controlled deployment of updates - i.e. first update to a test machine, then to all systems.
Paul Collins is senior security analyst for Sentryst Ltd (www.sentryst.com).
Sentryst are exhibiting at Infosecurity Europe, Europe's largest and most important information security event. Now in its 8th year, the show features Europe's most comprehensive FREE education program, and over 200 exhibitors at the Grand Hall at Olympia from April 29- May 1, 2003. www.infosec.co.uk
