Patching process

According to Rich Bentley, market segment manager for Altiris, it is a choice between speed and taking due care.

"From my experience, it falls into two general categories," Bentley says. "There's the ‘let's get it out as soon as we can and let's patch everything approach.' And then there's the ‘let's go slow, and take a measured approach and prioritize and test before anything goes out.'"

We examine three organizations, one large, one medium and one small, to see how they attempt to strike the right balance when that special Tuesday rolls around each month.

KURT HEITMAN
Position: System support manager, Maintech, 5,000 systems
Shortest patch cycle: One week
Longest patch cycle: Three weeks

Kurt Heitman, system support manager with Maintech, says that his organization tries to reasonably straddle the ground that Bentley mentioned between speed of patch delivery and taking caution against unexpected problems.

"We're somewhere in the middle," he says of the Wallington, N.J.-based company, a market leader in delivering custom-tailored information technology service solutions.

As a public company with over 5,000 users and 100 servers to contend with, Maintech has presented Heitman with myriad challenges when he was tasked several years ago with developing the company's patch management procedure.

"It is a pretty significant effort, and not only from a security perspective. You have to make sure it is Sarbanes-Oxley compliant — at least we do, we're publicly traded," says Heitman. "Environmentally, you can have issues, and a lot depends on your company's philosophy. If you have a very hands-off, don't-touch-my-workstation philosophy, boy, have you got your work cut out for you."

Heitman speaks from experience. "Being a multi-divisional, strategic business unit-type organization it's been very difficult to get everyone to play the same game," he says. He explains that he was able to get everyone on board only through "sheer determination and bullheadedness."

When Patch Tuesday rolls around, the process is set into motion at Maintech with the presentation of all potential changes before two committees, one strategic and one tactical, both of which will look at the patches and determine whether or not they are going to apply them. The strategic group, the Change Management Committee, is comprised of IT representatives from all of the major enterprise divisions at the company. The patching team will seek the committee's "rubber stamp," to make sure the patches fit within the company's change management strategy. The tactical group, made up of security gurus and IT administrators, will discuss all of the technical minutiae surrounding the patches and develop a plan of attack.

"Typically speaking, unless there is a negative connotation with a patch or we know it's going to cause problems with an internal application upfront, we will apply almost all security patches. So from that perspective it is pretty straightforward," Heitman says.

Once the two groups have given their go-aheads, it is on to the quality assurance (QA) process. The change management team will roll out the patches in a test environment for one to two weeks to ferret out any potential problems.

"We have a fairly exhaustive process, and there have been instances where patches have caused concern. There have been issues we've had to roll back from, so we're reasonably cautious and rely quite heavily on our QA process to make sure an update is not going to cause too much difficulty," he says.

JIM CZYZEWSKI
Position: Senior information systems specialist, MidMichigan Health, 2000 systems
Shortest patch cycle: Two days
Longest patch cycle: 2.5 weeks

At MidMichigan Health, Midland, Mich., the process is a bit less arduous. Usually the fun begins the day after Patch Tuesday.

"We have a group meeting in the early afternoon to review the patches and make sure we don't see anything that may be a possible problem," says Jim Czyzewski, senior information systems specialist for MidMichigan. "If we do see a possible problem, we make sure that it gets a little more attention during our testing process."

Unless there are serious issues, the organization begins the testing process on Thursday, Czyzewski says. MidMichigan does not have a test network, so the testing is done in a live environment. Because of this, Czyzewski and his team must take a three-step approach. The first is to start on an isolated system to make sure "the machine doesn't blow up," he says. From there it is on to steps two and three.

"We start with a very small localized group and then we broaden it to make sure there's a representative [user] for each application that we have in the organization," he says.

Testers in both groups are hand-picked power users in each division of the organization that can act as Czyzewski's eyes and ears to make sure there aren't any issues with the changes.

"They are people that we feel are a little more computer savvy and a little more aware of anything that might be out of the ordinary that a regular user might not notice," he says. Each group gets about a week to test before he rolls out the patches organization-wide.

While the thought of live testing may make some security professionals queasy, the ad hoc testing procedure does allow Czyzewski the flexibility to respond quickly to extremely critical patches.

"That process actually is during a normal patch cycle for something that there are no known threats for at the time," he says. "When there are highly critical patches with threats we are at-risk for in house, we can push up the process a bit."

For extreme risks, Czyzewski and his team can speed up testing turnaround to 24 hours. "We basically tell our test users to hammer at it as quick as possible, make it known that this patch is going out, and make sure apps work," he says.

That one-day turnaround is only for the highest-risk vulnerabilities. For medium-risk patches the turnaround is usually a week. Everything else takes approximately two-and-a-half weeks of testing, he says.

"We're actually laid back now that we have a product that provides us accurate patching and reporting. We feel comfortable with the process," he says.

GABRIEL SELMI
Position: Network administrator, Advanced Behavioral Health, Inc., 200 users
Shortest patching cycle: One week
Longest patching cycle: One month

Advanced Behavioral Health, Middletown, Conn., is a small organization with only about 200 computer users. As such, network administrator Gabriel Selmi doesn't have to deal with the bureaucracy of committee rule. But simplicity does have its downside: he's also the sole IT staffer responsible for keeping up on threats and potential patching issues.

Selmi says he tries to take a proactive approach by keeping up with security and bug tracking bulletins and leveraging the assets he has. Because he doesn't have much in the way of IT resources for testing, he depends on intelligence gathered by his patch management vendor, Patch Link, to give him the first approval on patches to ensure there are no known issues.

"Basically, for Patch Tuesday I hold off on the patches until PatchLink says they are OK, and then at that point I'll throw it out on my test network," he says.

"If it doesn't alter anything immediately, then I'll push to certain PCs I have images of for backups," he says. "I'll usually go roughly for about a week of testing with 10 or 15 users in different areas and get a feel for how [the patches] react in different environments."

He says that the goal from start to finish is the full month because he likes to give himself a little leeway. "It gives us a nice cushion if we shoot for a month."

Because he doesn't have a lot of other people in IT, he prefers to be cautious, even with highly critical vulnerabilities. "My experience with some of these patches that come out immediately from Microsoft is that they're not 100 percent certain of what's going to happen within their own environment," he says. "We're a 365 by 24/7 operation and we don't have any major redundancies in place in the systems that we can activate. So I tend to take the cautionary path even if there's major risk."

While no organization will have the exact problems your company has, it sometimes pays to learn from practitioners like Selmi, Czyzewski and Heitman. Although each handles Patch Tuesday differently, there is one common theme to their approach. Each has planned out their patching procedures and works on a predetermined schedule to minimize having to make hasty decisions.

"Sometimes, the number of patches that come out are pretty incredible, and there's a tremendous amount of pain associated with all of that. But by and large, it is like everything else," says Heitman. "Once you get a procedure working, it is not so much of a problem."

GETTING INVOLVED:
Organized crime