How to cut patchwork… and save a cool $100m


U.S. Air Force CIO John Gilligan was frustrated. With new software vulnerabilities popping up all the time, the Air Force was spending more money on patching systems than on the software itself.

"The point I made to Microsoft, a little over three years ago, was that the business model was a little out of whack," recalls Gilligan. "We would be far more willing to put that money and those resources towards better quality software and better mechanisms for fixing flaws when we found them."

So, in the summer of 2003, when the USAF began talking to Microsoft CEO Steve Ballmer about consolidating its many contracts for the vendor's products and services, Gilligan made sure improved security was the priority.

The multi-year, roughly $500 million deal he hammered out consolidates the Air Force's 38 software contracts and nine support contracts into two enterprise-wide agreements, producing three or four standard configurations that enforce strict security policies for all Microsoft desktop and server software. All 525,000 Air Force personnel will be required to use the standard settings to obtain network access through a policy called "comply or do not connect."

Having standard configurations will streamline the patching process, says Gilligan. The Air Force has hundreds of configurations across its desktops. Patching is a time-consuming procedure that combines manual and automated processes to test and install fixes.

Under the new deal, the Air Force will work closely with Microsoft when new vulnerabilities are discovered, test patches against its standard settings, and then rapidly distribute them using automated capabilities. The contract consolidation is expected to save more than $100 million over six years.

Sharing the knowledge

Gilligan says the Air Force is sharing its Microsoft experience with other organizations, both inside and outside of the government. "We're going to see whether or not the literal configurations that we're using could be used. If not, then certainly the model. There's nothing particularly unique about the model we have developed," he says.

Alan Paller, director of research at the SANS Institute, calls Gilligan a "hero" for striking a deal that he says will ultimately benefit many.

"He's testing the safe implementations and the patches against those implementations. So anyone else who adopts those configurations can avoid tens of millions of dollars of testing and can implement patches instantaneously, instead of weeks or months after they come out," explains Paller.

Paller, who has known Gilligan since his work as the Energy Department's CIO about seven years ago, describes him as a thoughtful and careful person who has earned the trust of the people running USAF. "But his principal characteristic is that he acts on his convictions," adds Paller. "Too many people whine and do nothing. It's nice to have someone who says 'I'm going to fix it'."

Gilligan hopes the Air Force can work out similar agreements with other vendors. Oracle is a prime candidate, requiring extending an existing enterprise contract. Others include suppliers of Linux and web services infrastructure.

Testing the security

According to Gilligan, a new vulnerability is discovered nearly every day in the commercial software products the Air Force uses – not just Microsoft, but also Linux, Oracle and Cisco Systems.

"What we are now reaping is the unfortunate consequence of an era of software development in the 90s, when the rush to get the product to market overrode the importance of correctness in the quality of the software."

And in a time of "net-centric warfare," a highly reliable network is essential for the Air Force. "The military, as most organizations are, is increasingly using commercially developed software," he adds. "When it proves to have significant vulnerabilities, that could, at a minimum, deny us the ability to use that system. That's a big deal."

With so much critical infrastructure using commercial software, more energy must be spent on improving the quality of software out of the box, "because from our perspective, the nation's security depends on it," declares Gilligan.

As planned, initial configurations will be ready for testing in April or May, says Kenneth Heitkamp, Air Force assistant CIO for lifecycle management.

"We'll go through a deliberate series of tests. We're not only addressing security, but also performance and features. It has to be a very deliberate process for an enterprise this large," he explains.

As well as Microsoft desktop Windows and Office software, the agreement includes Windows Server 2003, Exchange Server, Systems Management Server, SQL Server, and Office SharePoint Portal Server. Dell is supplying the software licenses; Microsoft consultants are supplying the support services.

Microsoft and USAF are working jointly on developing the secure configurations, using benchmarks from the nonprofit Center for Internet Security (CIS) and data from federal sources, and will build a test center for evaluating patches.

"We can test a new patch on known configurations in an enterprise and deploy it rapidly with a higher degree of confidence than in the past," says Al Horowitz, general manager of Microsoft U.S. public sector consulting services.

"This is a partnership, so we're going to learn a lot from the Air Force and they'll learn a lot from us," he says. "We'll feed the things we learn back to the product teams. That will help us as we continue to focus on security and new updates to our software."

Other organizations might also benefit, notes Curt Kolcun, general manager of Microsoft federal. The software giant is working with other federal agencies that have expressed interest in using the Air Force configurations, he says.

Clint Kreitner, CIS president and CEO, says that the Air Force "is doing exactly the right thing in terms of cleaning up and standardizing its operational structure and IT infrastructure, and being really clear about what it wants to buy."

Windows comes with thousands of security settings, such as password length and how long a PC idles before shutting off, but the prevailing practice by Microsoft and other vendors has been to ship their products with the security settings off, he says.

Necessary configurations

The CIS benchmarks represent a consensus among security experts from the public and private sectors on necessary security configurations.

"One essential requirement to clean up cyberspace is to move towards a greater level of standardization," maintains Kreitner.

Noting that "system complexity is the enemy of security," Adam Lipson, president and CEO of consultancy Network & Security Technologies, says the Air Force's Microsoft initiative is a big step in the right direction, "and should be taken as a cue by other, larger organizations that buy from Microsoft and other technology vendors."

Standard policies and procedures for making Microsoft software more secure have been around for years, but even if the operating system is secure, the applications installed on top of it might not be, stresses Chris Hoff, CISO at California-based Western Corporate Federal Credit Union (WesCorp).

"Companies and large governmental organizations exerting pressure to get more secure operating systems is a fine idea, as long as it translates to things that more people can use," he adds. "If the results are proprietary and/or classified, or things they have changed or added don't translate well to the end user community, it doesn't add value."

Gilligan, meanwhile, is pleased with the direction the Air Force is taking as he prepares for a reorganization that will combine his office with warfighting integration and communications operations in a single directorate. Plans call for a three-star general to head the new directorate and serve as the new CIO.

Although it is unclear what Gilligan will do after the reorganization, he calls it a "very positive step." He also says the software industry is moving in the right direction, albeit slowly, from an era of buyer beware to one in which software makers will be held accountable.

"The industry is taking very seriously its obligation to improve quality," he says. "It is a good trend, but it is not an overnight process. It will take years."

Get daily email updates

SC Media's daily must-read of the most current and pressing daily news

By clicking the Subscribe button below, you agree to SC Media Terms and Conditions and Privacy Policy.