Strengthening change and configuration management practices is critical to any organization, reports Angela Moscaritolo.
Despite having its offices in Port-au-Prince, Haiti, destroyed this January by an earthquake, the United Nations Children's Fund (UNICEF) was ready to provide support to victims soon after the catastrophe.
The process of getting its offices back up and running was not easy or without setbacks, says Stephen Fridakis (right), chief of IT programs and quality assurance at the humanitarian organization. But having strong change and configuration management (CCM) practices helped the IT division to quickly transition operations to a new location nearby while supporting the organization's critical mission of delivering help to those in need.
Configuration management is defined as the process for establishing, standardizing and deploying the IT settings necessary to achieve IT business service requirements, while change management is the process for identifying, authorizing and implementing alterations to an IT infrastructure that are not part of the existing baseline, says Steve Brasen, senior analyst at Enterprise Management Associates (EMA).
And such practices are critical to maintaining an overall level of security for a solution, system or even an entire organization, says Shay Zalalichin, CTO at information risk management consulting firm Comsec Consulting. In fact, according to a newly updated report from the Open Web Application Security Project (OWASP), security misconfigurations are one of the most critical web application security risks facing enterprises.
Headquartered in New York, UNICEF has 300 offices around the world, each with its own IT staff. The Port-au-Prince branch has always been one of the smallest facilities the organization operates, Fridakis says. After the 7.0-magnitude quake rocked the Caribbean nation in January, the UNICEF office was ruined.
“We lost all our buildings in Haiti,” Fridakis says.
After the disaster, UNICEF's IT operations were put to the test. Drawing on previous experience from other crises, UNICEF staff members were able to quickly mobilize and transfer the back office operation to a different UNICEF facility in the nearby Dominican Republic.
Complicating the situation, most third-party emergency personnel had computers that weren't compatible with UNICEF's network. Working through this challenge required each outside individual's computer be evaluated to determine whether it could be allowed on the network without impacting operations from a risk management perspective. Some could be accommodated, others could not.
“It wasn't just about providing access, it was determining if access could be enacted on their setup,” says Fridakis.
But the single most important challenge to overcome was getting the Haiti location's enterprise resource planning (ERP) system – used to handle purchase orders – back up and functioning in a different location. Without this system, members of the organization could not order and deploy essential aid to earthquake victims in Haiti, including safe water, sanitation and medical supplies, food or temporary shelter materials.
Overcoming this roadblock required outfitting the Dominican Republic office's ERP system with the appropriate access segregation to handle purchases from two separate countries, with two separate operating budgets, Fridakis explains.
“We had the ability and operational know-how to establish a back office operation that reflected one under crisis and had the appropriate segregation and access rules that could operate within the crisis,” Fridakis says.
The chain of security
OWASP in April released a new version of its Top 10 list of critical web application security risks, a ranking intended to help organizations better secure their web applications and services. Security misconfiguration, number six on the list, occurs when secure configuration settings are not defined and deployed for the application, frameworks, application server, web server, database server and platform. These security misconfigurations are easily exploitable and could allow an attacker to leverage unpatched flaws, unused pages or unprotected files to gain unauthorized access to a system and steal data. Experts say that a majority of external attacks take advantage of poorly administered or misconfigured computing environments.
John Johnson, senior security program manager at John Deere, says that before the change management discipline took hold, misconfigurations and unexpected problems took longer to recover from and had a greater impact on the business.
“I've made it a point to incorporate change management processes into all the security management applications I manage,” he says.
Additionally, strong CCM practices could result in fewer troublesome security events, along with better IT performance and service management, according to the results of a survey by Enterprise Management Associates (EMA). The company surveyed 224 organizations and designated them as high, medium and low performers based, in part, on the number of disruptive security events they experienced. High performers experienced roughly half the amount of disruptive security events compared to those classified as medium and low performers. Virtually all organizations in the high performers group had defined CCM processes, implemented them, monitored the environment for evidence of both authorized and unauthorized IT changes, and responded to unexpected or unauthorized change events.
CCM is also important to ensuring the stability of an environment, says EMA's Brasen. In EMA's survey, organizations classified as high performers also had lower incidences of unplanned IT work, a higher number of successful changes and more projects completed on time and within budget.
“In order for an environment to be reliable you need to make sure it stays consistent within established parameters,” Brasen says.
Process and pain points
CCM is often one of the most difficult challenges for organizations today, experts say.
“It turns out a lot of the problems we run into – security or otherwise – are the result of some changes,” says
Glenn O'Donnell, senior analyst at Forrester Research.
A large school district recently learned this firsthand when implementing a set of configuration standards for their Windows and Unix server environments, says Amrit Williams (left), chief technology officer at BigFix, a provider of CCM software. These configuration standards were not properly communicated, nor were they properly tested prior to implementation. When the organization installed the changes, many of the back office applications stopped functioning and it took months to identify the problem and reverse the process.
More challenges
One of the greatest challenges of a CCM effort is getting buy-in and participation from all stakeholders within an organization.
“The biggest problem with CCM is that it crosses silos,” Brasen says.
This leads to a bigger problem, according to Dwayne Melancon, vice president of log management for Tripwire, a provider of change management solutions. Security gets left out altogether of CCM processes at some organizations. “Once something bad happens, they get called in to clean up,” he says.
Experts say having the support of all stakeholders – including executive staff, enterprise architects, application developers, along with those in IT operations and security – is vital to a successful CCM effort. Being part of the process allows IT security professionals to participate in conversations about how best to manage the risk a change may introduce before it occurs.
Another common pain point is determining what elements to include in a standard configuration, Brasen says. It is helpful to begin the process by initially monitoring the environment to identify where settings can be tweaked from their defaults for optimal performance, security and reliability. Then, monitoring and reporting should be used to identify opportunities for improvement.
The goal is to have consistent configurations, Forrester's O'Donnell says. An organization with numerous Windows servers, for instance, should ensure they are all configured according to the proper policies and security standards.
“If we make sure we have things consistently configured, that winds up eliminating the majority of surprises that cause heartburn,” O'Donnell says.
To thwart security misconfigurations, developers and network administrators must work together to ensure the entire application stack is configured properly, according to OWASP. Development, quality assurance and production environments should be configured identically, and the process should be automated to easily set up a new secure environment. Automatic scanners can also be helpful for detecting missed patches and other misconfigurations.
Because there are so many components to keep under control, CCM is virtually impossible to perform with manual processes, experts say. A single server could, for example, include millions of configuration points. The various computing devices, operating systems, applications and users create too many variables and too much complexity for manual controls or oversight, Williams says.
“Ideally, organizations would move to automate as much of the CCM practice as possible to lower costs and improve efficiencies and effectiveness,” he says.
But, do not try to tackle a CCM process all at once, EMA's Brasen recommends. Instead, implement changes and improvements in a phased process by introducing CCM solutions for one server or set of servers, then move on.
“As you move from reactive firefighting to proactive problem prevention, it frees IT to move to the next project and introduce CCM solutions there,” he says.
Finally, understand that part of the learning process involves making mistakes early on, O'Donnell says. But ultimately, having poor CCM practices is no longer an option.
“CCM, whether you know it as that or not, really is at the core of everything we do,” O'Donnell says. “Getting it right is imperative, especially in the highly dynamic world we live in.”
Dealing with rapid change
Several years ago, a large European company discovered the impact that weak change management can have on security, says Comsec Consulting's Zalalichin. The organization invested a significant amount of time and money developing a new portal for users. Several members of the information security team were involved in the development process to ensure the system had the proper level of security built in.
The portal was successfully deployed, but several months later, members of the business wanted to introduce a new service, which required that a small change be made to the environment. Because of business demand, the change was made quickly, without the proper procedure and testing. As a result, a security vulnerability was introduced into the environment that would have allowed an attacker to take control of the system. The organization discovered the vulnerability and scrambled to fix it before hackers found out about the hole.
“They were lucky,” Zalalichin says. “They noticed it before they were hacked.”
The incident is a stark reminder, however, that an organization can invest ample resources to secure a system, but a small, uncontrolled change can wipe it all away, Zalalichin says.
“If the proper information security measures are not taken, the continuity of the security cycle will be broken,” Zalalichin says. “The chain of security is only as strong as its weakest link and one security bug or breach that results from unmanaged change processes can lead to the unraveling of all of the security controls in a domino-like chain reaction.”
UNICEF is also often forced to deal with rapid changes in emergency situations, Fridakis says. After the earthquake in Haiti, for example, the organization was faced with rebuilding the systems quickly, but also securely.
Without the proper CCM policy and framework, UNICEF would have taken an inordinate amount of time to deliver aid to victims, compromising the humanitarian effort and the organization's mission, Fridakis says. If it is done right, CCM not only offers vast operational benefits, but can also lead to increased security for back office systems, networks and desktops.
“If we apply change management properly, we will have better performance, and our business process will be more efficient,” he says. “If we mess it up, we will create a lot of conflict.”
[sidebars]
CCM: What it takes
The main parts to a configuration and change management effort include:
- Establishing a standard configuration for each IT component
- Creating and following a process for introducing changes to standardized configurations
- Actively monitoring the infrastructure to determine if inappropriate changes have occurred
CCM: The basics
Configuration management: The process for establishing, standardizing and deploying the IT settings necessary to achieve IT business service requirements.
Change management: The process for identifying, authorizing and implementing changes to an IT infrastructure that are not part of the existing enterprise configuration baseline.
