Karl Hart doesn't need to read articles or watch the evening news to know that financially motivated hackers nowadays are finding easy pickings at colleges and universities across the country.
The 35-year-old information security officer (ISO) at the University of Cincinnati need only review some of his own college's event logs to realize that academia's liberal networking environment often acts as an open invite to the criminal element cruising the internet.
“Universities and higher-ed are the first to get hit,” Hart says. “It's almost like a playground. It's like, ‘Let's try it here first.'”
So it is no wonder that Hart is spending a lot of time these days fretting over an easy-to-deploy emerging technology called server virtualization, in which multiple operating systems and applications run on a single machine at the same time. But Hart worries that professors and other collegiate end-users, averse to any centralized control, may overlook security in a rush to deploy this new architecture.
The benefits of the technology are game-changing: virtualization better utilizes system resources and increases performance, while reducing the time and money associated with space, power, cooling and management constraints. It is the future of computing, experts agree. According to a 2006 Yankee Group survey of 750 businesses, 62 percent of respondents already have or plan to deploy a virtualization solution. And further sweetening the pot in this market, vendor giants, such as Intel and Microsoft, now have gotten in on the game.
But even though there have been few, if any, in-the-wild attacks against virtualized machines, virtualization introduces an entirely new threat vector that, if not safeguarded properly, could attract even more security issues than the traditional desktop computing model.
The main concern about virtualization, say experts, is the danger an unprotected host operating system presents. If attackers can compromise the hypervisor — a thin layer of software that runs in the host and serves as the virtualization engine — they may earn free reign over every single guest, or virtual, machine (VM) operating on that host system.
“It's kind of like a single point-of-failure for multiple machines,” Hart says, referring to the hypervisor.
Neil MacDonald, a Gartner vice president, says that if IT departments do not properly plan for virtualization by implementing security best practices, any business gain may be negated.
“Virtualization, by definition, is a layer of abstraction,” he says. “It is a software layer that gives you this abstraction. It's a new layer. People overlook how important that layer is and that it must be secured and properly configured like any other layer in the stack.”
As organizations begin implementing virtualized solutions en masse, hackers will take notice and flaws will be discovered. Vulnerabilities in products from VMware — the Palo Alto, Calif. virtualization software leader — have jumped from just one in 2002 to 34 already this year, according to data recently compiled by Kris Lamb, director of the X-Force research team at IBM Internet Security Systems. A representative from VMware could not be reached for this story.
With the interest in virtualization growing, many experts believe it's only a matter of time before hackers start taking advantage of associated unprotected vulnerabilities that the technology presents.
“I believe that virtual machines are going to be the next great unexplored frontier for black hats,” says Chris Richter, vice president and general manager of managed security services at Savvis, a St. Louis area-based global IT services provider. “There's going to be a rush to develop new exploits for this platform.”
So just what are the dangers? Well, they're not much different than threats facing the traditional server environment, experts say.
“What is true is that virtualization brings the ability to compartmentalize and segment more effectively than a physical server implementation,” Lamb says.
But what business executives have failed to think about is that just because something is running virtually rather than physically on a server doesn't mean that the same problems of compliance, risk and OS hardening don't exist, he adds.
Key to success
The key, MacDonald says, is to make the virtual layer as thin as possible, thereby limiting the complexity and, in turn, the number of potential vulnerabilities. Simon Crosby, chief technology officer at open-source virtualization software maker XenSource (acquired by Citrix for $500 million), says hypervisors only contain about 60,000 lines of code compared to millions on a desktop OS.
But if an attacker can find his way in — XenSource competitor VMWare recently patched around a dozen vulnerabilities affecting its hypervisor solution — they control the keys to the kingdom. That's why a number of security vendors, such as Catbird, are starting to offer hypervisor-specific security solutions.
“A breach in the host affects the integrity or the reliability on every guest in the environment,” MacDonald says. “If I'm a hacker, do I want to hack into each of these guest OSs when I could just break into the basement? The bottom level has the highest privilege. It owns those machines. It's an attractive target. That's why you have to take extra care.”
To get that message to stick at a decentralized campus environment, such as the University of Cincinnati, Hart and his colleagues have their work cut out for them. At colleges, any student or professor can bring up a web server onto the network and they are not particularly excited when the security staff tries to intervene.
“We're trying to give them a good checklist,” says Hart, who estimates server virtualization exists in roughly 10 to 15 percent of computing environments across campus. It's also important to patch the host systems, he adds.
In addition, the IT security department, led by Kevin McLaughlin, is planning user awareness seminars to address virtualization, says Hart, who deploys Superior, Colo.-based StillSecure's Cobia unified network platform on some VMs and hopes to extend the solution to any department running virtual systems.
“We've seen servers under people's desks,” Hart says. “If you put it online, you're responsible for it. There are too many computers in the university to make sure they're all secured.”
Maintaining visibility
The seminar McLaughlin is planning may include a discussion on guest-hopping, another risk in which a compromised guest takes over another guest running in a virtualized environment.
“It's kind of hard to know which VMs are going to be living next to each piece of hardware,” says Thomas Ptacek, a researcher with New York-based Matasano Security. “When VMs are migrating from place to place, you have no idea if they are going to be living next to guest VMs you need to worry about.”
Because VMs constantly are being shifted from host to host to optimize infrastructure efficiency, administrators must maintain visibility. That includes ensuring offline images are patched and that mission critical VMs containing sensitive information are provisioned and isolated from other, potentially insecure VMs.
Todd Holloway, information security risk management architect at Network Appliance, says VMs can be added to the network quickly without much oversight.
“If you don't have visibility, you have no clue what's going on,” says Holloway, responsible for helping to secure IT at the Sunnyvale, Calif.-based network storage solutions provider.
Meanwhile, a topic that has generated a deeply divided debate for more than a year involves the concept of virtualized malware. Polish researcher Joanna Rutkowska kicked off the discussion when she told Black Hat conference attendees last year that she had discovered an undetectable hypervisor rookit, known as “blue pill.”
According to Rutkowska, the rootkit assumes control of the operating system without it knowing, and the malware backdoors the underlying hardware.
“One of the biggest threats, we think, is a VM that can hide itself,” says Richter of Savvis. “That's the worst nightmare, a stealth VM that is launching attacks. That's why the administrator and the hypervisor need to know when a VM exists. There's the ability for the VM to be created and then cloak itself, all without the administrator knowing it exists, and then for it to become a rogue VM.”
This fear, say experts, likely will spur a widespread need for network access control technology in the virtual layer.
Still some, such as Ptacek — who at Black Hat this year delivered the talk “Don't Tell Joanna: The Virtualized Rootkit is Dead” — disagrees that malicious software can ever go undetected.
And a new collaborative white paper, authored by engineers from VMware and Xen Source and two graduate students from Stanford and Carnegie Mellon universities, argues a similar view. The paper contends that virtual and physical platforms will remain inherently different. Therefore, VMs will be unable to cloak themselves and launch attacks or disguise themselves like native hardware.
The report concludes that building a transparent VM is fundamentally not feasible, as well as impractical from a performance and engineering standpoint.
It is not all bad news for security when it comes to virtualization. For one, administrators can test security patches on VMs to ensure they will not break machines when applied or cause major downtime disruptions.
“The No. 1 thing that virtualization solves is it simplifies patch management,” Ptacek says.
The technology also offers upside for such malware analysts as Hart. Should an end-user machine be compromised, Hart and his team make a virtual copy of the PC and place it in a VMware
session. The approach has a two-fold benefit: it keeps end-users online without the IT department confiscating the affected physical machine and it allows the security group to analyze the attack in a test setting.
“It's kind of like a honeypot situation,” Hart says. “The person still thinks they're there. They'll still continue to make that attack thinking the system is still there. We'll make the switch immediately. It goes from one machine to the other.”
Of course, if malware is developed that can detect when it is running in a virtual environment, or if it can stay in stealth mode like Rutkowska suggests, this technique may lose its value.
Hart, meanwhile, wonders how effective a forensic analysis of VMs can ever be once attackers start actively exploiting those systems in the coming years. Virtual hackers, he says, will be able to better cover their tracks.
“You can carry a virtual machine on a USB drive,” he says. “There's going to be no evidence [of the attack] in that file.”
People, processes and policies
As with any potential security risk, experts advise that the most important step for organizations to take is to develop, implement and enforce policies — especially with a relatively nascent technology such as virtualization.
That means keeping track of online VMs and developing a segregation-of-duty model that spells out who is responsible for what.
“The IT personnel need to make sure that the VMs they create are always provisioned at the most current level,” says George Heron, vice president and chief scientist at Santa Clara, Calif.-based McAfee. “They have to have the patches, the AV signatures, the latest policies from the enterprise.”
That may also include regularly scanning VMs for vulnerabilities, implementing a network firewall for each VM, detecting unauthorized VM management sessions, and monitoring internal VLAN traffic within the virtual server, Gartner's MacDonald says in a March report.
Mike Liou, senior product marketing manager at Islandia, N.Y.-based CA, says policies also must address the potential for insider attacks on VMs. Thus, managing identity and access rights is critical to protecting company assets running in the virtual data center.
“If someone gains access to that host operating system, it's a leverage point that's going to give that person access to do harm,” he says, adding that regular audits can detect and discourage unauthorized insider activity.
MacDonald says in the report that the principle of least privilege should extend to the individual VMs.
“Two VMs should never directly communicate with each other,” he writes. “This reduces the risk of a compromised VM that could ‘sniff' the traffic and data of the other VMs.”
But despite the precautions that organizations must take and the ominous predictions for what the future may hold, the virtual field remains safe to play in, at least for now. While vulnerabilities affecting VMs are growing, hackers currently are focusing their attention elsewhere.
“We're in the renaissance of web insecurity right now,” Ptacek says. “This is the golden era of the web attacker. In this environment, there isn't that much incentive to go after the virtualization stuff.”
BEST PRACTICES:
Standards, acquisitions
Virtualization may be a new frontier for computing, but adopters may no longer feel they are deploying the technology in the Wild West.
The Center for Internet Security (CIS), in September, became the first standards body to unveil vendor-neutral guidelines for configuring and installing virtual machines (VMs). The 30-page document is available for download on the nonprofit's website.
Neil MacDonald, a Gartner analyst, says the historic benchmark will help prevent some of the security shortfalls that may affect virtualization. “It's your baseline for whether a given system is securely configured,” he says.
Meanwhile, virtualization software leader VMware has take more of a leadership role by promoting best practices, MacDonald says.
The company, a subsidiary of EMC, strengthened its security offerings when it acquired Determina, provider of host intrusion prevention products, in August. Determina's technology will be integrated into VMware's platform, including its signature ESX hypervisor.
— Dan Kaplan
SPREADING SECURITY:
Volunteer efforts
Karl Hart, information security officer at the University of Cincinnati, believes in the savings in cost and manpower that virtualization can lend to cash- and personnel-strapped organizations.
That is why the father of two children volunteers to extend the technology to a dozen kindergarten-through-12th-grade schools in the Cincinnati area. He re-purposes old equipment no longer being used, such as Pentium processors, and turns it into a virtualization platform.
The technology allows the schools to easily boot up images in VMware sessions, without spending money on products to back-up and restore machines, says Hart, whose children are one and four. And a simple restart of the VM session will erase anything, including malware.
“When you're in a resource-constrained business such as education, being able to create a network with fewer pieces of gear or less hardware is a huge gain,” says Mitchell Ashley, chief technology officer of StillSecure. Hart uses the company's Cobia integrated solution to protect the schools' networks.
Hart also is working on a side project to develop scripts that would build separate VMware images so parents can control what their children access on the family PC. “It's my way of helping to spread security and keep everyone safe,” he says. “I'm going to do this to protect the kids.”
— Dan Kaplan
The 35-year-old information security officer (ISO) at the University of Cincinnati need only review some of his own college's event logs to realize that academia's liberal networking environment often acts as an open invite to the criminal element cruising the internet.
“Universities and higher-ed are the first to get hit,” Hart says. “It's almost like a playground. It's like, ‘Let's try it here first.'”
So it is no wonder that Hart is spending a lot of time these days fretting over an easy-to-deploy emerging technology called server virtualization, in which multiple operating systems and applications run on a single machine at the same time. But Hart worries that professors and other collegiate end-users, averse to any centralized control, may overlook security in a rush to deploy this new architecture.
The benefits of the technology are game-changing: virtualization better utilizes system resources and increases performance, while reducing the time and money associated with space, power, cooling and management constraints. It is the future of computing, experts agree. According to a 2006 Yankee Group survey of 750 businesses, 62 percent of respondents already have or plan to deploy a virtualization solution. And further sweetening the pot in this market, vendor giants, such as Intel and Microsoft, now have gotten in on the game.
But even though there have been few, if any, in-the-wild attacks against virtualized machines, virtualization introduces an entirely new threat vector that, if not safeguarded properly, could attract even more security issues than the traditional desktop computing model.
The main concern about virtualization, say experts, is the danger an unprotected host operating system presents. If attackers can compromise the hypervisor — a thin layer of software that runs in the host and serves as the virtualization engine — they may earn free reign over every single guest, or virtual, machine (VM) operating on that host system.
“It's kind of like a single point-of-failure for multiple machines,” Hart says, referring to the hypervisor.
Neil MacDonald, a Gartner vice president, says that if IT departments do not properly plan for virtualization by implementing security best practices, any business gain may be negated.
“Virtualization, by definition, is a layer of abstraction,” he says. “It is a software layer that gives you this abstraction. It's a new layer. People overlook how important that layer is and that it must be secured and properly configured like any other layer in the stack.”
As organizations begin implementing virtualized solutions en masse, hackers will take notice and flaws will be discovered. Vulnerabilities in products from VMware — the Palo Alto, Calif. virtualization software leader — have jumped from just one in 2002 to 34 already this year, according to data recently compiled by Kris Lamb, director of the X-Force research team at IBM Internet Security Systems. A representative from VMware could not be reached for this story.
With the interest in virtualization growing, many experts believe it's only a matter of time before hackers start taking advantage of associated unprotected vulnerabilities that the technology presents.
“I believe that virtual machines are going to be the next great unexplored frontier for black hats,” says Chris Richter, vice president and general manager of managed security services at Savvis, a St. Louis area-based global IT services provider. “There's going to be a rush to develop new exploits for this platform.”
So just what are the dangers? Well, they're not much different than threats facing the traditional server environment, experts say.
“What is true is that virtualization brings the ability to compartmentalize and segment more effectively than a physical server implementation,” Lamb says.
But what business executives have failed to think about is that just because something is running virtually rather than physically on a server doesn't mean that the same problems of compliance, risk and OS hardening don't exist, he adds.
Key to success
The key, MacDonald says, is to make the virtual layer as thin as possible, thereby limiting the complexity and, in turn, the number of potential vulnerabilities. Simon Crosby, chief technology officer at open-source virtualization software maker XenSource (acquired by Citrix for $500 million), says hypervisors only contain about 60,000 lines of code compared to millions on a desktop OS.
But if an attacker can find his way in — XenSource competitor VMWare recently patched around a dozen vulnerabilities affecting its hypervisor solution — they control the keys to the kingdom. That's why a number of security vendors, such as Catbird, are starting to offer hypervisor-specific security solutions.
“A breach in the host affects the integrity or the reliability on every guest in the environment,” MacDonald says. “If I'm a hacker, do I want to hack into each of these guest OSs when I could just break into the basement? The bottom level has the highest privilege. It owns those machines. It's an attractive target. That's why you have to take extra care.”
To get that message to stick at a decentralized campus environment, such as the University of Cincinnati, Hart and his colleagues have their work cut out for them. At colleges, any student or professor can bring up a web server onto the network and they are not particularly excited when the security staff tries to intervene.
“We're trying to give them a good checklist,” says Hart, who estimates server virtualization exists in roughly 10 to 15 percent of computing environments across campus. It's also important to patch the host systems, he adds.
In addition, the IT security department, led by Kevin McLaughlin, is planning user awareness seminars to address virtualization, says Hart, who deploys Superior, Colo.-based StillSecure's Cobia unified network platform on some VMs and hopes to extend the solution to any department running virtual systems.
“We've seen servers under people's desks,” Hart says. “If you put it online, you're responsible for it. There are too many computers in the university to make sure they're all secured.”
Maintaining visibility
The seminar McLaughlin is planning may include a discussion on guest-hopping, another risk in which a compromised guest takes over another guest running in a virtualized environment.
“It's kind of hard to know which VMs are going to be living next to each piece of hardware,” says Thomas Ptacek, a researcher with New York-based Matasano Security. “When VMs are migrating from place to place, you have no idea if they are going to be living next to guest VMs you need to worry about.”
Because VMs constantly are being shifted from host to host to optimize infrastructure efficiency, administrators must maintain visibility. That includes ensuring offline images are patched and that mission critical VMs containing sensitive information are provisioned and isolated from other, potentially insecure VMs.
Todd Holloway, information security risk management architect at Network Appliance, says VMs can be added to the network quickly without much oversight.
“If you don't have visibility, you have no clue what's going on,” says Holloway, responsible for helping to secure IT at the Sunnyvale, Calif.-based network storage solutions provider.
Meanwhile, a topic that has generated a deeply divided debate for more than a year involves the concept of virtualized malware. Polish researcher Joanna Rutkowska kicked off the discussion when she told Black Hat conference attendees last year that she had discovered an undetectable hypervisor rookit, known as “blue pill.”
According to Rutkowska, the rootkit assumes control of the operating system without it knowing, and the malware backdoors the underlying hardware.
“One of the biggest threats, we think, is a VM that can hide itself,” says Richter of Savvis. “That's the worst nightmare, a stealth VM that is launching attacks. That's why the administrator and the hypervisor need to know when a VM exists. There's the ability for the VM to be created and then cloak itself, all without the administrator knowing it exists, and then for it to become a rogue VM.”
This fear, say experts, likely will spur a widespread need for network access control technology in the virtual layer.
Still some, such as Ptacek — who at Black Hat this year delivered the talk “Don't Tell Joanna: The Virtualized Rootkit is Dead” — disagrees that malicious software can ever go undetected.
And a new collaborative white paper, authored by engineers from VMware and Xen Source and two graduate students from Stanford and Carnegie Mellon universities, argues a similar view. The paper contends that virtual and physical platforms will remain inherently different. Therefore, VMs will be unable to cloak themselves and launch attacks or disguise themselves like native hardware.
The report concludes that building a transparent VM is fundamentally not feasible, as well as impractical from a performance and engineering standpoint.
It is not all bad news for security when it comes to virtualization. For one, administrators can test security patches on VMs to ensure they will not break machines when applied or cause major downtime disruptions.
“The No. 1 thing that virtualization solves is it simplifies patch management,” Ptacek says.
The technology also offers upside for such malware analysts as Hart. Should an end-user machine be compromised, Hart and his team make a virtual copy of the PC and place it in a VMware
session. The approach has a two-fold benefit: it keeps end-users online without the IT department confiscating the affected physical machine and it allows the security group to analyze the attack in a test setting.
“It's kind of like a honeypot situation,” Hart says. “The person still thinks they're there. They'll still continue to make that attack thinking the system is still there. We'll make the switch immediately. It goes from one machine to the other.”
Of course, if malware is developed that can detect when it is running in a virtual environment, or if it can stay in stealth mode like Rutkowska suggests, this technique may lose its value.
Hart, meanwhile, wonders how effective a forensic analysis of VMs can ever be once attackers start actively exploiting those systems in the coming years. Virtual hackers, he says, will be able to better cover their tracks.
“You can carry a virtual machine on a USB drive,” he says. “There's going to be no evidence [of the attack] in that file.”
People, processes and policies
As with any potential security risk, experts advise that the most important step for organizations to take is to develop, implement and enforce policies — especially with a relatively nascent technology such as virtualization.
That means keeping track of online VMs and developing a segregation-of-duty model that spells out who is responsible for what.
“The IT personnel need to make sure that the VMs they create are always provisioned at the most current level,” says George Heron, vice president and chief scientist at Santa Clara, Calif.-based McAfee. “They have to have the patches, the AV signatures, the latest policies from the enterprise.”
That may also include regularly scanning VMs for vulnerabilities, implementing a network firewall for each VM, detecting unauthorized VM management sessions, and monitoring internal VLAN traffic within the virtual server, Gartner's MacDonald says in a March report.
Mike Liou, senior product marketing manager at Islandia, N.Y.-based CA, says policies also must address the potential for insider attacks on VMs. Thus, managing identity and access rights is critical to protecting company assets running in the virtual data center.
“If someone gains access to that host operating system, it's a leverage point that's going to give that person access to do harm,” he says, adding that regular audits can detect and discourage unauthorized insider activity.
MacDonald says in the report that the principle of least privilege should extend to the individual VMs.
“Two VMs should never directly communicate with each other,” he writes. “This reduces the risk of a compromised VM that could ‘sniff' the traffic and data of the other VMs.”
But despite the precautions that organizations must take and the ominous predictions for what the future may hold, the virtual field remains safe to play in, at least for now. While vulnerabilities affecting VMs are growing, hackers currently are focusing their attention elsewhere.
“We're in the renaissance of web insecurity right now,” Ptacek says. “This is the golden era of the web attacker. In this environment, there isn't that much incentive to go after the virtualization stuff.”
BEST PRACTICES:
Standards, acquisitions
Virtualization may be a new frontier for computing, but adopters may no longer feel they are deploying the technology in the Wild West.
The Center for Internet Security (CIS), in September, became the first standards body to unveil vendor-neutral guidelines for configuring and installing virtual machines (VMs). The 30-page document is available for download on the nonprofit's website.
Neil MacDonald, a Gartner analyst, says the historic benchmark will help prevent some of the security shortfalls that may affect virtualization. “It's your baseline for whether a given system is securely configured,” he says.
Meanwhile, virtualization software leader VMware has take more of a leadership role by promoting best practices, MacDonald says.
The company, a subsidiary of EMC, strengthened its security offerings when it acquired Determina, provider of host intrusion prevention products, in August. Determina's technology will be integrated into VMware's platform, including its signature ESX hypervisor.
— Dan Kaplan
SPREADING SECURITY:
Volunteer efforts
Karl Hart, information security officer at the University of Cincinnati, believes in the savings in cost and manpower that virtualization can lend to cash- and personnel-strapped organizations.
That is why the father of two children volunteers to extend the technology to a dozen kindergarten-through-12th-grade schools in the Cincinnati area. He re-purposes old equipment no longer being used, such as Pentium processors, and turns it into a virtualization platform.
The technology allows the schools to easily boot up images in VMware sessions, without spending money on products to back-up and restore machines, says Hart, whose children are one and four. And a simple restart of the VM session will erase anything, including malware.
“When you're in a resource-constrained business such as education, being able to create a network with fewer pieces of gear or less hardware is a huge gain,” says Mitchell Ashley, chief technology officer of StillSecure. Hart uses the company's Cobia integrated solution to protect the schools' networks.
Hart also is working on a side project to develop scripts that would build separate VMware images so parents can control what their children access on the family PC. “It's my way of helping to spread security and keep everyone safe,” he says. “I'm going to do this to protect the kids.”
— Dan Kaplan
