Michael Assante, CSO, North American Electric Reliability Corp. (NERC). Photo by Bill Bernstein.
Michael Assante, CSO, North American Electric Reliability Corp. (NERC). Photo by Bill Bernstein.

 

Last fall, when a Department of Homeland Security-commissioned video depicted a turbine exploding in a test lab during a simulated hacker attack, viewers may have been expecting the credits to roll at the end.

But the film, produced by the Idaho National Laboratory and leaked to the Associated Press, was not dreamt up for a Hollywood soundstage. In fact, the footage was quite genuine, showing what could happen if a hacker gets control of a power company's network by exploiting a real-life vulnerability.

Then in January, a senior CIA cybersecurity analyst, speaking at a SANS Institute event, told attendees that organized criminals, as part of an extortion ploy, were able to black out several cities outside the United States.

The analyst, Tom Donahue, would not reveal specifics. However, Alan Paller, SANS Research Director, said the problem was getting so rampant that even an agency as secretive as the CIA figured it had to go public in hopes of provoking some reform in an industry that has been far too complacent for far too long.

Combined, these events delineate in no uncertain terms that the nation's critical infrastructure is at, well, a critical juncture – and the ravenous hacker community could be poised to pounce.

The nation's power grid, oil and gas pipelines, wastewater treatment plants and chemical refineries – long thought of as only at risk to physical sabotage – are becoming increasingly susceptible to the same types of targeted IT threats that have been plaguing corporations for years.

Except, when it comes to these properties, the stakes are much higher.

“It's infrastructure,” says IT security analyst Rich Mogull. “You're not screwing up some corporate IT department. You let someone take over the wrong part of a control system, and you bring down power.”

The rise of NERC

Spurred on by the turbine video – and more broadly, the Sept. 11, 2001 terrorist attacks and the 2003 Northeast blackout – lawmakers are paying more attention to protecting these assets, says Michael Assante, the former infrastructure protection strategist at Idaho National Lab and the recently appointed CSO of the North American Electric Reliability Corp. (NERC).

Assante's appointment on Sept. 2 as CSO of NERC – a nonprofit charged by the federal government with overseeing the bulk power system in North America – is the first time the position has ever been filled, a telling sign that a tipping point is approaching, if not already here.

“There's a lot of awareness now that the critical infrastructure relies on control system technology and that's driven the cyber-research community to look at it,” Assante says. “The issue around cyberthreats as a nation is clearly becoming a top-level concern for national security and homeland security decision-makers.”

Case in point: On Jan. 17, the Federal Energy Regulatory Commission (FERC) approved eight cybersecurity standards that extend to all entities connected to the power grid. NERC will be tasked with enforcing them – violators can face fines up to $1 million.

The guidance covers asset identification, management controls, personnel and training, perimeters, physical security, systems management, incident response and reporting and disaster recover. The first compliance deadline is set for Dec. 31.

Going forward, NERC will adjust the standards to meet the evolving threat landscape, Assante says. (Chemical plants are bound to similar regulations under the Chemical Facility Anti-Terrorism Standards, established by the Department of Homeland Security, and which took effect in June 2007.)

But Assante's main responsibility at NERC will be using his experience, which includes a stint as CSO at American Electric Power, to build cooperation and education among all parties connected to the electric grid – in addition to overseeing periodic risk assessments to determine the level of preparedness by key players to respond to cyberthreats, such as external hacks.

“My job is to act as a single coordination point with the U.S. government and Canadian and Mexican governments,” he says.

That includes overseeing the formation of the Electric Sector Steering Group, which will include five chief executives representing different power industry stakeholders. The goal of the committee will be to guide NERC as it embarks on its recently announced Critical Infrastructure Protection program.

“I've been in the cybersecurity game, and it's very difficult to get senior executive leadership involved,” he says. “We want to bring people together in a collaborative environment and share information and share each other's expertise to be able to understand the problem and develop security solutions. They can turn around and speak for the industry and emphasize the importance of these issues.”

The inherent risk

When they were developed, Supervisory Control and Data Acquisition, or SCADA, networks – the often legacy systems that run industrial control plants – were physically isolated from private and public networks and built on proprietary hardware, operating systems and communications protocol, says Lawrence Johnson, IT security analyst at Minnesota Power, provider of electricity for 135,000 residents in northeast Minnesota.

This kept them safe from most harm. But times changed. Control system software is more understood than ever before because it is being modeled on off-the-shelf technology, such as Windows boxes.

“Vulnerabilities that used to show up in business applications on Windows or Linux or Unix, are showing up in SCADA systems because they're using the same operating systems, the same platforms,” says Jim White, vice president of infrastructure security at Irvine, Calif.-based Uniloc.

Most of all, though, the risk can be attributed to openness. Control systems largely are now connected to the public internet via corporate data networks, he says.

This means enhanced business good in areas such as information sharing, resiliency and total cost, according to experts. Power companies, for example, can take real-time operational data from their control networks and use it to make more intelligent decisions.

But it also means a potential pathway to destruction at the hands of a malicious outsider or opportunistic insider. To the latter, in 2001, an Australian man was imprisoned for using first-hand knowledge to cause a waste management system to dump millions of liters of raw sewage into local parks, creeks and a Hyatt hotel. He used a wireless connection to access the SCADA system.

According to a November survey from Secure Computing, recently acquired by McAfee, all of 132 respondents to its critical infrastructure protection survey believe connecting SCADA systems to corporate networks is a risk. But it is a risk that is at the point of no return.

“They always relied on security by obscurity because literally there was no public connection or interface to those systems,” says Paul Henry, who co-authored Techno Security's Guide to Securing SCADA. “But the hackers have learned they can find their way through the corporate network to those process control systems.”

Researchers getting involved

Sept. 5, 2008 may turn out to be a defining moment for the state of critical infrastructure security.

That was when popular exploit database, milw0rm, posted public exploit code that detailed how to take advantage of a previously patched vulnerability in industrial process software, from a Georgia-based SCADA solutions provider called Citect.

These types of exploits are rare and usually relegated to niche mailing lists, so it certainly is a sign of the times when a mainstream database, such as milw0rm, decided to pick it up, says Kevin Finisterre, the head of penetration testing at Foxborough, Mass.-based Netragard and author of the exploit.

“What I've done is kind of level the playing field,” he says. “I put them more on the same level as traditional IT vulnerabilities. Nobody would blink twice if someone published a fully detailed advisory on how to exploit Internet Explorer.”

Finisterre says he decided to drop the exploit to help raise exposure into the inherent weaknesses of these systems. But his decision resulted in some backlash – he and his boss fielded a number of emails questioning why Finisterre would want to jeopardize the nation's critical infrastructure.

On the flip side, Finisterre concedes that since he put out the advisory, a number of interested hackers have contacted him, realizing that “there's definitely a lot of low hanging fruit in this industry, and let's see if we can pick at some.”

Alan PallerPaller of the SANS Institute (right) says organized criminals are facing stiff competition from each other in the attack arena and may soon look toward SCADA systems as a viable alternative.

“There are so many people that there's massive competition for technique,” he says. “They've been looking for new targets for about 24 months.”

The SCADA threat though, at least from a terrorist perspective, is far from imminent, Paller admits. “It's too easy to do kinetic weapons,” he says.

But Paller warns of a jaw-dropping future scenario. It is possible that many of the world's SCADA networks have already been seeded with trojans that masquerade as good code. It will remain stealthy until one day – bam – it is called into action. This persistent threat, as Paller refers to this type of attack, is puzzling and quite dangerous.

“When you sit down with some of the smartest people and say, ‘How do we solve this,' they just stare at you,” he says.

Who is to blame?

So what has made many SCADA systems seemingly so unprepared to deal with this emerging risk?

There are a number of places to point fingers, but the overarching reason is history.

For as long as they have been around, SCADA operators and engineers have had little reason to worry about the possibility of an outside cyberattack. And why not feel this way? After all, process control plants are not required to report incidents, unlike corporations that suffer a data breach.

“IT people have been under siege for 20 years,” says Elan Winkler, director of solutions at San Jose, Calif.-based Secure Computing. “They get it. There's a real dividing line between corporate IT individuals and the individuals responsible for control networks.”

Combine that with SCADA personnel's general unwillingness to apply patches or run network scans – for fear of breaking systems that are required to always be up and running – and the state of security can look bleak to an observer.

“None of that matters to an attacker,” Finisterre says. “They don't care if you have an operational issue with the ability to patch your software. They don't care if your system is stressed out in the summer when there's peak water usage. That benefits them.”

In May, the Government Accountability Office, the investigative arm of Congress, issued a scathing report that questioned the adequacy of cybersecurity at the Tennessee Valley Authority (TVA), the nation's largest public power company.

Among the findings, the report said, “weaknesses in the separation of network segments” could allow malicious individuals to use the corporate network to tap into the control system network.”

The report blamed these weaknesses on the company's failure to implement an information security program that included the proper assessment of its control systems and the implementation of a mechanism to prioritize vulnerabilities. Meanwhile, the business network lacked sufficient anti-virus and firewall protection, the report concluded.

TVA responded, according to public reports, by saying the findings were hypothetical in nature, but vowed to correct its faults.

Paller says poor senior leadership and budgetary dollars are also often to blame. Based on his conversations with SCADA personnel, neither seems to be a priority at industrial control plants.

“The industry is reacting in an almost schizophrenic manner,” Paller says. “They'll say that this is a high priority and we're going to invest a lot, and the next day they'll say, ‘How can we spend less?'”

A real-life example: Johnson of Minnesota Power told attendees in September during the MIS Training Institute's IT Security World conference in San Francisco, that he has been lobbying his superiors for many months to hire an additional security body. So far, no luck.

Still, even with awareness growing among SCADA staff, a culture clash remains.

Many workers on the SCADA side are deeply possessive of their equipment and therefore are reluctant to work with IT personnel to solve security shortfalls.

“They're very sensitive about IT-related people and how they ‘don't understand how the control systems operate,'” Finisterre says. But that is no excuse for overlooking security.

“Folks like me need to be working with them,” he says. “They should have people like me on staff. The concepts of an attacker should not be foreign to them.”

Fixing the problem

The easiest solution to the SCADA security dilemma would be to entirely disconnect control networks from data networks. Of course, in today's environment, that is not a realistic option.

So, experts say these facilities should do the next best thing: Model the process control side out of the house after the business network.

Start by conducting a risk assessment and build a security framework, much like their counterparts in IT. Johnson suggests basing this model on ISO 27002 or COBIT.

When running patches, plants should ensure their vendor already has tested the fix, he adds. Meanwhile, scans should be run on backup networks to avoid a network collapse.

As for the network configuration, Johnson recommends creating a separate SCADA local area network (LAN) by using a properly configured firewall and locking down access to only a privileged bunch.

“Corporate users should never touch the SCADA LAN,” he says.

Mogull, in a blog post last year, opined using “virtual air gaps” to communicate between the two networks. An isolated server sits between two firewalls with only a single port for exchanging information.

“The odds of traversing to the process control network are pretty darn slim,” he says.

One company, Industrial Defender, provides intrusion detection systems for SCADA systems. The solution looks for anomalous traffic on industrial protocols and then takes action to mitigate the problem.

“The fundamental issue is that there's really no visibility about what's going on from a security perspective,” Brian Ahern, CEO of the Foxborough, Mass.-based company, says. “These systems were designed to operate plants, not detect security threats.”

Vendors also have a major role to play. In the same vein as Microsoft's push to build security into its offerings, a number of SCADA software and hardware providers are ramping up their own efforts.

Sensing that the research community was starting to lick its chops over the possibilities of exploiting control systems, vendors such as Citect spurred into action during the last couple of months. It recently started lunchtime training sessions for its coders.

“It's no longer possible that we can assume security-by-obscurity,” says CTO Paul Francis. “We made sure that our teams are educated in this and understand the importance of this.”

And remember the Idaho National Laboratory, the Department of Energy facility that produced the turbine video for DHS? Well, engineers there began a procurement program that both tests SCADA products and then advises end-users on how to build purchasing contracts to include security demands.

“We wanted the customers to ask for the right things and have the providers be able to enhance the security they provide in those products,” says Assante of his former employer.

Back at NERC, Assante will try to assemble a similar culture of assistance and education for the power industry.

“We're passing out of this phase of understanding what the problem is to the next phase where organizations are developing capabilities to enhance the security of the technologies that they're either developing or in the environments where they're operating,” he says. “We hope we serve as an effective model and other control industries can learn from it.”

 

[Sidebar]

SCADA: Persistent threat

Alan Paller, SANS research director, warns of a jaw-dropping future scenario.

It is possible, he says, that many of the world's SCADA networks have already been seeded with trojans that masquerade as good code. They will remain stealthy until one day – bam – they are called into action. This "persistent threat," as Paller refers to the type of attack, is puzzling and quite dangerous.

"When you sit down with some of the smartest people and say, ‘How do we solve this,' they just stare at you," he says. – Dan Kaplan