A few days after the fall semester ended in December at Connecticut College, a student was summoned to the help desk at the small liberal arts school. A network security appliance had determined his computer was possibly infected with malware.
The IT staff attempted to diagnose the problem and apply what it believed to be the appropriate treatment: Run a scan and install the latest anti-virus signatures and Windows security updates. Only this time, the traditional remedy failed to work.
“Less than five minutes later, I got the alert that there was an [information-stealing] botnet running on this IP address, which was the machine they thought they had cleaned,” recalls John Schaeffer, the systems and network administrator at Connecticut College in New London. “And they're hard to get rid of once you have them.”
The student's compromised PC was emblematic of the growing vulnerability of today's organizations. Often comprised of tens of thousands of compromised machines, and designed to evade detection, the modern day botnet is the most sophisticated, stealthy and profitable component in a cybercriminal's arsenal. At its peak, for example, the peer-to-peer Storm Worm botnet was estimated by computer scientists as being more powerful than any of the world's supercomputers. It was responsible for distributing 20 percent of the world's spam.
Botnets leverage unique binaries, severely limiting the effectiveness of anti-virus signatures and intrusion prevention systems. And many are now built with protocols that allow their drone machines to blend in with legitimate outbound web traffic while they communicate with their command-and-control servers to receive and act on instructions.
Compromising a machine – like the one belonging to the Connecticut College student – provides the coordinated computing power necessary to send junk mail, siphon sensitive data and launch destructive denial-of-service attacks that could jeopardize national security. Remember Estonia?
But the secret to botnet success begins several rungs up the ladder, with the internet service providers (ISPs), web server providers and domain registrars who supply the rack space, bandwidth and URLs necessary to host these attacks.
That is why at least one industry organization – the San Francisco-based Messaging Anti-Abuse Working Group (MAAWG) – has made botnet remediation its number one priority in 2009.
Network operators to take action
MAAWG, represented by scores of ISPs which collectively handle one billion of the world's email accounts, are scheduled to meet this month to develop best practices and examine new approaches for dealing with botnets, says Michael O'Reirdan, the group's chairman. He also serves as a distinguished engineer at a leading ISP.
For example, one subcommittee is investigating how Internet Protocol version 6 (IPv6), the next-generation internet layer that theoretically can provide trillions more unique IP addresses, will impact botnet detection.
“[IPv6] is a change to the way we will be doing things, and we want to make sure that we are looking at any new ways the spammers and others can use it to make delivery of spam more effective so that we are prepared for possible new attacks,” O'Reirdan says.
Another committee is reviewing a possible method by which senders of solicited bulk emails can detect whether recipients are infected with bot malware, according to MAAWG. Then, these senders could automatically notify a subscriber's ISP, which could isolate the offending machines.
“How do we get ISPs to work in a way where they are able to deploy better practices of mitigation? If you get a user who is detected to be compromised, what are the things that you are going to do to that user?” O'Reirdan asks.
He says the problem of botnets has become “phenomenally serious” during the past two years, and it is crucial that the ISPs step up.
“What you're seeing is cleverly written modular code,” he says. “People write this software to make money. It's intended to run quietly on people's machines.”
When ISPs lack the initiative, a startling scenario could unfold. Take the case of McColo, a now-defunct Silicon Valley-based company that was responsible for hosting the notorious Srizbi botnet in November.
Only enterprise reporting by the Washington Post's Brian Krebs prompted McColo's upstream providers, Hurricane Electric and Global Crossing, to stop routing its internet traffic. In an instant, spam dropped by as much as 75 percent, before returning to normal levels around the holidays. (The Srizbi creators had planned for such a scenario and included an algorithm in their code that generates unique domain names, allowing the botnet to return online.)
“[McColo] was an incredible case,” says Alex Lanstein, a senior security researcher at FireEye, a Milpitas, Calif.-based anti-botnet company. “It's mind-boggling how they got away with that for so long.”
A similar, but less precipitous drop in spam occurred a few months earlier when rogue ISP Atrivo was shut down.
But Lanstein says ISPs and server providers typically have little motivation to halt botnets. After all, the criminals are sources of revenue, too.
“None of the data centers [which provide the web servers] do active monitoring on the traffic,” says Lanstein. “All these guys are conduit connections.”
And even if the more trusted providers catch on to malicious activity, the bot herders will seek out a vendor overseas who is more apt to look the other way, Lanstein says, adding that U.S. data centers are the preferred choice, though, because they offer the best price, bandwidth and overall quality of service.
As for the ISPs, Lanstein says they do even less. “All they're doing is selling transit. I bet they don't even look at statistics. I guess the burden is almost on the security researcher to do active monitoring on this stuff.”
O'Reirdan admits that ISPs, such as McColo, are out there, but he was hesitant to discuss MAAWG's position on rogue carriers. “That's what laws are for,” is all he would say.
Legitimate providers, meanwhile, must straddle a fine line when it comes to monitoring their pipes for infection, he says. These companies could quickly find themselves immersed in a gray zone when it comes to privacy, regulations over which vary greatly from country to country. But, he adds, ISPs have a serious stake in the control of botnets because they want to avoid the cost of bandwidth consumption and customer complaints.
Network security provider Arbor Networks, in its annual Worldwide Infrastructure Security Report, published in November, said botnets are the single greatest threat facing network operators because many resources are being spent to deflect prevalent and sophisticated botnet-driven DDoS attacks. The report also mentioned growing concern around IPv6 threats.
“Machines are being used to do other things and are being slowed down,” O'Reirdan says. “People are pinging away, sending packets, attacking websites. There will be bandwidth issues. It really is the customer support and customer perception that the ISP is not giving them a good service.”
But security researchers say the personnel at the ISPs may not have the time or financial motivation to care, either. Notifying customers that their machines are infected – or fielding a help-desk phone call from a confused or angry subscriber whose IP address is blocked because it is being used to spew spam – costs money as well. And carriers typically are not in the business of working with law enforcement. However, an ongoing FBI dragnet, known as Operation Bot Roast, has led to the indictment of at least eight people, and turned up one million compromised PCs – reason enough to believe that authorities are making at least some headway.
André DiMino, co-founder and director of the Shadowserver Foundation, a volunteer internet watchdog that compiles data on botnets, says he understands the potential privacy implications that network operators could encounter, but says there are ways around it.
“There's certainly a way to deal with the flow of data and not the content,” he says. “It's just a matter of taking this type of information and being open with it, and finding a way to get past the privacy issues and deal with it on a more technical level. You don't need content to deal with the problem.”
Maturity of botnets
Botnets have been around for awhile, but only in the past few years have they attained such a negative connotation.
Toolkits used to create networks of compromised systems typically fetch hundreds of dollars in the criminal underground, compared to more inexpensive kits used to create phishing or SQL injection attacks, says Zulfikar Ramzan, technical director of Symantec Security Response.
Many of today's botnets are being assembled through browser exploits used in conjunction with drive-by-downloads, he explains. Malware authors are employing “server-side polymorphism,” which creates a new instance of the botnet threat every time it is downloaded – making it difficult to detect malicious code.
“It contains a payload,” Ramzan says. “If you get it to someone's machine, it will be able to communicate back with a command-and-control center to receive commands. The botnet has become the virtual high-priced real estate in the world of cybercrime.”
About 11 percent of machines worldwide – some 65 million to 90 million PCs – are compromised, says Paul Royal, director of research at Atlanta-based botnet security firm Damballa. An estimated five percent of corporate computers are seeded with bots.
Traditional security solutions struggle mightily to detect the presence of a zombie computer because the most prolific botmasters now use HTTP – instead of IRC (internet relay chat) – as their command-and-control protocol, he says. Businesses almost never block this traffic because, for the most part, it is legitimate web communication.
“Unless you're really looking for it, there's so much HTTP traffic that a corporate network has to deal with,” Shadowserver's DiMino says. “You have to know what you're looking for to detect it.”
Once loaded onto a corporate machine, bot malware will get immediately to work, probing the network for other vulnerable IP addresses and attempting to propagate through means such as tricking another user to click on a malicious instant messenger link. Often, they are instructed to scan the web for legitimate websites vulnerable to SQL injection attacks. Armed with that information, criminals can plant malware on those sites in hopes of infecting previously innocuous machines and thus expand their botnets.
It's a vicious cycle, really. And because of their sheer size, botnets are responsible for as much as 95 percent of the world's spam. The bigger the botnet, the more unsolicited email it can deliver.
Identity theft: a growing concern
But of increasing concern to organizations is the continued influx of targeted malware onto users' machines designed to perpetrate identity theft and steal valuable company data. Experts suggest botnets will start getting smaller in scope, allowing for them to unleash more specific attacks on companies without running the risk of being flagged by industry and law enforcement.
“It used to be that people were worried about viruses,” says Doug Camplejohn, CEO of Mi5 Networks, a Sunnyvale, Calif.-based provider of web security gateway solutions. “Now, the virus rates of what we've been seeing in the wild are steadily declining each quarter. Things that are opening backdoors are exploding. The best thing to do is put a piece of code on a system that can continue to extract value off that system. Botnets are one of the most profitable ways to do that.”
Mike Miller, director of security at Richmond, Va.-based Media General, which owns 25 daily newspapers and 23 network-affiliated television stations, says that if his department detects a backdoor on a machine, it will be rebuilt.
“[Botnets are] not like the ‘I Love You' virus,” he says. “If there's one service running, it's now not just that one service. It's everything it downloads. Even if you clean up one, it's working on downloading another.”
The key, Camplejohn says, is applying heuristics that allow for the monitoring of rogue outbound traffic attempting to reach the botnet host.
“The whole goal is to stay below the radar,” he says. “Desktop software is not looking at any outbound traffic in ports and protocols. Once they are in, they find it fairly easy to stay below the radar.”
Peter Firstbrook, a Gartner research director, says that if botmasters ever start caring about quality instead of quantity, the botnet issue could impact businesses in far worse ways than it does now.
For example, if a malware writer discovered that his bot was made up of nodes belonging to a bank or a pharmaceutical firm, he could start pilfering data and intellectual property or, perhaps worse, choose to sell the botnet to a competitor. For now, though, most cybercrooks are merely interested in infecting as many machines as possible.
“Most of this stuff is randomly distributed,” he says. “It's only by accident that they might discover it. [But] starting to sort out who they own will be the next trend.”
Businesses might also want to pay attention to something else on the horizon: the possibility that they someday may be held accountable for their compromised machines, if they are used to spearhead attacks, such as DDoS assaults. However, experts say they are not aware of any cases in which enterprises have been held liable for housing zombie machines.
Are ISPs the answer?
Observers say that many factors play a role in the defense against botnets, including educational, technical, legal and cultural. ISPs can have a hand in all four, says O'Reirdan.
Author and internet security expert Ira Winkler, an outspoken critic of ISPs, has suggested that users be blocked or fined if their machines are found to be infected.
MAAWG has not stood silent on the issue. In 2007, the group issued the industry's first-ever best-practices paper covering the handling of infected ISP subscribers. It recommended that compromised users should be directed to a safe quarantine, known as a “walled garden,” where they could receive the tools necessary to fix the problem without impacting privacy.
Some ISPs, such as Cox Communications, have implemented the concept. Others offer free anti-virus to customers. On the outbound side, Comcast decided in 2004 to block Port 25, the main email (SMTP) gateway on certain accounts suspected of sending spam. MAAWG has since published a separate best practices report on that method.
Firstbrook says these types of initiatives are going to happen more often, likely beginning this year. Eventually, security services will become a competitive advantage for ISPs, which will implement things such as monitoring systems to detect and block bot traffic, he says.
“The problem is the cost benefit to them right now is skewed to the cost side,” he says. “However, as the situation deteriorates, we will see them step up as a differentiator. It will be a domino effect once it gets going.”
And businesses have good reason to pay close attention. While the largest corporations have more control over their networks than ISP-reliant smaller organizations and home users, everyone has a vested interest in anti-botnet crusades, such as the one MAAWG is undertaking.
“By suppressing the number of bots on a network, you suppress the ability of bots to do things,” O'Reirdan says. “That's going to have an inevitable knock-on effect [for all].”
And end-users, too, can rest easy, knowing their zombie machines weren't used as weaponry in the latest cyberwar.