Incident Response, TDR

Bots account for more than half of all 2014 web traffic, report shows

The majority of traffic on the internet this year was from bots, according to Incapsula's Bot Traffic Report 2014.

This year saw 56 percent of all website traffic coming from bots, with 29 percent of those bots being considered ‘bad,' and 27 percent being ‘good,' the research shows. Last year, bot visits accounted for 61.5 percent of all website traffic, according to Incapsula's Bot Traffic Report 2013.

“The bulk of the decrease in bot traffic is contributed to a decline in the good bot activity, mostly in the activity of the bots employed by RSS services,” Igal Zeifman, product evangelist and security researcher with Incapsula, told SCMagazine.com in a Wednesday email correspondence.

Good bots perform functions that may be useful to users and website operators, such as measuring site speed and indexing content, Zeifman said. One example, he explained, is Googlebot, which crawls websites to be indexed in Google Search. 

Conversely, bad bots are malware tools used by hackers and spammers, Zeifman said, adding bad bots are becoming increasingly sophisticated by mimicking human user behavior better and, therefore, becoming much harder to spot.

In the report, bad bots are broken down into four types: Hacking Tools, Scrapers, Spammers, and Impersonators. Of note, Impersonator bots have shown consistent growth over the past few years, increasing to 22 percent of bad bots in 2014 from 20.5 percent in 2013.

“Impersonator bots are browser-like bots that can really belong to any of the above categories,” Zeifman said. “The only difference is that these are more advanced malicious tools that were modified to create a browser-like HTTP fingerprint, to circumvent security measures.”

He continued, “For example, this could be a hacker bot with extra features that allow it to bypass security challenges that would stop a lesser/generic version. These are also DDoS bots used in Application Layer DDoS attacks.”

The smaller the website, the greater percentage of bot traffic, the report shows.

Bots account for 80.5 percent of traffic on small websites bringing in 10 to 1,000 visits per day, 63.2 percent of traffic on medium sites bringing in 1,000 to 10,000 visits per day, 56.2 percent of traffic on larger sites bringing in 10,000 to 100,000 visits per day, and 52.3 percent of Alexa MVP sites bringing in between 100,000 and more than a million visits per day.

“Most bots don't care if your site is popular or not and will crawl, scan and hack it regardless of its popularity,” Zeifman said. “As a result, in relative terms, the percentage of bot visits is much higher on smaller and less popular sites [that] get much less human visits but are still frequented by hype-immune bots.”

Get daily email updates

SC Media's daily must-read of the most current and pressing daily news

By clicking the Subscribe button below, you agree to SC Media Terms and Conditions and Privacy Policy.