New research from Distil Networks has shown that in 2015, overall bot traffic, as compared to human traffic, decreased slightly from the levels monitored in 2013 and 2014.
Analysing 74 billion bot requests, anonymised data from several hundred customers, and web traffic from 17 data centres, the company's research has shown that from 2014 to 2015, good bot traffic decreased from 36.32 percent to 27.04 percent of website traffic, and bad bot traffic decreased from 22.78 percent to 18.61 percent. The result is that humans now make up 54.4 percent of all website traffic.
Internet traffic boffins from Distil attributed this to a significant influx of new internet users, especially from China, India, and Indonesia. That, and the fact that bot operators continue to improve their software, creating more Advanced Persistent Bots (APBs).
Of this year's top 20 ISPs having the highest percentage of bad bot traffic, six came from China. As directed from their servers to our customers, over 72 percent of the traffic from these ISPs were comprised of bad bots. China Unicom reached a whopping 90 percent of bad bot traffic.
Of Advanced Persistent Bots, a freshly coined term, Distil says these now make up 88 percent of bad bot traffic, up from 77 percent in 2014. Meanwhile, simple bots decreased by more than half, from 23 percent of bad bot traffic in 2014 to 12 percent in 2015.
Distil Networks says that APBs have several advanced capabilities which include mimicking human behaviour, loading JavaScript and external resources, cookie support, browser automation, and spoofing IP addresses and user agents. APBs are much harder to identify and block than simple bots as they fly under the radar of many existing security solutions.
The persistence aspect comes from their ability to evade detection using tactics such as dynamic IP rotation (from huge IP address pools), using Tor networks and peer-to-peer proxies to obfuscate their origin, and distributing attacks over hundreds of thousands of IP addresses. For example, one bot might go through 1,000 IP addresses to make one request apiece, instead of a single IP address to make 1,000 requests.
As the bad bot landscape continues to evolve rapidly, especially in relation to the sophistication of bot software and the number of bots coming from Chinese service providers, this now means that thanks to cheap or free cloud computing resources, anyone with basic IT skills could download open source software and become a botnet operator.
Distil says that this means IT infrastructure teams are under increasing pressure to accurately forecast and provision web infrastructure to meet the speed and availability demands of legitimate users and stopping IT security teams from ensuring that nefarious actors can't harvest their data or breach their defences.
“When we dug into the bot activity in 2015, we identified an influx of Advanced Persistent Bots (APBs),” said Rami Essaid, co-founder and CEO of Distil Networks. “ABPs can mimic human behavior, load JavaScript and external assets, tamper with cookies, perform browser automation, and spoof IP addresses and user agents. The persistency aspect is that they evade detection with tactics like dynamic IP rotation from huge pools of IP addresses, use Tor networks and peer to peer proxies to obfuscate their origins, and distribute attacks over hundreds of thousands of IP addresses. A whopping 88 percent of 2015 bad bot traffic were APBs. This shows that bot architects have already taken note of traditional bot detection techniques and are finding new sophisticated ways to invade websites and APIs, in an effort to take advantage of critical assets and impact a business's bottom line.”
