Bot traffic on the descent: Will the real GET request please stand up?
The humans are clawing back the interest as new research from Distil networks shows a slight decline in bot traffic for 2015
Bots are becoming more advanced
New research from Distil Networks has shown that in 2015, overall bot traffic, as compared to human traffic, decreased slightly from the levels monitored in 2013 and 2014.
Analysing 74 billion bot requests, anonymised data from several hundred customers, and web traffic from 17 data centres, the company's research has shown that from 2014 to 2015, good bot traffic decreased from 36.32 percent to 27.04 percent of website traffic, and bad bot traffic decreased from 22.78 percent to 18.61 percent. The result is that humans now make up 54.4 percent of all website traffic.
Internet traffic boffins from Distil attributed this to a significant influx of new internet users, especially from China, India, and Indonesia. That, and the fact that bot operators continue to improve their software, creating more Advanced Persistent Bots (APBs).
Of this year's top 20 ISPs having the highest percentage of bad bot traffic, six came from China. As directed from their servers to our customers, over 72 percent of the traffic from these ISPs were comprised of bad bots. China Unicom reached a whopping 90 percent of bad bot traffic.
Of Advanced Persistent Bots, a freshly coined term, Distil says these now make up 88 percent of bad bot traffic, up from 77 percent in 2014. Meanwhile, simple bots decreased by more than half, from 23 percent of bad bot traffic in 2014 to 12 percent in 2015.
The persistence aspect comes from their ability to evade detection using tactics such as dynamic IP rotation (from huge IP address pools), using Tor networks and peer-to-peer proxies to obfuscate their origin, and distributing attacks over hundreds of thousands of IP addresses. For example, one bot might go through 1,000 IP addresses to make one request apiece, instead of a single IP address to make 1,000 requests.
As the bad bot landscape continues to evolve rapidly, especially in relation to the sophistication of bot software and the number of bots coming from Chinese service providers, this now means that thanks to cheap or free cloud computing resources, anyone with basic IT skills could download open source software and become a botnet operator.
Distil says that this means IT infrastructure teams are under increasing pressure to accurately forecast and provision web infrastructure to meet the speed and availability demands of legitimate users and stopping IT security teams from ensuring that nefarious actors can't harvest their data or breach their defences.