The dark web is actually much smaller than you think, and the percentage of websites residing there that provide illegal content and services is also over-exaggerated, according to a new report from Intelliagg and Darksum.
The report, “Deeplight: Shining a Light on the Dark Web,” claimed to be the first comprehensive, definitive mapping of the largest dark net, Tor, where hidden websites are not indexed by search engines and content cannot be accessed without specialized encrypted software. This mysterious online frontier, shrouded in secrecy, is surprisingly composed of only about 30,000 websites, each featuring Tor's .onion top-level domain suffix, the report states.
Researchers at the two firms created an index of dark web pages by consulting preexisting lists of Tor links, and using a collection software application known as a spider to sniff out and follow additional links that connect various sites. To further probe the corners of the dark web, the researchers also leveraged over 100 servers to observe traffic requests on the Tor network in order to identify sites that otherwise had no discernible links. Indeed, 39 percent of analyzed sites could not be found with a link, making it critical for the researchers to monitor the Tor network for clues.
Only 46 percent of the 29,532 discovered websites were accessible during the research's sampling period — of these, the researchers used a combination of manual analysis and automated tools to classify their content. As it turns out, less than half of the computer-analyzed sites — only 48 percent — were determined to provide content or services that would be deemed illegal in the U.S. or U.K. This includes child pornography, drugs, guns, hate speech and other criminal activity.
The automated solution may have overlooked some illegal activity, as the separate manual analysis found 68 percent of 1,200 sampled sites carried illegal content. But that still leaves a lot of sites whose content is perfectly legal, though perhaps controversial or sensitive in nature, thus warranting discretion.
“I thought it would be mostly guns and drugs, but actually if you look at the content, there is much more file sharing, discussion boards and generally good uses for the dark net, said Thomas Olofsson, CEO of Intelliagg, in an interview with SCMagazine.com. “Some of the sites were anything from whistleblowing sites to sites where journalists are digging up research,” he continued. The report also references sites specializing in everything from political discourse to sexual fetishes.
However, just because these relatively innocuous sites exist doesn't mean that's where most dark web denizens are spending their time. “When you read just the Deeplight report, it sounds like the situation is not as bad as the media portrays,” said Eric Jardine, research fellow at the Centre for International Governance Innovation (CIGI) and a dark web expert, in an interview with SCMagazine.com. But “I think there's a step missing and that step has to do with traffic patterns,” he remarked, pointing to a separate research report by the Global Commission on Internet Governance, which in its own sampling found that more than 80 percent of requests on Tor were made to so-called abuse sites representing just two percent of the dark net's hidden services.
Intelliagg and Darksum's joint report also found that 76 percent of analyzed Tor websites were written primarily in English, with German (four percent) the next most common language. Olofsson noted he was surprised there were not more sites catering to users in Russian or Chinese-speaking regions, though he expects there will be in the near future as more cybercriminals from these areas turn to encrypted sites to do their business.
Ultimately, Jardine praised the report for shedding new light on the dark web. “I appreciate any effort to make sense of the dark web — what it is, how big it is — and to try to categorize .onion sites and catalogue them. This is a really good step,” said Jardine. However, he also noted that uncharted pockets of the dark web still remain, as this research focused exclusively on Tor. “Tor is the biggest route to the dark web, but not the only one,” he added.