In the time between a bug's initial disclosure and its listing in NIST's database, a race develops between white hats looking to solve the flaw and adversaries hoping to exploit it.
In the time between a bug's initial disclosure and its listing in NIST's database, a race develops between white hats looking to solve the flaw and adversaries hoping to exploit it.

Three-quarters of the bugs listed in the U.S. National Institute of Standards and Technology's official vulnerability database for 2016-17 were previously exposed on the open, deep or dark web, before the agency had an opportunity to add those flaws to its records, according to new machine learning-aided research.

This gap between the first online reference to a bug and its addition to NIST's National Vulnerability Database (NVD) could endanger organizations exposed to that particular flaw if they rely primarily on the NVD for bug updates and alerts. During this window of time, a race essentially develops between security professionals who are looking to spread awareness of the vulnerability and create a solution, and adversaries who wish to exploit the bug, explained threat intelligence firm Recorded Future, which published its research findings in a blog post on Wednesday.

According to the study, which looked at 12,517 CVEs published by NIST, the median (not average) gap between the first online reference to a bug and its subsequent appearance in the NIST database was seven days. But gaps could also be much longer: 25 percent of published CVEs didn't show up on the NVD for at least 50 days, while 10 percent weren't listed by NIST for at least 170 days.

"This median gap is increasing, complicating the ability of vulnerability management teams to stay current and, with increasing gaps, the situation is worsening," the blog post states.

Bill Ladd, chief data scientist at Recorded Future, told SC Media in an interview that publication in the NIST database was selected as a key benchmark because the NVD represents a centralized repository of bug information that many organizations rely on to stay secure. While researchers and companies also may publish their own bug disclosures, the sheer number of these reports on the web is "pretty staggering," he said.

"There's a lot of sites to try to be monitoring. The sheer complexity of where all of this data is coming from is hard for an individual to keep up with," said Ladd, explaining why the NVD is such an important resource to have, and why it's so important that it stay current. (Ladd also noted that the report in no way intends to fault NIST for any lag time between a vulnerability's disclosure and its appearance in the NVD.)

Recorded Future used natural language processing and machine learning to seek out online reports of various bugs, and then identify the earliest such references in order to accurately measure NVD publication lag time. The company found over 1,500 online sources that collectively reported on bugs prior to their appearance on the NIST database more than 114,000 times.

Ladd said that many references to vulnerabilities amounted to legitimate, responsible disclosures. Nevertheless, according to the report, five percent of the CVEs in question were detailed on the deep or dark web prior to NIST listing them, suggesting that in some cases cybercriminals were circulating bug reports to alert the underground community of a potential exploit opportunities. Major vendors such as Google, Apple, Microsoft, and Oracle were most often involved in such instances.

Recorded Future also noted that higher severity vulnerabilities tended to have shorter gaps between first disclosure and NIST listing their than lower severity counterparts, which indicates that these critical cases were prioritized.