Audit: FBI's threat prioritization process too subjective and sluggish
An audit of the FBI's Threat Review and Prioritization process found that “cyberthreats that require the greatest resources may not receive the highest priority.”
The FBI's process for prioritizing cyberthreats is currently too slow and subjective, hindering its ability to optimize allocation of resources to address these dangers, according to a newly released report from the Department of Justice's Office of the Inspector General.
The minimally redacted audit evaluates the FBI's annual Threat Review and Prioritization (TRP) process, which judges cyberthreats based on the potential damage they can inflict and the mitigation efforts they would require. The most severe threats are assigned the highest priority, so that agency leadership can assign proportionate human and financial resources to address them. However, the OIG's review of the TRP process from 2014-2016 found that it relies too heavily on human judgment, as well as criteria that are too easily open to interpretation.
The report states that that some FBI officials characterized the prioritization process as “vague and arbitrary,” or basically a “gut check."
“We found the criteria used in the TRP process are subjective and open to interpretation. As a result, the FBI's TRP process does not prioritize cyberthreats using an algorithmic, objective, data-driven, reproducible, and auditable manner,” the report reads.
Based on these observations, the audit concludes, “cyberthreats that require the greatest resources may not receive the highest priority.”
Also, because the TRP process is conducted only annually, there is a danger of not reacting quickly or sufficiently enough to newly emerging or quickly evolving threats, which suddenly might require an impromptu (and therefore likely sub-optimal) reallocation of resources.
The good news is that FBI may have at least a partial solution in its Threat Examination and Scoping (TExAS) tool, which the FBI's Cyber Division began developing in 2014. TExAS uses algorithms to assign a score to a particular cyberthreat, based on the responses to 53 impartial, weighted questions about the nature of the threat. (Responses must be backed up by documented evidence.)
The FBI already uses TExAS to “augment” the prioritization process, but the OIG's office believes that as the tool matures it could play an integral role, as it “represents a best practice that could streamline and improve the prioritization within the Cyber Division, and potentially across other FBI programmatic areas as well.”
However, the OIG did recommend several key tweaks after pointing out several weaknesses. For instance, TExAS is only updated with cyberthreat data once a year because the data entry process is entirely manual – a consequence of it not being integrated with the FBI's Sentinel case management system. Moreover, there is a lack of official procedures stating whose specific responsibility it is to enter the data, and exactly how the data should be factored into the TRP process. Additionally, several of the weighted questions used to inform the tool could use more definition, the OIG concluded.
To address some of these concerns, a letter sent from the FBI that was incorporated into the published report said that that the FBI is drawing up new agency policies for TExAS, and that Sentinel will be programmed to automatically send TExAS daily data updates by FY 2017, while employees will supplement these updates manually on a monthly basis.
“With more frequently refreshed data, we believe that TExAS, or a system of similar ability, has the potential to provide a current picture of the cyberthreat landscape, including emerging cyberthreats as well as known threats that are adapting techniques, tactics, and procedures that receive little emphasis in the annual FBI TRP process,” the report reads.
On a separate note, the audit report also critiques the FBI's Time Utilization and Recordkeeping (TURK) system, which tracks the allocation of human capital to various investigations. According to the OIG, the system is incapable of tracking time expenditure by specific threat, only by case classification – a broader category that can include multiple threats of the same basic type. Consequently, it can be impossible to distinguish precisely how much time an agent spent investigating one specific threat versus another. The FBI concurred with the OIG's recommendation to develop a record-keeping system capable of capturing this data, and confirmed in its letter to the DoJ it is actively looking into it.