Malicious actors have pounced on a pair of critical vulnerabilities found in SaltStack's open-source, event-based IT automation and configuration management tool Salt. In a series of quick strikes over the weekend, one or more attackers exploited the flaws -- disclosed and patched just days earlier -- to compromise the "Salt master" servers of several prominent users, including the Ghost blogging platform, the open-source mobile operating system LineageOS, and SSL certificate provider DigiCert.
The disruptive attacks highlight what some cyber experts say is an overlooked or underestimated threat vector among developers: Infrastructure-as-Code (IaC). Considered a key element of DevOps practices, IaC tools such as Salt typically allow developers to use code to automate the managing and provision of complex computer infrastructure environments, helping them avoid configuration discrepancies between machines that can hold up software deployments that might otherwise require manual intervention. But it's these helpful capabilities that can also make the exploitation of IaC tools uniquely dangerous.
"To understand the potential implications of an IaC, one must remember that IaC is designed to accomplish two fundamental objectives: consistency and speed," said Bill Santos, president and COO of Cerberus Sentinel. "IaC tools are designed to quickly deploy and update large environments in a very standardized way very quickly. The implication to an exploited IaC is significant: Whereas the consistency and speed is advantageous for 'approved' changes, an exploited change will get deployed equally quickly and equally consistently across that same environment, dramatically increasing its impact vs. other exploit approaches."
Santos added that many developers "are not appreciating the importance of IaC code and are not reviewing it, testing it, etc. at the same level they would application-level code. And in so doing, they are creating or increasing a very real threat vector."
Therefore, "It's important to elevate the significance of any automation code, especially IaC code, within the context of the development lifecycle," said Santos. "It is not 'second class' code, but rather carries the same importance and significance as any other code supporting an application. It needs to be reviewed, tested and assured in a [manner] similar to every other element of an application architecture."
Indeed, in the recently released Spring 2020 edition of the Unit 42 Cloud Security Report, researchers with Palo Alto Networks's global threat intel team warned that developers are failing to scan IAC templates for security issues whenever they are created or updated, which raises the likelihood of encountering exploitable cloud vulnerabilities.
"We found that nearly 200,000 IaC templates contained at least one vulnerability or misconfiguration, which range in severity from exposing systems to the public to disabling encryption and logging requirements. So yes, IaC is often overlooked as a serious threat vector," said Nathaniel Quist, senior cloud threat researcher with Unit 42. "As an industry, we should encourage all organizations to employ the proper implementation of IaC templates within a vetted and secure CI/CD Development Operations using Cloud Native Security Platforms (CNSP). IaC templates greatly increase the speed at which organizations can deploy business-critical applications, but without proper security oversight, they could also increase the speed in which they open themselves up for malicious attacks.
The various attacks took place after adversaries scanned the internet looking for Salt "masters" -- servers used to control "minions" that carry out tasks for the IaC tool -- that were both exposed over the internet and vulnerable to the two bugs. Users are vulnerable to exploit only if these conditions are met.
Ghost on May 3 reported an outage affecting its services, later reporting that an actor exploited vulnerabilities in its Salt server management infrastructure to install cryptojacking software. "The mining attempt spiked CPUs and quickly overloaded most of our systems, which alerted us to the issue immediately," the blogging platform stated.
In a subsequent update, Ghost said it removed the cryptominer and added "multiple new firewalls and security precautions," the introduction of which ironically further disrupted customer blog sites temporarily. "At this time there is no evidence of any attempts to access any of our systems or data," Ghost asserted. "Nevertheless, all sessions, passwords and keys are being cycled and all servers are being re-provisioned."
Jeremy Rowley, VP of business development at DigiCert, reported via a May 3 Google Groups post that a CT (Certificate Transparency) Log 2 key used to sign Signed Certificate Timestamps was compromised.
"We are pulling the log into read-only mode right now," the post said. "Although we don't think the key was used to sign SCTs (the attacker doesn't seem to realize that they gained access to the keys and were running other services on the [infrastructure]), any SCTs provided from that log after 7pm MST yesterday are suspect. The log should be pulled from the trusted log list." Rowley later said in an update that the log "should be distrusted for everything after 17:00:02 on May 2."
And LineageOS reported that on May 2, a malicious actor accessed its Salt master "to gain access to our infrastructure." LineageOS's services were knocked temporarily offline, forcing the developer to restore them in piecemeal fashion. However, signing keys and builds were unaffected.
Researchers with F-Secure, who discovered the flaws, reported last Friday in a blog post and corresponding advisory that attackers could exploit the bugs to "bypass the authentication and authorization controls used to regulate access to Salt implementations" and then remotely execute code with root privileges on the master, allowing for control of all its minions.
"Patch by Friday or compromised by Monday," said F-Secure Principal Consultant Olle Segerdahl in the blog post.
F-Secure says it conducted its own scan and found 6,000 instances of exposed Salt masters. "I was expecting the number to be a lot lower. There's not many reasons to expose infrastructure management systems, which is what a lot of companies use Salt for, to the internet," said Segerdahl.
However, Alex Peay, SVP of product at SaltStack, characterized the 6,000 instances as a "very small portion of the [Salt] install base," adding that "Clients who have followed fundamental internet security guidelines and best practices are not affected by this vulnerability."
According to SaltStack's official advisory, the two bugs, designated CVE-2020-11651 and CVE-2020-11652, were discovered in the salt-master process ClearFunc class of Salt versions prior to 2019.2.4 and 3000.2. The former bug is due to the improper validation of method calls, and "allows a remote user to access some methods without authentication. These methods can be used to retrieve user tokens from the salt master and/or run arbitrary commands on salt minions," the advisory states. The other flaw "allows access to methods that improperly sanitize paths. These methods allow arbitrary directory access to authenticated users," the advisory continues.
In a patch issued at the end of April, Salt fixed the validation process. However, attackers did not waste time taking advantage of users who did not immediately update one of the patched, secure versions.
"Although there was no initial evidence that the CVE had been exploited, we have confirmed that some vulnerable, unpatched systems have been accessed by unauthorized users since the release of the patches," said Peay. "We must reinforce how critical it is that all Salt users patch their systems and follow the guidance we have provided outlining steps for remediation and best practices for Salt environment security. It is equally important to upgrade to latest versions of the platform and register with support for future awareness of any possible issues and remediations."
James McQuiggan, security awareness advocate at KnowBe4, said that the Salt vulnerabilities can be abused for a lot worse than just the reported cryptomining scam.
"If organizations do not update their SaltStack, they are exposed to an attack where malware, ransomware or attack vectors can be initiated to gain control, steal intellectual property or hold an organization's data for ransom," said McQuiggan. "Incident response for organizations needs to be swift to implement testing and patching of the servers using SaltStack. If they cannot be updated, additional steps will be required to reduce access on applications, users and systems to only those necessary and required for access."
Quist from Unit 42 offered these key takeaways for IaC users: "Trust but verify all network operations. All user access events should be monitored and only authorized users should be given access. Changes or updates to all Salt master or minion nodes need to be vetted to ensure no security risks are present. No changes should be allowed to occur to any Salt IaC template without approval and changes need to be verified for integrity. All requests for change need to be properly authenticated and their integrity needs to be verified."
