A malware campaign is abusing the popular machine-learning (ML) framework Kubeflow in order to target Kubernetes clusters with a crypto miner, Microsoft's Azure Security Center (ASC) warns.
Tens of clusters running on the Kubernetes open-source container orchestration system have already been impacted, the ASC notes in a blog post published this week.
"Nodes that are used for ML tasks are often relatively powerful, and in some cases include GPUs. This fact makes Kubernetes clusters that are used for ML tasks a perfect target for cryptomining campaigns, which was the aim of this attack," explains blog post author Yossi Weizman, security research software engineer with ASC.
"Organizations should be mindful of the registries that users/clusters are allowed to download from," said Wei Lien Dang, Co-Founder and Chief Strategy Officer at StackRox. "They should use private trusted registries, whitelist allowed images, and take other precautions to verify source assets. As Kubernetes clusters get larger and more powerful (as in this case with GPUs to run ML), they'll become even more attractive for this type of attack. Organizations must take specific steps to ensure they’re protecting their container and Kubernetes assets across build, deploy, and runtime.”
Microsoft Azure researchers first observed the campaign in April after noticing that a series of different clusters -- most running Kubeflow -- contained a malicious image that runs the XMrig miner and came from a public repository. That repository was found to contain additional suspect images with similar mining purposes but different configurations.
"Kubeflow is a containerized service: the various tasks run as containers in the cluster. Therefore, if attackers somehow get access to Kubeflow, they have multiple ways to run their malicious image in the cluster," the blog post states.
The attackers have been using exposed Kubeflow dashboards as their attack vector to gain access. "The execution and persistence in the cluster were performed by a container that was deployed in the cluster," Weizman writes. "The attacker managed to move laterally and deploy the container using the mounted service account. Finally, the attacker impacted the cluster by running a cryptocurrency miner.
Kubernetes users are exposing their dashboards to attack, ASC explains, when they configure their Istio open-source independent service mesh service to Load-Balancer setting, thereby connecting the service to the internet, the blog post explains.
"We believe that some users chose to do it for convenience: without this action, accessing to the dashboard requires tunneling through the Kubernetes API server and isn’t direct," writes Weizman. "By exposing the service to the internet, users can access to the dashboard directly. However, this operation enables insecure access to the Kubeflow dashboard, which allows anyone to perform operations in Kubeflow, including deploying new containers in the cluster."
Once they have dashboard access, attackers can deploy a backdoor container within the cluster via the Jupyter Notebook open-source web application for creating and sharing documents with live code. According to ASC, attackers could either choose their own malicious image for the Jupyter notebook server, or they could deploy a malicious container from a genuine Jupyter notebook.
In addition to ensuring their dashboards are not internet-connected, Kubernetes users can use a command provided by ASC to ensure a malicious container from this attack isn't already deployed in their cluster. ASC also advises Kubeflow users to maintain awareness of authentication and access control settings, ensure sensitive cluster endpoints are not publicly exposed in an unsecured manner, regularly monitor the runtime environment and allow only trusted images and scan images for vulnerabilities.
"These kind of attacks are not new, and we have seen a continuous effort by bad actors to find and exploit any opening, no matter how small, to gain access to container runtime environments," said Tsvi Korren, field CTO at Aqua Security. "The fact that these environments blindly accept commands to pull (download) and run any publicly available image is giving a great incentive to attackers, because once they find an opening, they can run basically anything they want. In this case it was stealing computing resources, but it could have easily taken out data, explored more deeply into the network or cause major disruption."
"There are hundreds if not thousands of purposely-built malicious images in many public repositories, including the Docker Hub," Korren continued These can be brought in during an attack, but they can also be brought into organizations by developers who just want a shortcut to a useful piece of code. There needs to be broader awareness that any image out there could be embedded with code used to attack Kubernetes. The only way to defend against that is for organizations to have policies that require both static scanning and dynamic analysis of the images that they accept."
