Recently, Intezer cybersecurity researchers uncovered an attack utilizing a new Linux malware targeting publicly accessible Docker servers. The new malware, dubbed “Doki”, is part of an active Ngrok Mining Botnet campaign, primarily targeting exposed Docker servers hosted with popular cloud platforms such as AWS, Azure, and GCP among others. This sophisticated attack exploits misconfigurations in Docker features, which are both common and can be difficult to avoid, and drops the Doki backdoor as one of its payloads.
The initial report noted that “Doki” went unrecognized as malware on VirusTotal for over seven months and claimed it was a “fully undetected backdoor”. Combined with the initial infection’s container escape technique, this has led to fears that enterprises making use of Docker servers are left with little hope of detecting this new kind of attack in the wild, and pressure has naturally mounted on SecOps and DevOps teams to ensure all Docker instances are properly configured in a ‘best effort’ attempt to secure container and cloud workloads. However, while ensuring proper configuration is certainly a fundamental part of an effective security posture, it is also difficult and time consuming; more importantly, it is also not enough to stop attackers that have exploited existing misconfigurations or who go on to discover further container vulnerabilities.
In this post, we show how the container escape and Doki malware attack proceeds, step by step, and demonstrate that neither are “fully undetectable”. We show that this and similar threats can be detected and mitigated against by means of SentinelOne’s Container Escape Protection, part of the SentinelOne Linux and Kubernetes Sentinel Agents.
Container Escape and Privilege Escalation
The main prize for the attackers is to achieve remote code execution on the host, and to this end they leverage the Docker API Create to set up their own containers. As previously reported, by using a legitimate Docker alpine image with curl installed, the attackers are able to use a bind configuration, which internally calls mount syscall, to bind /tmpXXXXXX to the root directory of the hosting server.
Having managed to execute code in the container and get access to the host, the attackers have the option of implementing different persistence methods to overcome the challenge of the average short lifespan of any individual container. In this attack, the initial payload gains persistence in the early stages right after the bind mount configuration by mapping cron to the malicious container.
Detecting the Container Escape with SentinelOne
As Gartner have previously pointed out, enterprises that try to use standard EPP solutions to protect server workloads are putting their business at risk. The only way to detect behaviors that involve correlation between container operations on a host’s file system is through an advanced AI technology that has visibility and understanding of the whole system – both host and containers – at once.
The SentinelOne agent is able to stop this attack precisely because it is constantly monitoring all activities and the malicious cron modifications are immediately detected, as shown below in the console’s threat page. Note how the threat indicators map the activity to MITRE ATT&CK TTPs for the analyst’s convenience:
The console also offers a useful graphical overview of the process tree:
And full logs are readily available showing all events from the current threat within the same interface:
Detecting the “Undetectable” Doki Malware Payload
SentinelOne’s agent is fully able to detect the container escape, but what about the malware that went undiscovered on VirusTotal for so many months and which was said to be “undetectable”?
Certainly, the malware and the initial attack are different steps that attackers could easily use separately; the malware could be dropped from different attack vectors, and it’s equally likely now that Doki has been “discovered” we will see new malware that has yet to be found on VirusTotal or any other malware repository.
Fortunately, the SentinelOne agent does not rely on reputation or cloud connectivity, but analyses processes in real time locally on the device using our advanced machine learning model to detect and protect against abnormal behavior. The on-device agent monitors every process, file and network activity in both the host and containers together, allowing it to capture suspicious and malicious activity autonomously. As the following images show, Doki’s behaviour is immediately recognized by the SentinelOne agent as malicious.
Are There Other Container Escape Techniques?
The particular container escape used in this attack is not the only one available to threat actors. Last year, a security assessment of Kubernetes and Docker presented a different Proof of Concept for achieving a container escape. The PoC relied on another misconfiguration where the container has elevated privileges, either by the --privileged flag or the apparmor=unconfined flag. The escape can be triggered by an exploit using the Linux cgroups (control groups) mechanism and a ‘release_agent’ file.
Linux control groups are intended to allow multiple Docker containers to run in isolation while limiting and monitoring their use of resources. However, the ‘release_agent’ file contains a command that is executed by the kernel with full privileges on the host once the last task in a cgroup terminates. The PoC abuses this functionality by creating a ‘release_agent’ file with a malicious command, and then killing off all the tasks in the cgroup.
As the cgroup files are present both in the container and on the host, it is possible to modify them from either, which means an attacker can spawn a process inside the cgroup and gain code execution on the host.
# On the host docker run --rm -it --cap-add=SYS_ADMIN --security-opt apparmor=unconfined ubuntu bash # In the container mkdir /tmp/cgrp && mount -t cgroup -o rdma cgroup /tmp/cgrp && mkdir /tmp/cgrp/x echo 1 > /tmp/cgrp/x/notify_on_release host_path=`sed -n 's/.*perdir=([^,]*).*/1/p' /etc/mtab` echo "$host_path/cmd" > /tmp/cgrp/release_agent echo '#!/bin/sh' > /cmd echo "ps aux > $host_path/output" >> /cmd chmod a+x /cmd sh -c "echo $$ > /tmp/cgrp/x/cgroup.procs"
The SentinelOne agent’s Behavioral AI is able to detect this exploitation attempt, providing full visibility and the Storyline of the attack vector that led to this malicious activity.
Conclusion
It is a good strategy for defenders to be familiar with and execute core workload protection strategies, but as the recent Doki and container escape malware attacks show, as soon as there is a weak link in the chain, the attacker will take advantage and such strategies will fail to protect the enterprise.
Modern attack methods in containerized environments in the cloud are gaining traction and becoming increasingly sophisticated. Given the rewards, threat actors are clearly willing to expend more effort to stay under the radar and to defeat “best practices”.
To fully protect your assets, move to a container protection solution, powered by unmatched behavioral AI models, that can autonomously detect and block malware across both hosts and containers. SentinelOne’s server and workload protection is infrastructure agnostic and can be deployed either in containers themselves, or in the machines that host them, in servers or in the cloud. If you would like to see how SentinelOne’s solution can work for you, contact us for more information or request a free demo.
