Reflections on third-party testing and its importance
This week NSS labs announced the first public Advanced Endpoint Protection (AEP) test results. The test was conducted for more than six months and aims to be the most comprehensive, in-depth test to exist today.
As such it covers a large variety of attack vectors, malware types and usage patterns.
It clearly shows SentinelOne Endpoint Protection Platform (EPP) as a leader, with best TCO and nearly perfect detection scores. In this blog I want to share some thoughts about the test, it’s results and some future suggestions.
The process leading up to the NSS test
When NSS approached us in March 2016 to join this brand new test methodology, which aims to be the first test to cover malware types, exploits and false positives tests in a comprehensive way. We’ve had that exact same discussion with other testing labs, and our only real request from the labs was to make the playing field level. Specifically, this means:
- Malware is actually executed, not just copied in
- Exploits must actually be working; meaning the payload is dropped and running and something bad actually must happen on the user’s system
Quite simple, though these requirements do incur a significant overhead on the testing lab as they have to actually validate every malware is indeed running rather than throwing tens of thousands of “samples” and hoping for the best.
This is not the first time we’ve had this discussion with a testing lab. We’ve had the same exact conversation with AV-test, AV-comparatives and MRG-Effitas over the past two years. We see this process as the evolution of the testing ground to better reflect today’s threats, and better reflect the efficacy of the solutions tested.
We believe that participating in the tests is a beneficial tool when done correctly and can be a useful point of consideration for customers. We realize not all vendors agree and some have recently – publicly – shared their concerns. Some vendors also choose not to join the tests; we chose to participate.
Why?
First – participating in third party testing is a crucial step to being considered a serious player in the field.
Second – we believe participation in such tests helps to improve the overall performance of the product. In short, it puts a mirror in front of us and other tested vendors, forcing us to improve our products. Think of it as kind of an external red team that challenges the products.
The test results
SentinelOne received one of the highest security effectiveness ratings at 99.79%. I would like to drill down into where the remaining 0.21% came from.
False Positives – 0
HTTPS malware – 100% protection
Email – 100% protection
P2P – 100% protection
Local intelligence (i.e the agent is not connected to the internet while being infected) – 100% protection
Exploits – 100% protection
Blended threats – 100% protection
HTTP – 98.5% protection
Evasion – 93.8% (2 missed “evasions”)
Let’s look further at the evasions part of the NSS test. This part uses different techniques to hide the malware from being detected by the security product. Now the way SentinelOne EPP works, it should be completely agnostic to evasions, since the product observes the operations actually being performed on the machine, rather than looking just at the files. So it came as a surprise to us seeing these two evasions.
The NSS test provided the samples used, and we analyzed them in our research lab.
What we found:
- Both samples are basically the same sample just altered a bit.
- Both refuse to run on a VM. Instead, they try to communicate with their C&C server, letting it know it’s running on a VM. Then the C&C does not deliver the payload and the malware simply sits on the system, not actually doing anything malicious.
- When the malware is executed on a physical machine, it is running, and is indeed caught immediately upon executing the payload.
Of course, we disputed these results, claiming that a malware that doesn’t run should not be counted as a miss. However, by the time NSS tried to validate our claims, the C&C was already dead and did not deliver the payload at all. So the dispute is still open, but the results are already out.
Unfortunately, this is another case where nothing happened on the system, but the testing vendor considered the test case “malicious.” This is exactly the kind of problems we were hoping to avoid.
Future thoughts
We believe that NSS Labs took very significant steps forward, they performed the most comprehensive test we’ve seen so far in the testing industry, and they leveled the field while adjusting to the new reality of advanced threats and updated methodologies and products. For that, we appreciate the efforts made by the NSS Labs team on this test.
We believe that the test methodology must continue to evolve, in order to catch up with current threats. Things like documents, scripts, in-memory attacks, et., did not get their proper representation in this test, though they certainly exist in the real world. Also, the fact that most vendors achieved 95%+ scores – and many above 98% – means the test does not reflect the reality enterprise customers have to deal with.
Further, we believe that other aspects of innovation – such as security vendor warranties, like SentinelOne’s cyber protection warranty against ransomware attacks – will be points of consideration for testing labs as more customers push security vendors to offer such programs.
We challenge all testing labs to continue to push the limits of testing and live up to their mission statement to improve the overall security of the industry. We, at SentinelOne, are willing to work with any third party labs to improve the test methodologies and practices so the tests reflect real-life threats and show the differentiation between the different vendors.
Finally, I am proud to be part of the team here at SentinelOne producing these amazing results!
