Qualys Multi-Vector EDR Excels in 2022 MITRE ATT&CK Evaluation
Last updated on: December 21, 2022
MITRE evaluated Qualys Multi-Vector EDR against competing alternatives, and the results are in. This blog reviews the basics of MITRE ATT&CK evaluation, how our EDR solution performed, and how to interpret the ratings.
MITRE Engenuity has released the results of round 4 of its ATT&CK Evaluations based on the Wizard Spider and Sandworm APT Groups. With the release of Qualys Multi-Vector EDR in late 2020, we were able to fully participate in this year’s evaluation. MITRE’s latest ratings clearly show that Qualys can detect, analyze, and alert attacks throughout the entire chain of attack.
Bottom line: Qualys Multi-Vector EDR successfully detected MITRE’s simulated adversary all throughout the attack chain.
Overall, our solution detected 100% of the tested steps and returned 74% visibility into the entire attack chain. This is independent third-party verification that Qualys Multi-Vector EDR doesn’t overburden security teams with mountains of data to churn through, while accurately detecting attacks.
What’s more, our EDR solution enriches events using MITRE ATT&CK Tactics in addition to MITRE ATT&CK Techniques, something other EDR products simply do not do.
Webinar: Integrate VMDR with EDR to quickly reduce risk of compromise
Let’s review the MITRE methodology and our rating in more detail.
About the MITRE ATT&CK Evaluation
For the last four years, the MITRE Engenuity team has run a vendor evaluation program that tests security vendors against attacks based on real-world advanced persistent threat (APT) groups. In previous years the attacks were based off groups such as APT3, APT29, and Carbanak+Fin7. This year the tests were based off the Wizard Spider and Sandworm teams, which were prolific in deploying ransomware in 2016 and 2019, respectively.
Security vendors were invited to participate in this curated evaluation. Prior to the testing, a virtual environment was provided and Qualys installed our suite of Cloud Apps and configured it appropriately.
The main part of the evaluation focused on detecting the various attacks, with any mitigating technologies turned off. Many of the tools used throughout the attack are well-known hacking tools such as Mimikatz, which most endpoint solutions will detect and block. MITRE is more interested in the effectiveness of detections and mapping to MITRE ATT&CK classifications rather than blocking commodity malware.
The testing itself was split into multiple days where the Engenuity team acted as the red team attackers, while Qualys acted as the blue team defenders. Historically the tests themselves have been broken down into two days to either test using different attacker toolsets or to emulate a different adversary.
For example, this year day one was dedicated to Wizard Spider while the second was dedicated to Sandworm. For each day of testing, the Engenuity team broke down the attack into ten high-level stages, with each stage containing multiple sub-steps within it. Overall, there were 100 to 200 individual sub-steps during which the Engenuity team was specifically looking for some type of detection.
The detections themselves were broken down into one of six categories:
- N/A – specific to vendors like Qualys who did not deploy a Linux agent during the testing for that stage of the attack
- None – applied when the vendor had no insight into that specific event, which is common for high-volume events such as system API calls
- Telemetry – defined by MITRE as the first stage of Visibility and lays the groundwork for any detection the vendor would have. These are events which were collected but had no enrichment or detection applied to them.
- General – defined by MITRE as the first stage of Analytics, where the vendor created an alert but did not enrich it with any MITRE ATT&CK mappings
- Tactic – first part of the MITRE ATT&CK framework
- Technique – second part of the MITRE ATT&CK framework
Our overall goal was to collect evidence of the attacks and provide that to the Engenuity team for detection mapping. Categories 1-4 comprise how security products worked before ATT&CK became the de-facto framework for describing attacker behaviors. As mentioned above, MITRE groups and publishes its results along two main categories, either Visibility or Analytics.
Aside from N/A and None, anything flagged in categories 3-6 is grouped as Visibility, meaning there is some level of insight into the attack. Categories 4-6 are grouped in the Analytics bucket. Categories 5 and 6 are what the MITRE team wants to see. Vendors should be enriching the events with the Tactics and Technique names based on the ATT&CK framework. Ideally, most vendors want to find the right balance between collecting enough telemetry to detect attacks and providing quality alerts that are not overburdening users.
Interpreting the Results
The higher a vendor’s Visibility count, the more opportunities the product demonstrates to map events with an Analytic of either General, Tactic, or Technique.
Think of it this way: Events which are marked as “visible” are the hay in the haystack, while the analytic events are the needles. By having a low analytic count, those products leave their users to their own devices to try and hunt for adversarial behaviors. On the other hand, products having too high of an analytic count can overburden users and lead to alert fatigue.
Qualys Multi-Vector EDR Evaluation Results
Viewing the results of this latest round of evaluations, Qualys Multi-Vector EDR successfully detected the simulated adversary throughout a majority of the attack chain, as previously reported. Our EDR solution detected 100% of the tested steps and returned 74% visibility into the entire attack chain.
Unlike many alternative EDR offerings, Qualys Multi-Vector EDR was tested in its default “out of the box” configuration state. The only exception was our Anti-Malware Engine, which was downgraded to “Audit-Only mode” (instead of Auto-Quarantine) to allow the threats to execute their malicious payloads for the purposes of the MITRE ATT&CK evaluation. The result is superior protection without the configuration hassles of other EDR offerings.
Let’s review exactly how Qualys Multi-Vector EDR performed during the MITRE tests.
Immediate Detection
Digging deeper into the results, you can clearly see that Qualys Multi-Vector EDR provides insight immediately as soon as the malicious Word document was written to disk and executed.
While prevention capabilities were not tested in this round of MITRE ATT&CK evaluation, our EDR solution, when combined with our integrated anti-malware, would have blocked the attack. When set to “blocking mode”, it would have prevented the ransomware from ever running beginning from the moment the malicious payload was placed on disk.
Noise Cancellation
The attacker tries to persist in the environment via a registry run key, something which can cause headaches for security teams dealing with continuous infections after responding to events. Multi-Vector EDR provides rich context around this event and brings it to the users’ attention over the noise of the environment, as you can see here.
Rich Context
Another deep level of insight is the high level of context provided to the user as they are investigating events. As you can see as an example in step 10.A.1 not only was the event alerted, but Qualys Multi-Vector EDR also provides the context that this attack looks like the capabilities of the Wizard Spider team. There’s something unique here.
Competing EDR products do not enrich events with both MITRE ATT&CK Tactics and Techniques like Qualys Multi-Vector EDR does.
Incident Scoring Model
Another feature that helps cyber defenders protect their critical systems is our patent-pending incident scoring model. This newly introduced scoring model algorithmically calculates the impact that a particular attack technique will have on the asset. As seen in the example screenshot below:
What you see here is a risk score in addition to information about why the technique T1562.001 was flagged and deemed malicious. Defenders can use this to prioritize across multiple techniques and take remediation actions to protect their assets.
Power of the Qualys Cloud Platform
Qualys Multi-Vector EDR deploys a single agent on a single platform, combining all Qualys Cloud Apps such as Vulnerability Management, Policy Compliance, File Integrity Monitoring, and Cyber Security Asset Management into a single solution. Because of this, enterprises enjoy tremendous situational awareness when responding to events like what was tested in the MITRE ATT&CK evaluation. With the quick click of a button, users can see the security posture and system information of the asset being investigated, as shown below.
Webinar: Integrate VMDR with EDR to quickly reduce risk of compromise
Conclusion
MITRE Engenuity tested 30 security products during this fourth round of ATT&CK evaluations. Despite going head-to-head with EDR vendor offerings that have been in the market for over a decade, Qualys Multi-Vector EDR met the challenge. Our solution accurately detected MITRE’s simulated adversary across the attack chain. Qualys Multi-Vector EDR enriches events using MITRE ATT&CK Tactics as well as Techniques, a level of contextual analysis unique to current EDR offerings. These results attest to how Qualys Multi-Vector EDR leverages the Qualys Cloud Platform to sift through the noise, surface the data that matters most to the security team, and provide multiple layers of detection throughout any potential attack.
To help security teams protect, detect, and respond to advanced persistent threat groups like Sandworm and Wizard Spider, Qualys is offering Multi-Vector EDR free for 30 days.