MITRE ATT&CK Eval ER6: New Eval, Who Dis?
New format, new platforms, new eval.
Last week, MITRE released its latest ATT&CK Evaluation: ER6, marking another critical milestone in the ongoing effort to assess the capabilities of cybersecurity solutions against the evolving threat landscape. This year’s evaluation introduces significant changes, reflecting MITRE’s commitment to staying relevant in a world of increasingly sophisticated cyber adversaries.
This blog is the first in a two-part series: today, we’ll explore the updates in ER6 and why they matter to security leaders, practitioners, and organizations aiming to stay ahead of emerging threats. Next week, we’ll dive deeper into the results and highlight what they reveal about the current state of cybersecurity defenses.
What’s New in ER6?
The 2024 MITRE ATT&CK Evaluation: ER6 demonstrates that MITRE is listening to feedback and anticipating an industry's needs under constant pressure. This evaluation stands out for its focus on real-world threats, expanded platform coverage, and an enhanced methodology designed to deliver more actionable insights.
macOS in the Mix
MacOS was included in the evaluation alongside Windows and Linux for the first time. This reflects a growing trend: macOS, once thought to be relatively secure, is increasingly targeted by sophisticated adversaries, including nation-state actors like DPRK-linked groups.
macOS usage is increasing across enterprise environments, making it a very real threat vector for enterprises.
Hybrid OS environments are now the norm, and attackers target every layer of an organization’s infrastructure. A security tool that excels only on Windows is no longer sufficient.
Including macOS was a savvy move by MITRE. As part of their evaluation’s core value proposition of comparing vendors in “real world” situations, having broader OS support ensures vendors are evaluated on their ability to provide comprehensive, cross-platform protection.
Ransomware Readiness:
Ransomware remains one of the most destructive threats to organizations, so MITRE turned its attention to the tactics and techniques employed by e-crime groups like CL0P and LockBit. These groups are responsible for some of the most high-profile and costly attacks in recent years.
Ransomware as a Service (RaaS) is exploding, with a CAGR of over 40% (as this is a very real illicit market). Check out my blog on this.
This emulation puts ransomware at the forefront. By emulating ransomware operations, MITRE offers organizations a clear insight into how well their tools can defend against one of the most pervasive and damaging cyber threats today. This evaluation is especially relevant for CISOs and security teams who want to benchmark their defenses against these high-stakes scenarios.
False Positives in Focus:
One of the most significant updates in ER6 is its focus on alert fidelity. The evaluation assessed tools not just on their ability to detect threats but on the relevance and actionability of their alerts.
True positive and false positive rates were key metrics, and these are noted in the results (this is super exciting).
We’ll be digging into these a LOT next week, but the TLDR is that more false positives = worse performance in this evaluation.
It’s a sad truth, but security teams are overwhelmed by alerts, many of which turn out to be false positives. Tools that reduce noise while delivering meaningful, actionable insights enable faster, more efficient responses—a critical capability in high-pressure situations.
Bonus: Modular Attack Chains
ER6 introduced multiple shorter attack simulations, moving away from the single, long attack chain of past evaluations. This modular approach mirrors real-world adversary behavior, where attackers adapt tactics mid-operation and persistently explore weaknesses. By testing agility across diverse attack chains, MITRE provides a clearer view of how security tools perform under real-world pressure.
Key Takeaways
The updates in ER6 demonstrate MITRE’s focus on aligning evaluations with real-world needs. Here’s what matters most:
Cross-Platform Defense is Critical: By including macOS, ER6 underscores the importance of securing every endpoint in hybrid environments.
Ransomware Detection is Essential: By emulating e-crime groups like CL0P and LockBit, ER6 highlights the need for tools that detect pre-positioning tactics, lateral movement, and data exfiltration.
Alert Fidelity Drives Action: Reducing false positives while maintaining broad detection coverage allows teams to focus on real threats, improving efficiency and outcomes.
The Results: A Quick Overview
While a full analysis of the results will come next week, here’s a snapshot of some standout performances:
Microsoft Defender XDR: Achieved 100% detection coverage for Linux and macOS, with zero false positives, demonstrating strong cross-platform efficacy.
Sophos XDR: Delivered near-perfect detection across Windows and Linux, earning top marks for almost every sub-step evaluated.
Bitdefender: Balanced high detection rates with minimal false positives, highlighting its ability to provide actionable insights without overwhelming security teams.
For the full results, visit MITRE’s website.
Why Should You Care?
ER6 is more than a vendor report card—it’s a practical benchmark for assessing real-world capabilities. By introducing macOS, focusing on ransomware, and prioritizing alert fidelity, MITRE’s evaluations reflect the threats organizations face today. These insights help security leaders:
Align investments with evolving attack patterns.
Identify detection gaps across platforms and tools.
Reduce noise and improve response efficiency with actionable alerts.
What’s Next?
In next week’s blog, I’ll examine the results more closely. We’ll explore how specific vendors performed, what the results reveal about the industry's state, and how you can use this information to make smarter security decisions. Until then, stay vigilant and stay informed. The attackers aren’t slowing down, and neither should we.
Stay curious and stay secure,
Damien
Note: this blog is purely my views and opinions. It is neither an endorsement nor criticism of any of the vendors explicitly or implicitly mentioned (or omitted).
Great read!!