The Bad of MITRE ATT&CK: Navigating the Paradox of Transparency
Welcome back to our series on the MITRE ATT&CK Framework: The Good, the Bad, and the Ugly. In our previous post, we explored how MITRE ATT&CK has transformed cybersecurity by standardizing vendor assessments. Today, we’ll delve into the challenges associated with the framework, focusing on the paradox of transparency and the potential for misrepresentation if not critically analyzed.
It’s worth noting that MITRE’s evaluations will take steps this year to address some of the shortcuts that vendors have exploited in the past. However, only time will tell how effective these changes are and how much the marketing around these evaluations will adapt. The challenge remains—how do we prevent these evaluations from becoming a playground for vendors looking to showcase unrealistic best-case scenarios?
The Double-Edged Sword of an Unranked Framework
One key feature of the MITRE ATT&CK evaluations is their lack of a ranking system. While this approach offers nuance, it also enables vendors to cherry-pick favorable statistics, crafting narratives that showcase their strengths while downplaying their weaknesses. This flexibility can mislead organizations, particularly those without in-depth technical expertise.
For example, Vendor A might highlight their 90% detection of privilege escalation techniques but conveniently omit poor performance in lateral movement. Vendor B might emphasize strengths in credential dumping while failing in behavioral analytics. This selective data presentation can skew the perception of a solution's overall capabilities.
Gamification of Vendor Evaluations
Another issue is how vendors optimize their platforms for MITRE’s controlled test environments. While not inherently unethical, these optimizations often don’t reflect real-world performance. Vendors enable features or make manual interventions that aren’t typically active in everyday deployments. The evaluations may showcase superior results that aren’t replicable in dynamic, real-world scenarios.
MITRE's ER6 evaluation has introduced some key improvements aimed at addressing past issues, particularly around the flood of false positives that skew real-world usability. This new focus on false positives is a direct response to the way vendors gamed previous tests, flagging everything to inflate detection rates. Additionally, they’ve expanded to include macOS, which broadens the applicability beyond just Windows. However, the lack of a standardized scoring system still allows vendors to cherry-pick results and shape marketing narratives. For more details, check out MITRE's ER6 evaluation.
It’s crucial to consider how vendors configure their systems for testing. Were features activated that aren’t standard? Were there interventions that won’t happen in live environments? These are often overlooked in marketing summaries but are critical for understanding actual product performance.
The Marketing Trap: Misleading with Selective Data
The absence of a scoring mechanism gives vendors free rein to emphasize their successes and hide their failures. A vendor may focus on their high detection rate in one attack phase (e.g., initial access) while glossing over weaknesses in others (e.g., lateral movement). Such selective presentations create a false sense of security, potentially leading to suboptimal purchasing decisions.
The challenge here is twofold: organizations must dig deeper into the evaluations and assess how these results align with their specific security needs. Relying solely on vendor-provided summaries risks distorting the solution’s effectiveness.
My Two Cents: A Call for a More Balanced Approach
While MITRE ATT&CK has revolutionized the cybersecurity landscape, its evaluations still present challenges for fair comparisons. Vendors often emphasize their strengths while concealing weaknesses, leading to skewed perceptions. What’s needed is a standardized comparison that balances detection rates with real-world relevance, allowing for a more informed decision-making process.
Until then, organizations must scrutinize these evaluations closely. Analyze the evaluation reports in full, contextualize them within your security needs, and ask hard questions about how these solutions perform in real environments—not just test scenarios.
The Challenge for Buyers: Navigating the Evaluation Landscape
Ultimately, the onus is on organizations to critically assess MITRE ATT&CK evaluations. It’s essential to consider how well vendors’ configurations match your real-world environment and not rely solely on top-line results. Vendors may tweak their systems for tests, so understanding how they perform in real scenarios is crucial.
When evaluating vendors based on MITRE ATT&CK results, consider the following:
Context is Key: Look at how the vendor’s solution was configured for the evaluation. Does it match how you would use it in your environment?
Beware of Over-Optimizations: Understand that vendors may have tweaked their systems to perform well in the test environment. Ask how their solution performs in real-world scenarios.
Focus on Your Needs: The best vendor isn’t necessarily the one with the most detections—it’s the one that aligns most closely with your specific security needs.
Conclusion: The Road Ahead for MITRE ATT&CK
MITRE ATT&CK remains a valuable tool for cybersecurity professionals, but organizations must approach the evaluations critically. The improvements in the ER6 evaluation are promising, particularly in addressing false positives and extending to macOS. However, the lack of standardized scoring leaves room for vendors to skew results in their favor.
It will be fascinating to see how future evaluations adapt to the changing threat landscape and how vendors respond to the challenges posed by an ever-expanding list of adversary techniques. One thing is certain: the MITRE ATT&CK Framework will remain a crucial tool in the arsenal of cybersecurity professionals for years to come.
Stay secure and stay curious, my friends!
Damien
Note: this and subsequent posts are neither an endorsement nor deterrent of MITRE or any other security vendor mentioned in this blog. Views are my own.