MITRE ATT&CK: The Good, The Bad, and The Ugly
Part I: The Good - Know Thy Enemy
Part I: The Good - Know Thy Enemy
Introduction
Welcome back to our regularly scheduled ABCbyD format, where we make the esoteric accessible for our amazing (and rapidly growing) cyber-curious community!
This week, we’re digging into something I’ve had exposure to since its public inception: the MITRE Adversarial Tactics, Techniques, and Common Knowledge (ATT&CK - pronounced “attack”) Framework. This is the first in a three-part series discussing the good, bad, and ugly of one of the best—and most popularly used—cyber defense frameworks.
This week, we’ll be discussing the “Good” of MITRE’s ATT&CK framework: its benefits to cybersecurity practitioners, explosive growth, and utility. In the coming weeks, we’ll engage on the complexities and (not necessarily MITRE’s fault) risks of ATT&CK with posts on the “bad” and “ugly” of MITRE’s framework.
The MITRE ATT&CK Framework has become the gold standard for network defenders. We love it because it is detailed, comprehensive, and, much like how art imitates life, it imitates actual adversary activity!
But what is ATT&CK? It’s a public, free evaluation of cybersecurity tools against advanced attacker activity, painting a picture of what coverage you (as a user of a security vendor’s tool) would see if you were attacked by a sophisticated cyber threat.
It’s a combination of expert and crowd-sourced intelligence, and it remains one of the most comprehensive cybersecurity frameworks today.
History of the MITRE ATT&CK Framework
The MITRE ATT&CK Framework was born out of necessity. In the early 2010s, MITRE, a not-for-profit organization that operates U.S. federally funded research and development centers, identified a pressing need within the cybersecurity community: a common language and framework to describe and categorize adversarial behaviors.
This need became apparent during an internal research project at MITRE called FMX (Fort Meade eXperiment), which focused on understanding how adversaries operated within compromised networks. The project sought to capture and document the specific tactics and techniques used by threat actors during real-world intrusions.
By 2013, MITRE had formalized these observations into what is now known as the MITRE ATT&CK Framework. Initially, it was a small, internal project used to map out specific adversary behaviors in active intrusions. However, the potential impact of such a framework quickly became evident.
The MITRE ATT&CK Framework provided a structured way to analyze and discuss adversary tactics and techniques, allowing cybersecurity professionals to share knowledge, develop more effective defenses, and better understand the behavior of threat actors.
Growth of the MITRE ATT&CK Framework
Since its inception, the MITRE ATT&CK Framework has seen tremendous growth in terms of scope addition of tactics and techniques, participation and usage. What started as a relatively small project focused on Windows environments has expanded into a comprehensive and versatile framework covering multiple platforms, including Linux, macOS, mobile devices, and even Industrial Control Systems (ICS). If its breadth of coverage wasn’t enough, evaluation results are free to the public—no subscriptions, magic quadrants, or paid studies. Free.
The introduction of ATT&CK for ICS in 2017 was a significant milestone, reflecting the growing concern over threats to critical infrastructure. ATT&CK for ICS was created in response to the increasing frequency of attacks targeting these systems, focusing on the unique challenges and tactics used in this domain.
Similarly, the development of ATT&CK for Mobile, also launched in 2017, addressed the rising threat landscape associated with mobile devices, which have become integral to both personal and professional life.
A conference called ATT&CKcon, began in 2018 to bring together cybersecurity leaders and practitioners to strengthen their capabilities and engage with others in the field indicating the broader acceptance and usage of the ATT&CK framework.
Today, the MITRE ATT&CK Framework is an ever-evolving repository of adversary techniques, updated regularly to reflect the latest threats and attack patterns observed in the wild. It has become an indispensable tool for red teams, blue teams, and threat hunters alike, enabling them to map their defensive and offensive strategies against a globally recognized standard.
MITRE ATT&CK Evaluations
Recognizing the value of the ATT&CK Framework, MITRE decided to take it a step further by conducting evaluations to assess how well commercial security products can detect and respond to the techniques documented in the framework. The first MITRE ATT&CK Evaluation took place in 2018 and focused on APT3, a well-known Chinese threat group also known as Gothic Panda. This evaluation was groundbreaking as it provided a standardized way to compare the capabilities of different cybersecurity products against a realistic adversary scenario.
Since then, the MITRE ATT&CK Evaluations have become an annual event, with each iteration focusing on different threat groups and scenarios. The list of participants has grown significantly, with vendors from all corners of the endpoint security (and endpoint-adjacent) industries eager to have their products evaluated. Each evaluation is a rigorous process, where vendors' solutions are tested against a predefined set of adversary techniques based on real-world threat intelligence.
By the numbers:
I’ve plotted the number of participants and a line of best fit. Clearly MITRE evaluations (were) popular, with an over 100% increase in participants over the course of two years. More on why we saw a drop in 2024 in a moment…
Why the Drop-off in 2024?
The drop in vendors participating in the 2024 MITRE ATT&CK evaluation can be attributed to a more specialized focus on managed EDR solutions targeting sophisticated threats like menuPass and ALPHV/BlackCat. This round emphasized automation, response orchestration, and seamless integration within enterprise environments—criteria that narrowed the playing field.
Key reasons for the vendor drop-off include:
Narrow focus on managed EDR and ransomware defense.
Emphasis on automation and enterprise integration, raising participation barriers.
Misalignment with certain vendors’ roadmaps, particularly those focused on other areas like cloud security.
Unlike previous evaluations that offered broader coverage, the 2024 round zeroed in on automation capabilities and the ability to counter ransomware and supply chain attacks. Consequently, many vendors chose to opt out, citing misalignment with their core product strategies.
The combination of these factors made this year’s evaluation more exclusive, leaving fewer players in the ring. It also indicates that many organizations are looking for managed EDR providers and a combination of threat-specific capabilities like ransomware defense, with the ease of use in the form of a managed, or co-managed, EDR.
Customer Sentiment and Feedback about the MITRE ATT&CK Framework
Customer sentiment around the MITRE ATT&CK Framework is overwhelmingly positive. It’s appreciated for providing transparency and fostering innovation across the security industry. Vendors also leverage their participation in these evaluations as a benchmark for credibility. However, while MITRE ATT&CK is widely respected, it’s not without its challenges.
Why It’s Useful and Why These Evaluations Can Be Tricky
MITRE ATT&CK Evaluations are undeniably beneficial, but they can be tricky. MITRE's decision not to include a standardized scoring system means vendors have significant latitude in interpreting and presenting results (more on this in Part II and III). This can lead to an oversaturation of data and marketing noise, where results are cherry-picked and selectively highlighted by vendors to paint a rosier picture of their performance.
This creates a lot of noise on social media and in marketing materials, where vendors position themselves as the best based on highly specific—and often incomplete—metrics. As a result, organizations must be cautious when evaluating products solely based on how they performed in the MITRE ATT&CK Evaluations.
So What’s the Catch?
While the MITRE ATT&CK Evaluations are an invaluable tool for assessing the capabilities of cybersecurity products, the absence of a scoring mechanism can create challenges in comparing products fairly.
Organizations must take the time to thoroughly analyze the evaluation reports and consider the broader context in which these products will be used. By doing so, they can make more informed decisions and ensure that they are choosing the right solutions to protect their environments.
In the end, it’s essential to remember that while the ATT&CK Evaluations provide valuable insights, they are not the be-all and end-all. Here’s why:
Real-world environments are complex, and adversaries are unpredictable.
A product’s performance in a controlled evaluation does not always reflect how it will perform in a live environment.
Organizations should use these evaluations as one part of a broader strategy, balancing them with their specific threat landscape and operational needs.
Conclusion
The MITRE ATT&CK Framework and its subsequent evaluations have revolutionized how we approach cybersecurity. They offer:
Benchmarking: A standardized, universally recognized framework that helps organizations map out their defenses and test their resilience against real-world adversaries.
Scope: The MITRE ATT&CK Evaluations have grown in both scope and participation, reflecting the increasing complexity of today’s threat landscape.
Benefits: By focusing on detection and visibility, the framework has enabled vendors and organizations to improve their security postures, providing a common language that bridges the gap between red and blue teams.
Check it out here: https://attack.mitre.org/
As the cybersecurity landscape continues to evolve, so too will the MITRE ATT&CK Framework. It will be fascinating to see how future evaluations adapt to the changing threat landscape and how vendors respond to the challenges posed by an ever-expanding list of adversary techniques.
One thing is certain: the MITRE ATT&CK Framework will remain a crucial tool in the arsenal of cybersecurity professionals for years to come.
How we use it and evaluate those who are assessed by it is even more important.
Be sure to check out next week’s post on “the bad” of ATT&CK. Stay secure and stay curious, my friends!
Damien
Note: this and subsequent posts are neither an endorsement nor deterrent of MITRE or any other security vendor mentioned in this blog. Views are my own.