Why Traditional Security Isn't Enough for AI
SolarWinds’ Spectre: Part II
Last week, we examined the SolarWinds breach, a wake-up call that exposed the fragility of modern software supply chains. But as AI becomes deeply embedded into security operations, critical infrastructure, and enterprise decision-making, the stakes are even higher.
The AI supply chain isn't just a continuation of software security challenges; it's an entirely new attack surface. Unlike traditional software, AI systems are built on vast, interdependent networks of third-party models, training datasets, and open-source components. If an attacker can compromise any link in this chain, whether data sources, model weights, or training pipelines, they can poison outputs at scale, manipulate enterprise decision-making, or bypass security controls entirely.
We aren't just dealing with hypothetical risks anymore. AI supply chain attacks are already happening. The question is: Are organizations prepared?
This week’s write up took me down a few rabbit holes, and leaves me with more questions than answers. It’s clear we’re able to adopt new capabilities that unleash step function improvements in efficiency, but are we really thinking about how implicit trust is opening backdoors to our organizations?
AI Supply Chain Attacks: Already Here, Already Exploited
The SolarWinds breach showed us how a single compromised vendor could be weaponized to impact thousands of organizations. Now, we're seeing similar patterns emerge in the AI ecosystem, but with potentially more insidious effects.
In December 2022, PyTorch, one of the most widely used machine learning frameworks, disclosed that their nightly package repository had been compromised. Attackers uploaded a malicious version of a component called "torchtriton" that exfiltrated sensitive data from developers' machines. Any organization running those builds had their SSH keys and credentials stolen without their knowledge. This incident demonstrated how deeply intertwined our AI development tools have become with our security posture.
More concerning are the subtler forms of attack. In 2023, researchers from Cornell Tech demonstrated a technique called "PoisonGPT," showing how language models could be compromised through a "sleeper agent" attack. By injecting a single altered example during fine-tuning, they could create a backdoor allowing an adversary to trigger harmful outputs on demand, while the model behaved normally in all other contexts. The most disturbing aspect? These poisoned models passed all standard quality and safety evaluations.
Then we have the most recent security debacles around DeepSeek in January/February 2025. Less of a corrupted AI model and more so a consequence of poor CI/CD hygiene leading to exposed sensitive information. For context, I’ve written about it here, and the TLDR is if we’re going to trust AI vendors, we’ve got to keep them more accountable.
Even the repositories we trust for AI components aren't immune. Security researchers have identified potential risks with untrusted models on open repositories like Hugging Face, highlighting the possibility that models could contain hidden backdoors or malicious components designed to activate only under specific circumstances.
These incidents reveal three critical gaps that organizations must address as they adopt AI at scale: training data integrity, third-party model trust, and inference manipulation detection. Without solving these challenges, we risk building enterprise decision systems on fundamentally compromised foundations.
The Open Source AI Security Challenge
The explosion of open source AI introduces unique security risks that traditional software supply chain controls weren't designed to handle. When we review code, we can examine it line by line to identify malicious functions. But neural network weights, the core of modern AI systems, are essentially black boxes of numeric values with no clear relationship to their functionality.
This opacity creates a perfect hiding place for attacks. A backdoor in a neural network might be just a subtle adjustment to a few weights among millions, virtually undetectable through conventional inspection. Combine this with the dependency chain complexity of modern AI systems, which may rely on hundreds of interdependent packages, and you have an attack surface that dwarfs traditional software vulnerabilities.
What makes this particularly challenging is that adversaries can potentially influence model behavior without direct access to your systems. By contributing subtly biased training data to public datasets or releasing pre-trained models with carefully crafted vulnerabilities, they can affect downstream applications built on these resources.
Forward-thinking organizations are addressing these challenges by implementing automated scanning of model weights against known malicious signatures and establishing approval processes for integrating open source models. The most security-conscious teams are creating isolation boundaries between open source AI components and sensitive enterprise data, ensuring that even if a model is compromised, its damage potential is limited.
Existing Regulations and Standards: Lagging Behind AI-Specific Threats
While the threats are evolving rapidly, our regulatory frameworks are struggling to keep pace. The current landscape is a fragmented patchwork of early-stage initiatives, none of which fully address the unique challenges of AI supply chain security.
The EU Cyber Resilience Act represents one of the more comprehensive attempts, enforcing "secure by design" principles for digital products, including AI systems. However, its provisions focus primarily on traditional software security practices and don't directly address model integrity or training pipeline security, the areas most vulnerable to AI-specific attacks.
In the United States, the NIST AI Risk Management Framework (AI RMF 1.0) provides valuable high-level guidance but lacks the enforcement mechanisms necessary to hold AI vendors accountable for supply chain security. Without compliance requirements, many organizations will prioritize speed-to-market over security.
Meanwhile, China's AI Governance Framework takes a more direct approach, requiring government vetting for models before public release, with specific security requirements for AI systems. While effective in theory, this level of centralized control introduces its own challenges and isn't easily adaptable to Western contexts.
The OWASP community has made important strides by developing the Top 10 for Large Language Model Applications, identifying critical risks like prompt injection attacks, model extraction, and supply chain poisoning. This framework provides a valuable starting point, but the industry still lacks standardized assessment methodologies and certification processes to verify compliance.
The fundamental, dareisay systemic issue is that we still lack an AI-specific equivalent of SOC 2, ISO 27001, or FedRAMP (for our government friends) to certify that AI supply chains are secure. Until governments mandate stronger security standards, organizations must self-regulate or risk becoming the next headline in a major breach disclosure. Challenge is, when we’re being heterogeneous in our approach, we’re going to get mixed results.
AI-Specific Threat Modeling: Adapting Our Security Thinking
Effective security begins with understanding the threat landscape, but traditional threat modeling approaches fall short when applied to AI systems. Security teams need to extend their thinking to encompass the unique vulnerabilities introduced by machine learning architectures.
Consider how the classic STRIDE threat model transforms in an AI context. Spoofing evolves from impersonating users to model impersonation through style transfer or output mimicry. Tampering moves beyond unauthorized data modification to include training data poisoning and weight manipulation. Information disclosure extends from database breaches to training data extraction and model memorization leakage.
Even denial of service attacks take on new dimensions in AI systems, where adversaries can exhaust computational resources through carefully crafted inputs designed to trigger maximum processing. And elevation of privilege? In an AI context, this might involve prompt engineering to bypass safeguards and restrictions, a technique that requires no traditional hacking skills.
Organizations at the forefront of AI security are conducting specialized threat modeling sessions that consider these unique vectors before deploying AI systems in production environments. These exercises bring together data scientists, security professionals, and business stakeholders to identify potential attack paths and mitigations specific to each AI implementation.
Building a Zero-Trust Approach to AI Security
As the threat landscape evolves, organizations need a structured approach to securing their AI supply chains. A zero-trust philosophy, never trust and always verify, provides the foundation for effective defense in depth.
The journey begins with comprehensive verification of every model and dependency. Security leaders familiar with Software Bills of Materials (SBOMs) must now develop Model Bills of Materials (MBOMs) that track where a model was trained, who contributed to its dataset, and how its weights have been modified. Google's Supply Chain Levels for Software Artifacts (SLSA) framework provides a foundation for supply chain security that organizations are beginning to adapt and extend to AI artifacts.
This verification process requires cryptographic signing for AI model weights to prevent tampering during training and deployment. Microsoft is pioneering this approach with signed AI artifacts in Azure AI, preventing malicious model injections through cryptographic validation. The implementation isn't particularly complex, a standard digital signature approach applied to model weights, but it provides a cryptographic guarantee of model provenance that traditional approaches can't match.
The security of training pipelines themselves is equally critical. Threat actors can inject subtle biases or hidden backdoors during model training, so organizations must treat these pipelines as critical infrastructure. This means implementing immutable training datasets with cryptographic integrity checks, deploying anomaly detection to flag unexpected dataset changes, and exploring federated learning with differential privacy to limit data exposure. IBM is at the forefront of this effort, developing AI red-teaming techniques to identify model poisoning attempts before deployment.
Perhaps most importantly, organizations must apply zero-trust principles to AI APIs and model outputs. AI models are not inherently trustworthy and must be validated at runtime just like any other system component. This means validating every AI-generated response before execution, limiting AI access to sensitive systems using role-based controls, and continuously monitoring AI behavior for deviations from expected patterns. Microsoft's Security Copilot implements this approach with real-time output validation to prevent erroneous or manipulated responses from triggering security incidents.
Beyond Prevention: Monitoring and Responding to AI Security Incidents
Even the most robust preventive measures will eventually face sophisticated attacks. Organizations need to develop AI-specific monitoring and incident response capabilities to detect and remediate security events when, not if, they occur.
Effective monitoring for AI systems requires a multi-layered approach:
Input monitoring: Tracks unusual patterns in prompts or inputs that might indicate injection attacks
Output analysis: Watches for sensitive data leakage in model responses
Resource utilization monitoring: Detects unusual computational patterns that might signal model poisoning or extraction attempts
These capabilities extend traditional security monitoring to address the unique characteristics of AI systems.
When incidents do occur, security teams need an AI-specific incident response framework. This begins with preparation, developing specialized playbooks for different types of AI security incidents. For example, if a language model suddenly generates inappropriate content, the response might include isolating the instance, capturing all recent inputs, reviewing training data for potential poisoning, and checking for prompt injection patterns.
The detection, containment, eradication, and recovery phases of traditional incident response all need AI-specific adaptations. How do you isolate a potentially compromised model without disrupting critical business functions? How do you determine whether an anomaly indicates an attack or simply a legitimate edge case? How do you restore service with confidence that the vulnerability has been addressed? These questions require cross-functional expertise from data science, security, and business teams.
Navigating the Global AI Landscape: Collaboration and Geopolitical Challenges
The global nature of AI development creates additional complexity for security teams. AI components may come from jurisdictions with different security standards, creating supply chain fragmentation that complicates verification efforts. Training data requirements may conflict with data localization laws, forcing difficult trade-offs between model quality and compliance. International collaboration, while essential for advancing AI capabilities, may inadvertently expose sensitive techniques to potential adversaries.
Forward-thinking organizations are addressing these challenges through technical controls that enforce data residency requirements while enabling collaboration. They're developing vendor assessment frameworks that consider geopolitical risks alongside technical capabilities. And they're creating contingency plans for disruptions in international AI supply chains, ensuring business continuity even if key partners or regions become unavailable.
The Path Forward: Accelerating AI Security Maturity
The AI supply chain is already under attack, but unlike traditional software, we don't yet have mature, enforceable security frameworks to protect it. Organizations can't afford to wait for regulation to catch up; they must take proactive steps to secure their AI systems today.
This means implementing MBOMs for tracking AI dependencies, applying cryptographic signing for model verification, extending zero-trust principles to AI APIs, and establishing continuous monitoring for AI model behavior. It means developing specialized threat models, incident response playbooks, and security architectures designed for the unique characteristics of machine learning systems.
My gut tells me it’s only a matter of time before we have an “AI-driven SolarWinds” breach. The potential impact of compromised AI systems extends beyond data breaches to manipulated decision-making, corrupted insights, and eroded trust in automation itself. The time to act is now, before attackers develop more sophisticated techniques for compromising these increasingly critical systems.
What's Next?
I wish I had a great answer for you. With new AI models, frameworks and capabilities rolled out, it’s a challenge to stay aligned with what’s happening, let alone secure it. So I turn this question back to you, dear reader. If your organization is deploying AI at scale, how are you securing your AI supply chain? What security frameworks are you following, and what gaps are you encountering? I'm particularly interested in hearing about practical approaches to model verification and training pipeline security.
Feel free to leave a comment or shoot me a message, I’d love to know.
Stay secure and stay curious my friends.
Damien