A Test in Time

What two decades of breaking in taught Jaime Rodriguez about the assumptions that quietly get you owned

In this edition of Signal to Noise, I sat down with Jaime Rodriguez, a security leader who runs threat intelligence, red teaming, and detection engineering at Sutter Health, with a growing focus on AI red teaming. Jaime spent the first half of his career breaking into things, buildings included, as a pentester and red teamer. The second half he has spent building the programs meant to stop people exactly like his younger self.

That arc is the whole reason this conversation was worth having. Most people who can find the vulnerability never learn to run the program. Most people who run the program have never felt what it is to be on the other side of the door. Jaime has lived in both seats, and it shows in how he talks about risk.

We covered what signal actually means when the data never stops, why the biggest organizations are not the safest ones, how the attacker keeps migrating to whatever you trust most, and the one thing he still tells every team to dig into first.

From Breaking In to Building the Thing That Stops It

Jaime has been doing some form of security for nearly two decades. The early years were offensive, and he is plain about what that taught him. Finding the vulnerability is the easy part of the job. As he put it, building a program that actually reduces risk systematically is a whole other animal.

That gap is the part of security nobody likes to talk about. We love big-minded programs and we love deep technical detail. The connective tissue between the two is where almost everyone struggles. Jaime spends his days inside that tissue, and his answers all come back to it.

Drowning in Data, and Now AI Slop

When I asked him what makes signal so hard, he did not reach for false positives. He went straight to volume.

Most teams are drowning in data, and he listed the flood without flinching: alerts, dashboards, vulnerability findings, threat reports, and now what he flatly calls AI slop. Some of what the models produce is genuinely useful. A lot of it is just more summary to wade through on the way to a decision.

So the job stops being collection and becomes triage of meaning. Out of all of this, what actually requires action from me or my team? His example landed it cleanly. A feed that tells him a ransomware group exists, with nothing else attached, is just another data point. A feed that tells him the group is actively hitting healthcare, using a specific technique his stack does not currently detect, is a high-fidelity signal he can act on. Same category of input. Completely different worth. Signal is the part you can do something about.

The Attacker Always Moves to Whatever You Trust

I wanted to understand how we got so uneven, so good at some signals and so blind to others. Jaime answered it from the attacker’s chair, which is the only honest way to answer it.

We got good at the obvious web flaws first. Cross-site scripting, SQL injection, the things you could not ignore because they were everywhere. So the attacker mindset adjusted. Once defenders stood up WAFs and firewalls, the soft target moved inward, and identity became the prize. Everyone rolled out MFA and 2FA, so attackers went hunting for the holes in how those controls were actually implemented.

Then the target moved again, to the place it always eventually moves. Humans are the weakest factor in any security landscape, and that has not changed. What changed is the tooling around it. Jaime pointed out that you can now social engineer an LLM the same way you social engineer a person, prompting it into handing over things it should not. The trust boundary keeps relocating, and the attacker keeps following it. You harden one layer, and the soft spot just moves one layer over.

Trust, but Verify, Times Four

This is where I told the story of my mom. She is a 71-year-old retired lawyer who will happily hand ChatGPT everything she has. My standing advice to her is to get the pro license and to never, ever feed it her W-2. Some people extend that kind of implicit trust to these systems, on both the input and the output. Others stay deeply skeptical. I asked Jaime how he has watched that trust boundary shift as agentic systems arrive.

He said the unpopular thing, which is that it depends entirely on the person. A lot of people have quietly offloaded their thinking to the model. They trust what they put in, they trust what comes back, and it is the old “I read it on the internet, so it must be true” reflex wearing a new coat.

Then there is the other camp, the one Jaime lives in. Guarded about what they feed the system, more willing to question the output, sometimes running the same question past three or four different models before they believe any of it. Trust, but verify, is doing a lot of work in that sentence. The hard part is doing it while the content keeps coming at full speed.

What’s Old Is New Again

So how do you verify when the volume never lets up? Jaime’s answer was almost stubbornly unglamorous. It comes down to experience and community, to what you have personally seen go wrong before.

He keeps watching the same vulnerabilities from the year 2000 reappear in new lighting. Agent-to-agent systems are surfacing failure modes we have technically seen for a quarter century. The eyes are new. The underlying mistakes are not. His through-line for any team he builds is the one nobody wants to hear, because everyone wants the latest and greatest tool instead.

What are we doing with the basics? Do we still understand how to live off the land? Do we understand how any of this works without the shiny thing in the rack? That fluency, he argues, is what fires the spidey sense that says I don’t think that’s right. And the discipline he prizes most is being skeptical of your own assumptions, not just everyone else’s.

Assumption Is the Vulnerability

That set up the part of the conversation I keep thinking about. Jaime has worked across plenty of tech-forward companies, so I asked which assumptions about big enterprises turn out to be flat wrong.

His first answer was a single word: assumption itself. We see a large organization with a strong reputation and we quietly conclude its security must be excellent and its teams must be well staffed. Sometimes that could not be further from the truth. Big environments are often under-resourced precisely because they are so sprawling and complex. Layer on the acquisitions, legacy systems, and organizational friction that come with scale, and visibility at a billion-dollar company can be every bit as broken as it is at a startup.

It can be great, and often it genuinely is, but it carries every problem a small business has, just at scale. We also assume environment size tracks company size, and that is frequently wrong too. The same fundamental question haunts everyone. What do we actually own? In twenty years, Jaime has never once seen perfect asset inventory, not at a giant and not at a small shop, nobody who could honestly claim full command of everything they run and every path their data takes. Big does not mean covered.

A Test Is a Point in Time

The human and process side gets worse the same way, through assumption compounding quietly over time. We assume our controls do what they are supposed to do. We assume we understand our identity risk because tools are feeding us telemetry. We assume we are detecting what matters and that nothing has shifted lately.

Then Jaime made the point that I think every security leader should tape to their monitor. We hire a third party, or we run our internal red team, and we treat the result as a verdict. But that engagement is a test in time, true only for the moment it ran. Five or ten minutes after the report closes, someone ships an app full of holes, or an automated firewall rule swings a door open nobody noticed. We put all our eggs in the basket of this passed, so we are good to go. The pentest expired the moment it finished.

Selling Continuous to Leadership

That is the case for continuous security, and I wanted his read on the hard part, which is paying for it. Most people assume the fight is convincing executives that continuous beats point-in-time. Jaime says that part is usually intuitive. Of course checking once and trusting it a year later feels wrong.

The real argument he makes to leadership is about posture, not philosophy. With a continuous program he can say, with a straight face, that at least as of today the controls are what they think they are and the attack paths are clear. He can report what changed daily and weekly, and name the emergent threat of the week. Most of all, he can get ahead of the inevitable question. A board member reads something on Google News, or a CISO friend mentions some new thing at dinner, and the answer is already yep, we ran the automation, the hunt and the simulation are going, you will have the report within the hour. That is the difference between proactive and reactive, and it is the whole sell.

Red, Blue, and the End of the Handoff

The mechanics underneath that are where Jaime’s two careers finally fuse. In most large companies, the red team runs on its own, threat intel runs on its own, and detection runs on its own, and they barely talk. He has the rare ability at Sutter to run all three under one umbrella, feeding each other on a daily basis, though he was honest that even with weekly syncs people tune out under their own deadlines.

The flow he is building goes like this. Threat intel feeds the red team, the red team simulates and exposes the gaps, and detection plugs them. He runs a breach-and-attack-simulation platform that maps to MITRE, fires in a controlled part of the environment, and reports coverage. The catch is the same catch as always. That coverage holds only until the tactics change, and tactics die fast. Burn the DNS, move to another zone, swap the records and the hashes, and your clean report is suddenly a snapshot of a campaign that no longer exists.

So he is extending the same loop to threat hunting, to the gaps that never throw an alert because you do not yet know they are there. What he described next genuinely changed how I think about the red and blue relationship. For years it ran one direction. Red team tells blue team, here is every assumption you got wrong and every way we walked through. Hunting runs the inverse. It tells the red team, yes, you found that, but look at everything you walked right past. The two functions stop scoring points off each other and start leaning on each other, continuously. He calls it purple teaming, but says what it really comes down to is continuous defense in depth.

If You Do One Thing: Follow the Identity

I closed the way I like to close. Suppose I am sizing up my program today, on-prem and cloud, the usual stack of CrowdStrike, Okta, and CloudTrail. What is the one thing I should be digging into right now?

He waved off the obvious answer of logs and went straight to identity. Can you actually tell where these controls are consistently catching and stopping attacks, versus where they are merely supposed to be? Can a single compromised account move laterally through every one of those systems? That blind spot, he says, is the gap that matters. Identity is the path now, the route to everything and through everything in the day-to-day, and a real handle on your service accounts and user accounts is the biggest risk he still sees unaddressed.

Finding Signal

While there are plenty of takeaways, there was a clear sticking point from my discussion with Jamie which is how much of security failure traces back to one quiet habit…our assumptions.

We assume the big company is covered. We assume the passing test stays true. We assume the control is doing its job and the model is telling us the truth. Every one of those assumptions is a door someone is paid to find.

The antidote running through everything he said is continuous, skeptical validation, applied first to your own conclusions. Signal is not the report that says you passed. Signal is knowing, today, that you still are.

Stay secure, and stay curious, my friends.

Damien