Don't ask the model if the agent did its job

We operate an offensive agent against real targets across multiple industries in Latin America: banking, fintech, marketplaces, telecom, insurance, retail, and others. Since we started, we've executed over 250 Threat Emulations across more than 100 real targets and analyzed every single execution one by one.

That process led us to a conclusion that turned out to be more important than we expected: evaluating an offensive agent isn't just about measuring what vulnerabilities it finds. The genuinely hard problem is understanding when it fails, where it fails, and why.

This article is about what we learned trying to answer that question, the error patterns we found, how we started measuring them, and the decisions we made along the way to build an agent that's increasingly reliable.

The problem that appears when the agent already works

Getting an offensive agent to run is no longer the hard part. The hard part comes after: it delivers a report, and understanding what went wrong is far more expensive than it looks. Take the most common phrase of all: "no findings." It can mean five completely different things.

The target was down during execution. A WAF blocked the scanner in the first minute. The credentials we provided didn't work. The container didn't have the binary the agent itself was invoking. Or the agent genuinely tested and there was nothing there.

Five causes, five different owners, and a single phrase in the report. Four of them aren't the agent's fault: three get fixed by managing access with the client and one with a Dockerfile.

Confusing them has two costs, and one is far more serious than the other. Internally, you ask the team developing the agent to improve something that worked fine, while the real problem stays untouched. Externally, the cost is of a different magnitude entirely: telling a client "the agent found nothing" when the truth is "the agent never got in." That company stays calm. They close the report, archive the empty finding, and keep operating on a surface nobody ever tested, convinced it was tested. It's not an incomplete report. It's a false sense of security, which is worse than having done nothing, because it displaces the decision to invest in reviewing it.

For the Strike team this is precisely what's at stake. We don't want to sell an agent that runs, returns "no findings," and that's where the service ends. Today anyone can choose from dozens of offensive agents, one open source, another that costs ten dollars a month, and all of them will deliver a report. The question very few answer well is the only one that matters when that report comes back empty: how do you know there was genuinely nothing there?

That's where we put the work, and that's why the question that organizes us isn't "did the agent find something?" It's this: would re-executing recover value, and who needs to fix what?

The hypothesis we had to discard: asking an LLM for the analysis

It's the approach any team would probably reach first: take the execution log, pass it to a model, and ask for the diagnosis. We got there too, and it was the most reasonable hypothesis, so we evaluated it seriously before building anything else.

We didn't discard it by intuition. We instrumented it, ran it across our corpus, and contrasted its conclusions against the agent's actual telemetry. That's what lets us explain with numbers why it's not enough, and there are three distinct reasons, not one.

1. The log doesn't fit, and what fits isn't all the agent's work. A model has a limit on how much text it can receive at once. Our logs have a median of 6,024 lines and the largest reaches 73,428, so they almost never fit: they have to be trimmed before being passed in. Measured across the entire corpus, out of 624 MB of logs the model gets to see 46 MB. That's 7.4%. Less than one in fifteen logs enters complete, and in the largest ones the model reads around 1%.

The problem isn't only how much is lost, but that you don't know which piece it read. And since the trimming depends on file size, it's not the same piece across two different executions either.

Worse: a good portion of what does fit isn't the agent's work. We measured that 24.8% of the corpus is garbage the agent copied from the client's site, compressed JavaScript and HTML that it captured and dumped into the log for no reason. In our largest log, at 19.8 MB, four lines alone account for 14.9 MB. The model charges to read that. And the coverage metric was corrupted in an almost comical way: "we read 100% of the log" improved the more the agent polluted its own log.

2. It guesses what's already measured. This was the finding that changed our design the most. We were asking the model to judge three things: whether the execution was efficient, whether it wasted tokens, and whether it got stuck without advancing. Then we compared each of its verdicts against the exact telemetry of that same execution.

The model concluded "efficiency: medium" about the most expensive execution in the entire dataset: $116, 2 hours, 345 tools. Another one that cost $79 it rated as "good." It said "wasted tokens" about a $0.93 execution. On the $79 one it said there was no waste. It said "got stuck without advancing" about an execution that finished normally. And the only one that genuinely hit the step ceiling didn't get that label.

All three verdicts weren't simply wrong: they were reversed. And the pattern repeated across the full set of qualitative labels the model produced: 8 of 9 didn't separate anything, and 3 pointed in the opposite direction, the execution marked with the defect measured better than the one that didn't have it.

But the important thing isn't that the model got it wrong. It's why it had no business being there in the first place. The agent already writes those numbers itself, exact, in its own log. How much it cost, how long it took, how many steps it ran, how many tools it invoked: it records them as it works, in a format a machine reads without ambiguity. There's nothing to interpret. They're there.

And what we were doing was this: deleting those lines so the log would fit in the model's budget, then paying a dollar and twenty minutes per execution for the model to estimate those same numbers by reading whatever was left. We were throwing away the exact data and buying a worse approximation.

When we saw it written like that, the decision became obvious: you don't ask a model for that. You read it.

3. The correct citation with the wrong conclusion. We audited a complete reading, line by line. On honesty, impeccable: 11 of 11 citations existed literally in the log, zero fabricated. And even so, in 2 of 7 cases the diagnosis was wrong.

That difference is the one that cost us the most to see: that a model doesn't invent the citation doesn't mean it understands what it cited. And the diagnosis is not a detail, because the action flows from it. If it says "the execution came back empty," the right thing to do is re-run it. If it says "a WAF blocked the scanner," the right thing to do is ask the client to whitelist us. With the wrong diagnosis, the system recommends re-running against a WAF that's going to block again: the exact opposite of the correct advice.

The extreme case: in a Threat Emulation against a bank, the reading declared it critical that an antibot had blocked the login, citing a reCAPTCHA it found at the beginning of the log. Since the file doesn't fit at once, it reads in chunks, and three chunks later, the same file said: "the login succeeded via reCAPTCHA … now authenticated as user." The claim was contradicted by its own log. We tested it with two different models on the identical file, one twenty times more expensive, and neither of them connected the two ends: each one read its chunk and drew its conclusion. The defect belongs to the method, not the model: a system that reads in parts and never cross-checks one part against the others will assert things its own evidence refutes, with any model and at any price.

Separating measuring from narrating

The rule that ordered everything else: what's measurable gets measured on the complete log, without an LLM. The LLM only narrates over evidence that's already been extracted and cited.

Put another way: "don't hallucinate" stops being an instruction in the prompt and becomes a property of the architecture.

The deterministic layer runs 18 detectors across the complete log, with no LLM. Cost: zero, runs every time. Every finding cites the exact line and byte. If a detector can't cite, it doesn't report. Reproducible: two passes over the same log produce the same result.

The reading layer runs the LLM across the full log in partitions. Median cost is roughly $11 per Threat Emulation. Every citation gets verified against the log; if it doesn't appear literally, it's discarded. It's a discovery instrument, not a production one.

Three design details that turned out to be more important than they seemed.

Citation verification isn't decoration: it's what makes the paid layer usable. Every assertion from the model has to come with a fragment that appears literally in the partition it just read, and we verify it programmatically. Measured across accumulated readings: out of 759 assertions with citations, 297 carried a citation that doesn't appear in the text, 39%, and were discarded automatically. Half of those concentrate in the free vocabulary, when the model invents a new label instead of using one from the catalog. Without that filter, four in ten assertions would enter the database as if they were evidence.

The prompt receives facts, never verdicts. The reading layer receives the measured telemetry but never what a detector concluded. If the model knows what a detector concluded, anything it says that coincides stops being evidence and becomes an echo.

The free layer isn't the poor version of the paid one. This surprised us: we measured that the deterministic layer finds more defect classes than the LLM layer in 4 out of 10 executions. It's not a superset, each one sees things the other doesn't. There's a structural reason: many defects are properties of the complete log, and a single page can't evidence the number of pages in the book.

And when both layers combine well, the expensive finding gets paid for once. In 4 of 8 Threat Emulations across different clients, the LLM layer found the same line: the browser binary didn't exist in the image. Converted into a free detector and measured across the entire corpus, it was present in 66 of 180 executions. Our previous detector caught only 14, it was blind by construction because it anchored on shell errors and the browser gets invoked via library. Fifty-two executions with an image that no detector was seeing. The fix was one line in a Dockerfile.

When all browser-based instrumentation of an agent is dead, the conclusion isn't that the agent is bad: it's that it's blind. And that diagnosis, once purchased, now covers every new execution for free.

Categorizing: what broke, who fixes it

Knowing that "the agent failed" isn't useful for anything. What changes a decision is knowing which subsystem broke, because that determines who fixes it and with what.

The most useful distinction we found is between tool and environment. They look the same, in both cases "a tool didn't work," but they're not: the fix for "tool" is code; the fix for "environment" is a Dockerfile. Mixing them sends the team searching in the wrong place, and that's exactly what happened to us with a content discovery module for months.

Today we classify every defect across three orthogonal axes, what broke, who fixes it, and what action it enables, across nine subsystems and 32 classes: access, tool, observability, memory, coverage, harness, deliverable, environment, and provider. The payoff shows when you compare companies. Two clients with eight weak Threat Emulations each: in one the problem concentrates in deliverable, the agent worked but the report doesn't reflect it; in the other, in memory. Those are two different conversations with two different teams. Before, both received the same generic message.

What we can actually tell a client

Measuring this way also forces you to be explicit about the limits. And the limits are what enable you to say something verifiable.

"No findings" is not the same as "secure." There is no denominator: the total number of vulnerabilities a target has doesn't exist. That's why we don't claim coverage, but something smaller and verifiable: what surface was discovered and left unattacked. The vulnerabilities nobody saw aren't measurable, and presenting them as covered would be inventing.

That's why every Threat Emulation also carries a block with what cannot be concluded about it, and it never returns an empty list: an empty list would read as "there's nothing we don't know," which is never true.

With that in hand, when a bank or an ecommerce asks us "did you find anything?", the answer "no" stops being a bare word. Alongside that "no" we can put four things on the table: what surface was discovered and what part received real attack traffic, with the line and byte of the log where it's recorded; what went untested, declared rather than estimated, so it's a known gap and not an assumption; what couldn't be reached and why, a WAF that blocked the scanner, credentials that didn't work, a host that was down, so "we found nothing" never gets confused with "we couldn't get in"; and what cannot be asserted about that particular execution.

A "no findings" without that backing isn't a result. It's an absence of information in the format of a result.

And here's the uncomfortable point for our industry. Every week a new agent appears promising to cover your entire platform. The question that almost never gets answered isn't how much it finds, but how it knows what it left without looking at. We dedicate ourselves to measuring that second part, and to showing it even when the number doesn't favor us.

That includes auditing our own measurements. Our coverage detector once reported that the agent had left 92% of the surface unattacked in a Threat Emulation. When we checked it against the raw log, the real number was 5%: it wasn't counting as attack traffic what the agent sends from the browser, which was precisely its main tool. A suspiciously high coverage number turned out to be the signature of the false positive, not of the serious problem.

We don't claim to have the best analyzer in the world. What we maintain is more modest and, we believe, more useful: we prefer to deliver a smaller, verifiable number than a big one with no backing. An invented coverage percentage reassures everyone until the day of the incident.

That's why, for us, validation has to be hybrid by design. Not because the human is faster, they're not, but because of the division of what each one can assert: the machine measures what's measurable over complete evidence; the model narrates and discovers over evidence already cited; and a person decides what it means. That also defines how we're organized: the hacking team and an independent artificial intelligence team look at the same execution from different sides, because whoever builds the agent can't be the one who measures whether the agent did its job.

An observed data point beats a declared one. That's the principle of the project.

We wrote this because we suspect any team operating agents in production, offensive or otherwise, will hit similar walls: the log doesn't fit, you don't know what it read, it guesses what's already measured, and the correct citation can come with the wrong conclusion. At least in our case, the way out wasn't a better model. It was separating what gets measured from what gets narrated, and demanding from the platform itself the same proof it demands from the agent.

This observability model can probably still be improved, and it surely will as we keep accumulating executions and evidence. But there's something especially interesting about working with this kind of agent: nobody has the last word. There is no definitive recipe today for building, evaluating an offensive agent. We don't have it either. What we do have is something that, in practice, turns out to be equally valuable: hundreds of real executions, errors we were able to understand, hypotheses we had to discard, and decisions we had to make because the data didn't leave us any other option.

‍