Stop Counting Mistakes. Start Measuring Escapes.

Two teams. One quarter. Here are their numbers.

Team A found 100 defects internally. Five reached the client.

Team B found 10 defects internally. Six reached the client.

Which team has the better quality process?

Most people, shown these numbers in a meeting, react to the 100 first. A hundred defects sounds alarming. Ten sounds tidy. Somebody will say that Team A clearly has a problem.

Team A caught 95% of its known defects before they escaped.

Team B caught 63%.

Team B also let more defects reach the client in absolute terms, from a tenth of the internal findings, which tells you something uncomfortable: Team B is almost certainly not producing fewer defects. It is finding fewer of them.

This is the seventh article in this series, and it is about replacing the number most organizations track with the two that actually describe their quality process.

Counting Defects Measures Effort, Not Quality

The count of defects created is close to useless as a quality metric, for a reason that has nothing to do with culture.

It conflates two completely different populations: defects that were stopped, and defects that were not. Those are opposite outcomes. Adding them together produces a number that goes up when your process is working well and also goes up when your process is working badly, which is the definition of a metric that cannot be acted on.

Worse, it is trivially gameable in the one direction you do not want. There is exactly one reliable way for a team to reduce its defect count without changing anything about the work: look less carefully.

That is not a hypothetical failure mode. It is the default one. A team told that its defect count is too high will reduce it, and the fastest available lever is the thoroughness of its own review.

Assume Error, the first article in this series, put the underlying point in one line: creating a defect and allowing a defect to escape are not the same event. Everything here follows from taking that seriously enough to change what gets recorded.

The Two Numbers

Split the population, and two metrics fall out immediately.

Internal Catch Rate. Of all known defects, what proportion was caught before crossing to the client?

Escape Rate. Of all known defects, what proportion was not?

They sum to one, so in a narrow sense they are the same metric. They are worth stating separately because they answer different questions and get used by different people. Internal Catch Rate is the number the team should watch. Escape Rate is the number the client's experience is actually made of.

For the two teams above:

The reframing this produces is the entire point:

A high internal finding count is not evidence of a quality problem. It is evidence of visibility.

The number that should alarm you is not 100. It is 63%.

Not All Escapes Are Equal

Escape Rate treats every escape identically, and the previous articles in this series have already established that this is wrong twice over.

Not Every Typo Is Low Severity established that defects differ enormously in potential impact. The Trust Chain established that where a defect was stopped determines who absorbs it.

So the single most useful metric in the set is a narrow one:

Critical Escape Rate. Of the defects that could have caused serious harm, how many crossed a boundary?

This is the number that should be reviewed most often, because it is the only one that maps directly to the thing the organization is actually trying to prevent. A team can run a 15% Escape Rate quite comfortably if every escape is a spacing inconsistency on an internal deck. The same team is in trouble at a 2% Escape Rate if the escapes are prices and dates in live campaigns.

Averaging severity into a single quality percentage destroys exactly this information. Keep them separate. A dashboard with an Escape Rate and a Critical Escape Rate side by side tells you something. One with a blended "quality score" tells you nothing you can act on.

Where, Not Just Whether

Escape Rate answers whether. It does not answer where, and where is what you fix.

Record, for every finding, the last boundary it crossed before someone stopped it: the creator, internal review, the client, the client's customer or the public, and, separately, whether it was amplified beyond its original audience. The distribution of that field over a quarter is the most directly actionable artifact a quality process produces, because a cluster at one boundary names the missing control precisely.

Escapes concentrated at the client boundary mean there is no verification step immediately before delivery.

Escapes concentrated at the public boundary mean the gap is between approval and publication: in practice, almost always a scheduling tool, a deploy step, or a person copying content from one system to another.

That is a small, specific, fixable problem. "Improve quality" is not.

The formal version of this field, with defined levels, arrives later in the series in From QA Checklist to Creative Risk Management. You do not need the formalism to start. You need one column.

Prevented Impact

There is one more metric, and it is the one that changes how people behave.

Prevented Impact. How many Medium, High and Critical findings were contained internally?

It is the same underlying data as Internal Catch Rate, expressed as an output instead of a ratio. That sounds like a cosmetic difference. It is not, because a ratio is something a team is measured against and an output is something a team produces.

"Our internal catch rate was 95%" is a compliance statement.

"We prevented eleven High and two Critical defects from reaching clients this quarter" is a result, and it is the one worth reading aloud in a meeting where the account leads are present.

This metric also survives a translation that the others do not: it is the only one on this list that a client can be shown. It is the whole argument for a QA function, expressed in the only terms that a person paying for it will find persuasive.

Whether a Critical finding caught internally should be celebrated rather than investigated is a large enough question that it gets its own article: A Critical Defect Caught Internally Is a Win.

The Denominator Problem

Everything above is a ratio, and every ratio has a denominator that has quietly been doing a lot of work.

"Of all known defects" is a denominator you cannot observe. You only ever know about the defects somebody found. A team that reviews poorly has a smaller denominator and can post a respectable Internal Catch Rate while shipping more broken work than the team next to it.

This is the same structural problem as Team B, one level up, and it means Escape Rate cannot be compared between teams without also comparing how hard each one is looking.

There is a second denominator problem underneath it, which is volume: a team producing ten thousand assets and a team producing ten cannot be compared on rates at all, and neither can two people. That problem is large enough to be the next article in this series: No Rate Without a Denominator.

For now, one practical guard is enough. Escape counts, unlike escape rates, are hard to game downward by looking less carefully, because an escape is discovered by someone outside your process whether you were looking or not.

Track escape rates to understand your process. Track escape counts to keep yourself honest about it.

What to Stop Reporting

If you adopt nothing else from this article, adopt a deletion.

Stop reporting defects per person.

That number has no legitimate use. It is not a quality measure, because it counts caught and escaped defects together. It is not a performance measure, because it has no denominator and no severity weighting. And it has one extremely reliable effect, which is to teach everyone in the organization that being associated with a finding is bad.

Once that lesson lands, findings stop being reported, the internal count drops, the dashboard improves, and the escape rate quietly rises, six to twelve months later, in a quarter where nobody connects the two events.

That mechanism is the subject of Metrics Can Destroy the Culture They Were Designed to Improve, later in this series. It is the most common way quality programs fail, and it always begins with a metric that was introduced for entirely reasonable reasons.

The Reframe

The shift this article is arguing for is small in implementation and large in consequence.

Stop asking how many mistakes the team made.

Start asking how many got out, which ones, and where they crossed.

The first question makes people careful about being seen. The second makes them careful about the work.

And a team that is genuinely good at this will look worse on paper than one that is not, right up until the moment a client notices something first.

That is the trade the whole series is built on. It is worth making explicitly, in advance, with the people who read the dashboard.