Metrics Can Destroy the Culture They Were Designed to Improve

The quality program starts well.

Findings get logged. Severity gets assessed. Escapes get tracked. Within a quarter there is a dashboard, and the dashboard shows something real: most escapes are happening at one boundary, a control gets added, and the number improves.

Then someone senior asks a reasonable question.

"Can we see this broken down by person?"

Everything after that is the subject of this article.

Because a quality framework is not a neutral instrument. It is a set of incentives, and it will be optimized against by intelligent adults who understand it perfectly well. The only question is whether they optimize the work or the number.

This is the tenth article in this series.

The Mechanism

The underlying dynamic is old and well documented outside of creative work: a measure that becomes a target stops being a good measure. What is specific here is how fast it happens and how quietly.

Quality metrics are unusually vulnerable to this because of a structural fact established earlier in the series, in Stop Counting Mistakes. Start Measuring Escapes.: the internal finding count is generated by the people being measured.

This is not true of most metrics. Revenue is not self-reported by the salesperson in a field they control. But a defect log is populated by the same team whose performance is being read out of it, which means it responds to pressure directly and immediately, without anyone having to do anything dishonest.

Nobody deletes findings. That is not how it fails.

It fails through a hundred small, individually defensible judgment calls.

Is that really a finding, or just a preference? Probably a preference; do not log it.

Should I raise this now, or fix it quietly and mention it if it comes up? It takes thirty seconds to fix. Fix it.

Is this Medium or Low? It never actually reached anyone. Low.

Is it worth logging something I caught in my own work? It never left my desk. No.

Each of those decisions is reasonable in isolation. Together they are the entire failure, and they take about two quarters.

What the Dashboard Looks Like While It Is Failing

The insidious part is that the numbers improve.

Internal findings drop, which reads as fewer defects. Severity distribution shifts downward, which reads as lower risk. Repeat findings decline, which reads as controls working.

Every trend line points the right way, and the program gets praised.

The escape rate does not improve, because escapes are discovered by people outside the process (clients, customers, the public) and cannot be judgment-called out of existence. But escapes are rare enough that a few quarters of flat escape numbers against improving internal numbers looks like normal variance rather than a contradiction.

By the time the contradiction is undeniable, the log has been unreliable for a year, and the historical data that would let anyone diagnose it is the same data that was corrupted.

This gives the single most useful warning sign a quality program has:

A falling internal finding count with no change in process is not an improvement. It is a symptom.

If findings drop, the first question should be what control was added or what work stopped being produced. If the answer is neither, the drop is a reporting change, and it should be treated as an incident in its own right.

The Four Ways It Gets Weaponized

Frameworks do not become punitive by intent. They become punitive through specific, recognizable moves.

Per-person defect counts. The most common, and the one that starts most of the damage. The previous articles have already established why the number is invalid. It mixes caught with escaped, it has no denominator, it has no severity weighting. What matters here is what it teaches: that appearing in the log is bad. Once learned, that lesson is very hard to unlearn.

Severity used as a character judgment. "John had two Criticals this quarter" reads as an assessment of John. But severity describes potential impact, which is largely a function of what work someone was assigned. Put a person on the biggest client's paid campaigns and they will accumulate high-severity findings by definition. That is a staffing fact being read as a competence fact.

Findings entering performance reviews. Once a defect log is a review input, it stops being a defect log. It becomes a record people manage, and every honest entry becomes a small voluntary cost. Reasonable people do not pay voluntary costs indefinitely.

Targets on the internal number. "Reduce defects by 20%" is the purest version, because it can be achieved perfectly, immediately, and without touching the work at all. There is precisely one reliable lever and everyone can find it.

Safeguards That Actually Hold

The safeguards that survive contact with a real organization are structural. They are about who can see what, not about how the numbers are described.

Never publish per-person defect counts. Not in dashboards, not in reviews, not in a channel where the numbers can be diffed. If a name is needed for a specific investigation, that investigation is a conversation, not a column.

Aggregate by defect class first. When Does a Mistake Become a Pattern? argued that patterns sorted by class point at processes while patterns sorted by name point at workloads. Building the reporting that way is the most effective structural protection available, because it makes the punitive view require deliberate effort rather than a default sort order.

Keep severity and detection point in separate fields, always. A Critical finding caught internally and a Critical finding found by a customer are opposite events. Collapsing them into one number is what makes "Critical" sound like an accusation, and it is the change that most reliably starts the underreporting.

Set an aggregation floor. No individual-level number is reported below a sample size where it would be meaningless, which as No Rate Without a Denominator showed, is most of them.

Separate the quality log from the performance system, explicitly and in writing. Not as a promise of goodwill, but as a rule about data access that outlives whoever made the promise. The person who introduced the framework will not always be the person reading it.

Report Prevented Impact alongside Escape Rate. A quality program that only ever produces bad news creates constant pressure to produce less news. Prevented Impact is the same data, expressed as an output, and it is the only number on the dashboard that a team has an incentive to make larger honestly.

The Test

There is one question that reveals the state of a quality culture faster than any metric, and it can be asked in a corridor.

What happens, socially, in the ten minutes after someone says "I found something"?

If the response is a version of "good catch," if the finder gets credit and the conversation moves immediately to the control, the framework is working, whatever the dashboard says.

If the first response is "whose was it?", the framework is already inverted. It may be producing beautiful numbers. It is producing them by suppressing exactly the information it was built to collect.

Nothing in the metric definitions fixes this. It is set by what senior people do in that ten-minute window, repeatedly, in public, and it can be undone by one reaction in one meeting.

The Sentence

The whole risk compresses to this:

Accountability without safeguards eventually becomes blame with better metrics.

The framework does not prevent that. It accelerates it, because it supplies precision and a vocabulary and a defensible-looking record. A punitive culture with a good quality framework is more damaging than a punitive culture without one, since the numbers give the punishment an appearance of objectivity that it has not earned.

This is not an argument against measuring. The alternative (no measurement, no vocabulary, "just a typo," everything by feel) is what the whole series has been arguing against, and it fails in its own ways.

It is an argument that the measurement system has to be designed with its own failure mode in mind. Which is, appropriately enough, the thesis the series opened with: assume error, including in the thing you built to catch errors.

The framework will be misused. Design for that in advance, or discover it in a quarter where the numbers look excellent and nobody can work out why the client is unhappy.