No Rate Without a Denominator

Jane produced ten thousand assets last quarter. She had fifteen findings.

John produced ten assets. He had two findings.

Put those four numbers in a review meeting and watch what happens.

Somebody will notice that Jane has seven and a half times as many findings as John. If the organization stops there, Jane has a quality problem and John does not, and both of those conclusions are wrong.

Jane's rate is 1.5 findings per thousand assets.

John's rate is 200 per thousand.

That is a factor of one hundred and thirty in the opposite direction from the one the raw counts suggested.

This is the eighth article in this series, and it is about the single most common way quality numbers produce false conclusions.

Absolute Counts Punish Production

The failure above is not subtle, and organizations commit it constantly, because absolute counts are the numbers that are easiest to obtain.

A defect log naturally produces counts. Every finding has a name attached. Sorting by name and counting takes one query, and the result looks like information.

It is not. It is a measurement of exposure.

Anyone who produces more work will accumulate more findings, in the same way that anyone who drives more miles will accumulate more incidents. Treating the count as a quality signal creates an incentive that is precisely backwards: the safest way to have a clean record is to produce less.

In an agency, that incentive lands on exactly the wrong people. The person handling the high-volume social calendar, the developer shipping twenty template variations, the designer producing a hundred paid ad sizes: these are the people whose names appear most often in any defect log, and they are frequently the people the operation depends on.

An absolute defect count is a statement about workload wearing the costume of a statement about quality.

Rates Fix One Problem and Create Another

The obvious correction is to divide by volume, which is right, and which is where most teams stop.

But John's 200 per thousand is calculated from ten assets. It is not a rate in any meaningful sense. It is two events, expressed with three digits of unearned precision.

Consider what happens if John's next asset has a finding. His rate goes from 200 to roughly 273 per thousand. One event moved the number by seventy points.

Now do the same to Jane. One additional finding moves her from 1.50 to 1.51.

The two numbers are printed in the same column, in the same units, to the same number of decimal places, and one of them is a measurement while the other is a rumor.

This is the part that survives the "just use percentages" fix, and it is why the percentage often does more damage than the raw count did. A percentage looks authoritative. It carries no visible indication of how much data went into it.

Small Samples Do Not Have Rates

There is a threshold below which you should refuse to compute a rate at all, and the discipline of refusing is more valuable than any particular number you might pick for it.

The reasoning is intuitive without any statistics. If a person's output in a period is small enough that a single additional finding would meaningfully change their rate, then their rate is measuring luck.

Most creative work sits in this territory. A senior designer might produce twenty substantial deliverables in a quarter. A developer might ship eight. Comparing rates across people at that volume is arithmetic performed on noise.

Two practical responses, both better than computing the number anyway:

Widen the window until the denominator is adequate. A year of data for a low-volume role is more informative than a quarter, and the loss of timeliness is real but smaller than the loss of validity.

Or do not compute a rate. For low-volume, high-value work, individual findings are worth reading individually. Twelve deliverables and two findings is not a data set. It is two events that you can simply look at, which will tell you more than any ratio derived from them.

Choosing a Denominator That Means Something

Assume the volume problem is handled. There is still the question of what you are dividing by, and it turns out that "assets" is usually the wrong unit.

A thousand paid ad sizes generated from one master file is not a thousand independent opportunities to make a mistake. It is closer to one, replicated. Counting it as a thousand makes the person who built it look extraordinarily reliable.

Conversely, one bespoke landing page might contain forty independent facts (prices, dates, claims, links, names), each of which is a separate chance to be wrong. Counting it as one asset makes the person who built it look careless.

Better denominators exist, and the right one depends on what you are trying to learn.

Verifiable elements, when you want to measure care. Count the facts that had to be checked, not the files that were produced.

Boundary crossings, when you want to measure containment. Count how many times work moved to a client or to the public, since that is what an escape rate is actually a rate of.

Deliverables, when you want something crude that is at least consistent, and you are honest with everyone about what it is missing.

The choice matters less than making it deliberately and applying it uniformly. A denominator that was never chosen is still a denominator; it is just an accidental one.

Compare Like With Like, or Do Not Compare

Even a well-chosen denominator does not make people comparable.

Jane's ten thousand assets are probably repetitive, templated, and produced under a well-understood process. John's ten are probably bespoke, novel, and produced under conditions nobody has encountered before.

Those two kinds of work have different natural defect rates for reasons that have nothing to do with the people doing them. Novel work has more unexamined assumptions in it, which is not a character flaw; it is the definition of novel work.

Pooling them produces an average that describes nobody and a comparison that penalizes whoever draws the harder assignments. Over a few quarters, a team will notice, and people will start declining the interesting work. That outcome takes about two review cycles to appear and years to undo.

When you cannot construct comparable groups, there is a fallback that works and is chronically underused:

Compare a person against their own history rather than against each other.

A designer whose rate on comparable work has doubled is a signal, and it is a signal that is immune to almost all of the problems above, because the denominator, the work type, and the person are all held constant.

The Baseline Is the Team, Not Zero

A rate also means nothing without something to compare it to, and the reference point most organizations reach for is either zero or the best performer.

Both are wrong.

Zero is not achievable, and the entire premise of this series is that designing around it produces fragile processes. A team measured against zero learns to report fewer findings, which is the failure documented in Stop Counting Mistakes. Start Measuring Escapes.

The best performer is worse, because it converts your most valuable person into everyone else's problem. That deserves its own treatment, and it gets it in the next article, Your Best Performer Should Be a Laboratory, Not a Weapon.

The usable baseline is the team's own rate on the same class of work. Is 1.5 per thousand good? Only in relation to something. If the team runs at 1.4, it is normal variation and there is nothing to discuss. If the team runs at 0.3, it is a signal worth investigating, and investigating is still not the same as concluding.

Where the Numbers Are Actually Safe

Everything in this article is a warning about applying rates to individuals. None of it applies with the same force to processes.

A defect class pooled across the entire team, over a long window, with a consistent denominator, produces genuinely reliable numbers. Pricing errors per campaign. Broken links per launch. Layout failures per template. Those denominators are large, the comparisons are like-for-like, and the conclusions point at a control rather than a person.

This is the same asymmetry that ran through When Does a Mistake Become a Pattern?: the statistics that are too weak to judge an individual are perfectly adequate to diagnose a process.

Which is convenient, because the process is the thing you can actually fix.

The Rule

The whole article compresses into one refusal, and it is worth being pedantic about, because the number will be produced anyway if you are not.

No rate without a denominator. No comparison without a baseline. No conclusion without a sample.

Every one of those three is routinely skipped. Skipping the first produces a workload metric. Skipping the second produces a number nobody can interpret. Skipping the third produces confident statements about people, derived from noise, that are extremely difficult to walk back once said out loud in a room.