A Critical Defect Caught Internally Is a Win

An hour before a campaign goes out, someone on the review pass notices that the price in the hero is wrong.

Not slightly wrong. Wrong in the direction that would have had customers arriving at a checkout expecting a different number, across a paid campaign, under the client's name.

The finding gets scored. Materiality 2, Trust 2, Reach 2, Reversibility 1, Business Consequence 2. CQIS 9: Critical.

Now watch what the organization does in the next ten minutes.

In most agencies, the room goes tense. Somebody asks how this got so far. Somebody else notes that it was nearly out the door. The person who built the asset explains where the price came from. The word "Critical" sits in the middle of the conversation doing enormous damage.

Every part of that reaction is aimed at the wrong event.

This is the thirteenth article in this series.

Two Things Happened, and Only One of Them Is Interesting

Separate the events.

Event one: a defect existed. Somebody had a wrong price in a source document, or transposed two digits, or worked from an outdated deck. This event is unremarkable. It is the base rate of human and automated work, and it is the premise the entire series started from in Assume Error.

Event two: a control fired and stopped it. An hour before publication, a verification step that the organization designed, staffed and paid for did precisely what it was built to do, against a defect that would have been expensive.

The second event is the news. It is the only part of the sequence that the organization can take any credit for, and it is the part that gets no attention at all in the reaction described above.

A Critical score attached to a finding caught at E1 is not a report of a near-disaster. It is a measurement of how much the control was worth.

The Score Was Never the Verdict

The reason the room reacts badly is that "Critical" carries fifteen years of connotation from contexts where it did mean something had gone wrong.

Under CQIS it means something narrower, and the distinction was built into the framework deliberately, as From QA Checklist to Creative Risk Management set out.

Severity describes potential impact.

Escape level describes what actually happened.

A CQIS 9 at E1 reads, in full: this could have been very expensive, and our process contained it before anyone outside the team saw it.

That is a sentence about a system working. Read only the first half and it becomes a sentence about a person failing, which is a different sentence, about a different subject, that the data does not support.

This is why the two fields must never be merged into a single quality number. Merged, the framework produces alarm in proportion to how well it is working.

Near Misses Are the Cheapest Data You Will Ever Get

Industries where errors kill people worked this out decades ago, and their conclusion is stronger than the one most creative teams have reached.

In aviation and in clinical medicine, the events that generate the most process improvement are not the accidents. They are the near misses: the incident that was caught, the wrong medication that was intercepted, the approach that was aborted. Near misses are vastly more numerous than accidents, they carry nearly the same information about how the system fails, and they cost nothing.

The entire value of that data depends on one condition: people have to report them voluntarily, including when they were the person involved.

Which is why those industries invested so heavily in making reporting safe, and why they treat a drop in near-miss reports as a warning rather than an achievement.

A Critical finding caught at E1 is a near miss. It is the highest-information, lowest-cost event a creative quality process can produce, and it exists only because somebody looked hard and then said so out loud.

An organization that reacts to that by looking for who to talk to is spending down the exact resource it most needs.

Adversarial Without Dread

There is an obvious objection to everything above, and it is the one that decides whether any of this survives in a real office.

If a single typo can cost an account, as Not Every Typo Is Low Severity argued it can, and if the team is now trained to hunt for exactly that, have you not just built a machine for producing anxiety? A room where everyone is permanently rehearsing the worst version of their own work?

That outcome is real and it is common. It is also avoidable, and the reasons are structural rather than motivational.

A severity score describes something that did not happen. This is the part that gets lost. A CQIS 9 at E1 is a measurement of a world that was avoided. Nothing is currently wrong. Nobody is currently exposed. The number is the size of the thing that is not happening, which is close to the opposite of an emergency, and reading it as one is a category error the framework itself does not commit.

Ranking is what lets you stop worrying about most things. This is the counterintuitive part, and it is worth stating plainly: the absence of a severity model is what produces free-floating dread, not its presence. When every finding is described with an adjective, nothing is bounded. Any of them might be the one that ends an account, and the only rational response is to treat all of them as potentially fatal, forever. A scale ends that. It says: these four are Low, they were contained, you can genuinely stop thinking about them; this one is a High on a paid campaign for a client in a tense quarter, and it deserves the attention you were previously spreading uniformly across everything.

Anxiety is what you get when risk is real but unranked.

Being adversarial is only frightening if you can be blamed for what you find. Hunting for defects in your own work is not intrinsically stressful. What makes it stressful is knowing that a successful hunt produces a record with your name on it. Remove that, as Metrics Can Destroy the Culture They Were Designed to Improve argues you structurally must, and adversarial review becomes the safest activity available, because it is the one where finding something is the win condition.

The opponent has to be named, and it must not be a person. The team competes against the escaped defect. That is not a slogan; it is a design requirement. A team with no named opponent will default to competing with each other, because that is what scoreboards do when nobody specifies what they are for.

Put those together and you get a room that is genuinely hostile toward the work and genuinely relaxed about the people in it. Those two states are not in tension. They only look like they are when the framework has failed to separate impact from attribution.

What Recognition Actually Looks Like

"Celebrate catches" is the kind of advice that produces a Slack emoji and no behavior change. The mechanics matter more than the sentiment.

Record the outcome, not the attribution. The log entry is Critical Impact Prevented (E1). It is not Jane found John's mistake. The finding belongs to the system. If a name appears at all, it is the name of the person who caught it, because that is the part worth reinforcing.

Say the number. "We prevented a Critical" carries weight that "good catch on the pricing" does not, and the weight is deserved. This is the whole reason to have a score: it lets prevention be described in the same units as failure, which is the only way prevention ever competes for attention.

Report Prevented Impact upward on the same slide as Escape Rate. A quality function that only ever brings bad news will, over time, be given less to look at. Stop Counting Mistakes. Start Measuring Escapes. introduced Prevented Impact as a metric; its real function is political, and that is not a criticism. A QA function needs a defensible account of what it produced.

Tell the client, selectively. "We caught an incorrect price before the campaign went out and we've added a pricing verification step" is not an admission of weakness. It is the only evidence a client will ever see that a quality process exists. Clients do not observe the defects you prevent. They observe the ones you do not.

Let the review pass be a named step with a named owner. Recognition attaches to roles that exist. A verification that happens informally, whenever someone has time, cannot be credited when it works, and will be the first thing dropped when the week gets tight.

Then Do the Unglamorous Part

Recognition is not the whole response, and an organization that stops at congratulation has made a different mistake.

A Critical caught at E1 means the control at E1 held. It also means nothing upstream of E1 caught it, and that is a finding of its own.

The questions worth asking are about the control, not the person:

Where did the wrong price originate, and does that source get verified anywhere?

Was this caught by a defined step or by luck? Did someone happen to notice, or did a step require them to check?

If it was luck, the control does not exist yet and today's outcome was a coin flip.

Would the same defect be caught in a different channel, or does the verification only exist on the path this asset happened to take?

That investigation is entirely compatible with recognition, provided the sequence is right. Credit the catch, then examine the control. Reversing the order teaches everyone that a Critical finding is the beginning of an interrogation, and the reporting will adjust accordingly within a quarter, the mechanism documented in Metrics Can Destroy the Culture They Were Designed to Improve.

The Obvious Objection

If catches are celebrated and scores drive the celebration, somebody will eventually notice that scores are assigned by people.

This is a real risk and it deserves a direct answer rather than a reassurance.

Severity inflation is possible, and it is self-limiting in a way that most gaming is not, for a structural reason: the score has to be defensible in five specific dimensions with three levels each, in front of colleagues who work on the same accounts. Arguing that an internal deck has Reach 2 is not a subtle manipulation. It is a claim that anyone in the room can check.

Two light guards handle the rest.

Score before you know the outcome where possible: assess impact when the finding is raised, not after you have decided how to describe it.

And review the distribution occasionally. If Critical findings are 30% of the log, either the work is genuinely dangerous or the scale has drifted, and both are worth knowing.

Compare that with the failure mode on the other side. Severity deflation, quietly scoring things low so the log stays calm, is invisible, requires no coordination, and destroys the data permanently. Of the two directions the scale can drift, the one that comes with celebration attached is by far the less dangerous.

Why This Is the Load-Bearing Part

Every other article in this series describes something an organization could implement on paper and still fail at.

You can adopt the score, define the escape levels, build the log, and produce a dashboard, and none of it will work if the first Critical finding of the quarter is met with a search for whose it was.

Because the framework runs entirely on voluntarily reported information. There is no sensor. The data exists because a person decided to write down something they noticed, and that decision gets made hundreds of times a quarter by people reading the room.

The whole system is downstream of what happens to the person who says "I found something."

Get that ten minutes right and everything upstream of it becomes possible.

Get it wrong and you will have a well-designed framework, a clean dashboard, and no idea what is actually reaching your clients.