From QA Checklist to Creative Risk Management
The full CQIS specification: materiality, trust, reach, reversibility and business consequence, escape levels E0 to E4, the metrics it supports, and the four rules that keep it from becoming punitive.
Eleven articles of argument produce an obligation.
If severity depends on context, somebody has to decide what the context was. If escapes matter more than defects, somebody has to record where each one stopped. If the same finding can be a disaster or a success depending on who found it, that distinction has to survive being written into a row in a spreadsheet.
Otherwise all of it stays a philosophy, and philosophies do not get applied consistently on a Thursday afternoon with four deliverables outstanding.
So here is the framework the series has been building toward. It is called CQIS, the Creative Quality Impact Score, and it is deliberately small.
This is the twelfth article in the series.
What It Is Actually For
CQIS gives a team a shared language for two separate questions about every finding:
How much impact could this defect have had?
How far did it get before somebody stopped it?
Those two answers are recorded independently and never combined into a single number. Everything that makes the framework survivable follows from keeping them apart.
It evaluates the work. It does not evaluate the person who produced the work.
Part One: The Impact Score
Every finding is scored across five dimensions. Each dimension is worth 0, 1, or 2 points, giving a total from 0 to 10.
Five dimensions with three levels each is a deliberate choice. It is coarse enough to be scored in under a minute by someone who would rather be doing something else, which is the only kind of framework that gets used.
A. Materiality: how wrong is the output itself?
| 0. Cosmetic | Minor spacing, capitalization, alignment or stylistic inconsistency, with little effect on meaning. |
| 1. Noticeable | Typo, grammar issue, minor visual problem, awkward wording, incorrect formatting. |
| 2. Material | Wrong fact, date, price, link, image, CTA, claim, audience, account, product information or functionality. |
B. Trust Impact: how much could this affect confidence in the client or the agency?
| 0. Negligible | Unlikely to affect anyone's perception of quality or competence. |
| 1. Noticeable | Could make the work appear careless or unprofessional. |
| 2. Significant | Could materially damage confidence in the agency or the client's brand. Misspelling an important person's name, publishing something obviously incorrect, using another client's assets, an error in a highly visible campaign. |
C. Reach: how many people or systems could be exposed?
| 0. Limited | Internal material, draft, staging environment, very small audience. |
| 1. Normal | Standard website page, organic social post, client communication, limited campaign. |
| 2. Broad | Paid advertising, large mailing list, major campaign, high-traffic page, reusable template, syndicated content, or content distributed across multiple channels. |
D. Reversibility: how difficult is it to undo once released?
| 0. Easy | Can be corrected before meaningful external exposure. |
| 1. Recoverable | Can be edited or removed, although some people may already have seen it. |
| 2. Difficult or irreversible | Email already delivered, printed material distributed, paid campaign widely served, confidential information disclosed, screenshots circulating. |
E. Business Consequence: what could reasonably happen if it escapes?
| 0. Minimal | Little meaningful consequence beyond fixing the work. |
| 1. Meaningful | Client complaint, wasted ad spend, campaign confusion, additional labor, customer support load, some loss of confidence. |
| 2. Serious | Potential client churn, significant revenue loss, legal or regulatory exposure, confidentiality problems, fraud, major reputational damage. |
The score
Add the five values.
Materiality + Trust + Reach + Reversibility + Business Consequence
| Score | Rating |
|---|---|
| 0 | Informational |
| 1 to 3 | Low |
| 4 to 5 | Medium |
| 6 to 7 | High |
| 8 to 10 | Critical |
The score represents potential impact. It is not a description of what happened, and it is not a judgment about anyone.
Part Two: The Escape Level
Impact alone does not tell you whether QA worked. A Critical defect caught by the person who made it and the same Critical defect found by a customer are opposite outcomes.
So every finding also gets an Escape Level, recording the last boundary it crossed before somebody stopped it.
| Level | Who found it | Outcome |
|---|---|---|
| E0 | The creator, before review | Prevention |
| E1 | Internal QA: a colleague, a reviewer, an automated check | Prevention |
| E2 | The client | Quality Escape |
| E3 | The client's customer or the public | External Quality Incident |
| E4 | Amplified beyond the original audience: complaints, screenshots, shares, press, paid distribution, legal escalation | Major Quality Incident |
This is the trust chain from the third article in this series, collapsed into a field you can actually fill in.
The mapping is deliberately not one to one. The chain is a structure of relationships; the escape level is a record of detection, and for scoring purposes two of its nodes behave identically. A defect seen by the client's customer and one seen by the general public produce the same kind of loss, so they share E3. What earns a separate level is amplification, which is not a place in the chain at all. It is what happens when the error acquires its own distribution.
Recording it costs one dropdown. It is the single most useful column in the log, because a cluster of escapes at one level names the missing control precisely.
Part Three: Reading the Two Together
The framework's whole behavior comes from combining the score and the level into a phrase.
Take a typo in a public social post. Under normal circumstances:
- Materiality 1, Trust 1, Reach 1, Reversibility 1, Business Consequence 1.
- CQIS 5: Medium.
Caught by internal review:
CQIS 5 / E1: Medium Impact Prevented.
That is a QA success, recorded as one.
Now put the identical typo in a major campaign for a client whose customers expect polished communication:
- Materiality 1, Trust 2, Reach 2, Reversibility 1, Business Consequence 2.
- CQIS 8: Critical.
Caught internally:
CQIS 8 / E1: Critical Impact Prevented.
The person who found that did not find a failure. They prevented one.
Found instead by the client's customer:
CQIS 8 / E3: Critical External Quality Incident.
The typo never became more incorrect. The context and the distance it traveled changed everything about what it cost.
Part Four: The Log
In practice the entire framework is one table.
| Finding | CQIS | Escape | Result |
|---|---|---|---|
| Minor spacing issue | 1 | E1 | Low Impact Prevented |
| Typo in social copy | 4 | E1 | Medium Impact Prevented |
| Wrong event date | 7 | E1 | High Impact Prevented |
| Broken CTA found by client | 6 | E2 | High Quality Escape |
| Wrong price seen by customers | 8 | E3 | Critical External Incident |
| Another client's asset posted publicly | 9 | E3 | Critical External Incident |
| Confidential information distributed | 10 | E4 | Critical Major Incident |
Four columns. No tooling required to start, and no tooling should be bought until several months of rows exist and the team knows which questions it actually wants to ask.
Part Five: What to Measure
The log supports a small set of metrics, and the set is small on purpose.
Internal Catch Rate: the proportion of known defects contained at E0 or E1.
Escape Rate: the proportion that reached E2 or beyond.
Critical Escape Rate: how many High and Critical findings reached E2, E3 or E4. This is the number to review most often; it maps directly to what the organization is trying to prevent.
Repeat Defect Rate: how frequently the same class of defect reappears. Recurrence usually indicates a process problem rather than a people problem, with the caveats from When Does a Mistake Become a Pattern?
Prevented Impact: how many Medium, High and Critical findings were contained at E0 or E1. The healthiest metric in the set, and the only one a team has an honest incentive to increase.
The reasoning behind choosing these rather than a defect count is Stop Counting Mistakes. Start Measuring Escapes., and the reason every one of them needs a denominator before it is compared between people is No Rate Without a Denominator.
Part Six: The Rules That Make It Survivable
A scoring framework is an incentive system. This one has four rules, and they are not optional garnish. Remove them and the framework becomes a more precise version of the problem it was built to solve.
Never build a leaderboard. Do not measure "John created four High-severity defects this month." That number creates immediate incentives to hide defects, minimize their severity, avoid logging them, or attribute them elsewhere. Measure the effectiveness of the system, not the fallibility of individuals.
A high internal count is a good sign. A team that reports many internal findings and very few external ones has a better QA culture, not a worse one.
Keep the two fields separate, permanently. The moment severity and escape level are collapsed into a single quality number, "Critical" starts sounding like an accusation, and the underreporting begins. Metrics Can Destroy the Culture They Were Designed to Improve is the full account of how that unfolds.
Recognize prevention explicitly. Do not record "Jane found John's mistake." Record "Critical Impact Prevented (E1)." The finding belongs to the system, and the credit belongs to whoever caught it. The next article, A Critical Defect Caught Internally Is a Win, is entirely about this.
The cultural rule underneath all four is the one from Red-Team the Work, Not the People:
Challenge the work adversarially. Do not treat your coworkers as adversaries.
What This Is Not
CQIS is not a maturity model, a certification, or a substitute for judgment. Scoring a finding still requires someone to decide what "broad reach" means for this client, and two reasonable people will sometimes differ by a point.
That is fine. The value is not in the precision of the number. It is in the fact that the conversation now happens in the same units every time, and that the answer to "how bad is this?" stops depending on who is in the room and how the week has gone.
It is also not a substitute for controls. A framework that scores defects while the process that produces them stays unchanged is an elaborate way of documenting decline.
The Premise, One More Time
The framework begins from a single assumption, which is the one the series opened with.
Errors will happen.
A mature quality process does not demand perfection from the people doing the work. It assumes human and technological fallibility, and it builds systems capable of catching mistakes before they become consequences.
Which is why the organization has to be able to tell the difference between creating a defect and allowing a defect to escape.
Those are not the same event, and until they are recorded as separate things, no amount of goodwill will keep an organization from confusing them.
A defect caught internally is not a QA failure. It is evidence that QA worked.
The objective was never a workplace where nobody makes mistakes.
It is a workplace where preventable mistakes find it increasingly difficult to reach the people whose trust matters most.