Quality Postmortem

<objective> Analyze escaped defects, test suite health, and quality process gaps through structured, blameless postmortems. Every postmortem produces 1-3 concrete, tracked action items -- not vague commitments to "be more careful." The goal is systemic improvement, not individual blame. </objective>

Quick Route

Discovery Questions

Check .agents/qa-project-context.md in the project root first — it carries quality goals, risk areas, and test suite details that anchor any postmortem. Use it and skip anything already answered there. Then clarify:

1. Do you have a regular retro cadence? Per-sprint, monthly, or only after incidents? Regular cadence catches slow-burn problems. Incident-only cadence misses patterns until they explode.

1. What triggered this postmortem? A production incident? A pattern of escaped bugs? A feeling that the test suite is not catching enough? Test suite degradation? The trigger determines the focus.

1. What data is available? Bug tracker with severity and discovery phase? CI history with pass rates? Flaky test reports? Coverage trends? Without data, postmortems devolve into opinion sessions.

1. What happened with previous postmortem action items? Were they completed? Tracked? Forgotten? If past action items are abandoned, the team has learned that postmortems do not matter. Fix the follow-through before running another postmortem.

1. Who should participate? Engineers who worked on the affected area. QA who tested (or did not test) it. Product owner if the impact was user-facing. Engineering manager if systemic changes are needed. Keep the group to 4-8 people.

1. What are the current test suite health concerns? Rising flakiness? Slow execution? Coverage gaps in critical areas? Stale quarantine? Health reviews are proactive postmortems -- they prevent incidents instead of reacting to them.

Core Principles

1. Blameless Means Systemic

Blameless does not mean "no one is accountable." It means the analysis focuses on systems, processes, and tools rather than individual performance. "Why did the system allow this defect to escape?" is a blameless question. "Why did the developer not write a test?" is a blame question that stops the analysis too early. The developer did not write a test because: the test framework was hard to use, the PR checklist did not require it, there was no pairing to transfer knowledge, or time pressure made it feel optional. Those are systemic issues with systemic fixes.

2. Focus on Patterns, Not Incidents

A single escaped bug is an anecdote. Three escaped bugs in the same feature area over two months is a pattern. Postmortems should aggregate incidents to find recurring themes: same root cause, same team, same test gap, same phase of the pipeline. Patterns are actionable. Individual incidents are just fire-fighting.

3. Every Postmortem = 1-3 Concrete Action Items

An action item is concrete when it has: a specific deliverable ("add integration tests for the coupon API"), an owner ("assigned to Alex"), a deadline ("by end of sprint 14"), and a verification method ("PR merged, tests passing in CI"). "Improve testing" is not an action item. "Write 5 integration tests for the payment service edge cases by March 30" is.

4. Track to Completion

Action items that are not tracked are not completed. Use the team's existing work tracker (Jira, Linear, GitHub Issues). Tag them (postmortem-action or equivalent). Review completion status at the start of the next postmortem. If items are consistently abandoned, either the items are too large (break them down) or they are not prioritized (make them sprint commitments).

5. Measure Improvement With Two Metrics, Not One

After implementing action items, measure whether the problem recurred. If the postmortem identified a gap in payment testing and the action was to add integration tests, track: did another payment bug escape? Without measurement, postmortems are rituals, not tools.

Track two metrics together:

• Defect escape rate (did similar bugs reappear?)
• Action-item-closure rate (what fraction of action items shipped within their committed window?)

A high closure rate with rising escape rate means the team is doing the work but doing the wrong work. A low closure rate means the postmortems are theater. Modern incident response platforms (incident.io, Rootly, FireHydrant) track action-item follow-through natively — owner, due date, completion status — so derive both numbers from what's already there before building a dashboard.

6. AI Drafts the Timeline; a Human Owns the Judgment

If your team uses AI SRE tooling (Rootly AI SRE, incident.io's AI SRE / auto-drafted post-mortems), let it draft the incident timeline and propose candidate root causes from logs and traces. Then a named blameless RCA owner — distinct from the incident commander who managed the response — runs the 5 Whys, picks the real root cause, and writes the action items. AI is good at correlation across noisy data; it is bad at deciding what mattered. Treat AI output as a starting deck, not the conclusion. For cheap timeline drafting, Sonnet 4.6 is sufficient; reserve heavier models for ambiguous causation.

Bug Pattern Analysis

Categorizing Escaped Defects

When a bug reaches production, classify it along three dimensions to identify prevention opportunities. The single-bug worksheet (Escaped Bug Analysis) lives in references/templates.md.

Dimension 1: Root Cause Category

Dimension 2: Which Test Level Should Have Caught It

Dimension 3: Prevention Opportunity