QA Testing Strategy

QA Testing Strategy

Risk-based quality engineering guidance for modern software delivery. Use this skill to decide what to test, at which layer, with which gates, and how to keep the signal trustworthy.

Start with references/operational-playbook.md for the navigation hub. Use current official sources from data/sources.json when you need vendor or standards guidance.

Scope

• Create or update a risk-based test strategy
• Choose a test shape (pyramid, trophy, honeycomb) based on architecture and defect origin
• Define merge gates, deploy gates, and release evidence — including merge queue interaction
• Choose the smallest effective layer: unit, component, contract, schema fuzzing, integration, E2E, property-based
• Make failures diagnosable with artifacts, correlation IDs, traces, and ownership
• Operationalize suite health: flake SLO, quarantine policy, execution budgets, dashboards

Use Instead

Quick Reference

E2E Gate Topology (Default)

Use three distinct E2E scopes instead of one monolithic suite:

• Smoke: PR gate and fastest feedback on the highest-risk journeys.
• Targeted batch/spec: local triage and deflake work for one journey, subsystem, or dependency chain.
• Deploy-gate replay: dependency-chain or critical-journey replay used only when proving release readiness.

Rules:

• Do not use full local E2E as the first response to a single failing journey.
• Treat rerun-pass as unresolved flake debt.
• Promote scope only after the smaller scope is green.

Default Workflow

1. Clarify scope and risk: critical journeys, failure modes, compliance constraints, and non-functional risks.
2. Define quality signals: SLOs, budgets, contract checks, accessibility target, and what blocks merge vs deploy.
3. Choose the smallest effective layer first: unit, component, contract, schema fuzzing, integration, then E2E.
4. Make failures diagnosable: logs, traces, screenshots, videos, build links, request IDs, trace IDs, and owners.
5. Operationalize the suite: explicit smoke vs targeted-batch vs deploy-gate scopes, quarantine with expiry, suite budgets, retries with evidence retention, and dashboards.

Decision Rules

Need to test: [Change or Risk]
    │
    ├─ Pure business rule or invariant?
    │   └─ Unit test
    │
    ├─ UI behavior or component state in isolation?
    │   └─ Component test in a real browser
    │
    ├─ API compatibility between teams/services?
    │   └─ Contract test
    │
    ├─ API parser/validation edge cases against the schema?
    │   └─ Schema-aware fuzzing + core integration smoke
    │
    ├─ Real dependency boundary or persistence behavior?
    │   └─ Integration test with real DB/queue/service doubles only at external edges
    │
    ├─ User-critical cross-page workflow?
    │   └─ Thin E2E test
    │
    ├─ Universal invariant or property that should hold for all valid inputs?
    │   └─ Property-based test (fast-check / Hypothesis / jqwik)
    │
    └─ Capacity, resilience, or reliability regression?
        └─ Performance, resilience, or synthetic monitoring tests

Principles

• Prefer the smallest layer that can prove the behavior.
• Keep pre-merge gates fast: contracts, static checks, unit tests, selective component/integration smoke.
• Prefer targeted batch reruns locally; reserve full E2E for deploy gates or scheduled regression.
• Use full E2E only for critical journeys or risks that cannot be proven lower in the stack.
• Treat flaky tests as reliability defects, not harmless noise.
• Favor web-first assertions and stable locators over custom waits or brittle selectors.
• Treat accessibility automation as partial coverage. Pair it with manual checks and inclusive design review.
• Use telemetry as evidence. Production traces, incidents, and support signals should drive new tests.
• Use AI for brainstorming and triage only when evidence stays attached. Do not weaken assertions to “heal” tests.
• Treat AI-authored test oracle quality as a first-class risk, peer to flake debt. AI-generated tests routinely hit high line coverage while passing trivially (hardcoded returns, shallow assertions). Gate them on mutation score, not coverage: a test that cannot fail when the business logic is reverted is not a test.

Core Targets

Resources

• references/operational-playbook.md: start here
• references/component-testing-browser-mode.md: real-browser component strategy with Vitest Browser Mode
• references/playwright-webapp-testing.md: current Playwright guidance
• references/schema-aware-api-fuzzing.md: schema-driven API fuzzing with OpenAPI
• references/contract-testing.md: Pact, Specmatic, and contract decisions
• references/observability-driven-testing.md: OpenTelemetry-first debugging and trace-based validation
• references/quality-metrics-dashboard.md: metrics, dashboards, mutation coverage, and anti-patterns
• references/production-testing-and-shift-right.md: synthetic monitoring, dark launches, feature flag rollouts, MTTR-flake SLO, production replay, observability-driven gates