AI Test Generation

<objective> LLMs will happily emit fifty plausible-looking tests that assert nothing, target endpoints that do not exist, and duplicate each other. This skill is a staged pipeline that forces structured intermediates — assumptions, coverage matrix, oracle definitions — out of the model BEFORE any test code, so what you get is traceable, reviewable, and grounded in the real codebase instead of ad-hoc generated noise.

Before starting: Check for .agents/qa-project-context.md in the project root. It carries tech stack, test frameworks, naming conventions, selector strategy, and known risk areas that dramatically improve generated test quality. </objective>

Quick Route

The pipeline is the same for every input; only the Step 1 extraction emphasis changes. Jump to the matching row, then run Steps 2-7 unchanged.

Playwright projects also pick an agent integration mode — see Discovery Q2.

Discovery Questions

Check .agents/qa-project-context.md first — if it exists, use it and skip anything already answered there. Then clarify:

1. What is the input source? PRD / spec, user story + AC, code diff, bug report, or API schema. Determines Step 1 extraction emphasis (see Quick Route). For an LLM/AI feature spec, stop — generate eval datasets in ai-system-testing, not Playwright specs.

1. What is the target test framework, and (for Playwright) which agent integration mode?

• E2E: Playwright (preferred), Cypress. Unit: Jest, Vitest, pytest. API: Playwright APIRequestContext, Supertest, requests.
• Playwright CLI + agents (recommended for Claude Code / Codex / Cursor): npx playwright init-agents --loop=claude scaffolds planner/generator/healer agents into .claude/agents/ as markdown. They are interactive dev tools that produce standard Playwright tests which run unchanged in CI. Token-efficient; runs inside the agent's loop.
• Playwright MCP (npx @playwright/mcp@latest): higher overhead, right when the agent must drive a live browser interactively over a long session.
• Neither — hand-write tests using AI as a scratch-pad helper.

1. What project context is available? Existing test patterns, Page Objects / helpers, data factories / fixtures, CI constraints (timeout, parallelism). More context = less cleanup.

1. What is the review workflow? Full pipeline → human review → merge (default); scenarios only → human writes code; or code → human refines iteratively.

1. What domain knowledge is needed? Regulated industry (healthcare, finance) compliance, domain invariants (money never negative, appointments cannot overlap), known risk areas from past incidents.

Core Principles

1. Pipeline before code. Never generate test code before establishing what to test, why, and how to verify it. The seven-step pipeline exists to prevent premature code generation that targets the wrong things.

1. Structured intermediates are the product. The assumptions document, coverage matrix, and oracle definitions are more valuable than the test code itself. They are reviewable, traceable, and reusable.

1. Separate what from how. Scenario generation (what to test) and oracle design (how to verify) are distinct cognitive tasks. Mixing them produces scenarios biased toward what is easy to assert, with assertions tacked on as afterthoughts.

1. AI generates the first draft; a human reviews and refines. Never ship AI-generated tests without human review. The AI accelerates — it does not replace judgment.

1. Context is everything. Feed the LLM your conventions, existing patterns, selector strategy, and data setup. The more context, the less cleanup.

1. Quality over quantity. Each test has a maintenance cost. Focus on critical paths, complex logic, and known risk areas — not test count.

The Pipeline

Mandatory workflow — agents MUST follow this order:

Step 1: Extract   → Requirements, entities, business rules from input
Step 2: Analyze   → Risks, invariants, edge cases, ambiguities
Step 3: Map       → Coverage matrix (requirement → scenario → priority)
Step 4: Generate  → Candidate scenarios (happy + boundary + negative + security + a11y)
Step 5: Design    → Assertions and oracles SEPARATELY from scenarios
Step 6: Code      → Test code (only after all above exist)
Step 7: Review    → Human review with traceability back to source

Full prompt templates for every step (extraction, risk analysis, scenario, oracle, code) live in references/prompt-patterns.md. Below is the shape of each step's output.

Step 1: Extract Requirements and Entities

Parse the input into structured elements: Entities (with roles/states/attributes), Business Rules (numbered), Explicit Requirements ([REQ-N], stated in source), and Implicit Requirements ([IMP-N], inferred — flag every one for human confirmation). Separating explicit from inferred is the rule that prevents testing assumptions as if they were specifications.

Step 2: Risk Analysis and Invariants

Derive what can go wrong, what must always be true, and where the source is silent.

• Risks — table of Risk | Likelihood | Impact | Source Requirement (e.g. race condition on stock decrement, email delay > 30s).
• Invariants (must ALWAYS hold) — stock >= 0, order total = sum(items) + tax + shipping, user sees only their own orders.
• Ambiguities (need human answers) — "Does free shipping apply before or after discount codes?" Capture these explicitly; do not silently pick one.
• Edge cases derived from risks — two users buy the last item, payment succeeds but email service is down.

Step 3: Coverage Matrix

The single most important artifact — it prevents both gaps and duplicates. Map every requirement to scenarios with category, priority, and oracle type:

After building it, verify: every requirement has ≥1 happy and ≥1 negative scenario; every invariant has a direct test; every Step-2 risk has a scenario; no two rows test the same thing.

Step 4: Generate Candidate Scenarios

For each matrix row, write the full scenario in Given/When/Then with explicit test-data requirements (Given: user with 99 items in cart (max 100); When: adds one more; Then: count = 100). Cover these categories systematically: