Agentic Browser Testing

<objective> A scripted Playwright test breaks the moment a button moves or a class renames; writing one for a dashboard that changes weekly is a maintenance treadmill. This skill stands up a goal-driven browser agent instead: it reads a natural-language goal, explores the app via the accessibility tree (Playwright MCP browser_snapshot), and asserts the outcome against an explicit oracle. The failure mode it prevents is the one that makes teams distrust agents — an agent that reports "success" while stuck on the login page because nothing forced it to prove where it landed. You leave with a deterministic, CI-gated agent run and a graduation path to a durable scripted test once the flow stabilizes. </objective>

Quick Route

Discovery Questions

First, check .agents/qa-project-context.md in the project root and skip anything it already answers (stack, environments, seed/reset tooling, model access).

1. Which flow, and how often does its UI change? Fast-changing/experimental UI favors intent-driven; a stable critical path (login) favors scripted. This decides the whole approach.
2. Is there a seeded fixture and a way to reset state? Determinism is impossible without seeded data and a per-run reset. If neither exists, that is step zero.
3. Can you deep-link past auth to a seeded entry point? Re-driving login every run is the biggest avoidable cost; a seeded entry URL scopes the goal and cuts steps.
4. What is the unambiguous success oracle? Specific account text, a /dashboard URL, an order number — plus a forbidden state. "No error" is not an oracle.
5. Does the target render to canvas / WebGL? No accessibility tree means snapshot-first won't work; plan the vision fallback or instrument the canvas with ARIA.
6. Which model and budget? Pin a model id and a step budget up front; tier cheap steps to Haiku 4.5 / Sonnet 4.6 and reserve Opus 4.8 for genuinely ambiguous flows.

Core Principles

1. Intent, not instructions — but only where churn earns it. The agent reads a goal and finds its own path through the accessibility tree, so it survives a moved button or renamed class that would break a selector. That resilience costs 2-5x the time and money of a scripted run, so spend it on fast-changing UI and hard-to-locate flows, not on stable critical paths.

1. An agent run is untrustworthy until it is deterministic. Same goal, same seeded app must produce the same verdict. That requires temperature 0, a pinned model id, seeded data with a reset, a bounded step budget, and an explicit pass/fail assertion. Without these you have a coin flip, not a test.

1. The oracle lives outside the agent. Never let the LLM self-grade "looks good." Success is a checkable assertion against the final browser_snapshot — specific expected text, a URL, AND a forbidden-state negative check — evaluated by your harness, not the model.

1. Accessibility tree first, pixels last. browser_snapshot returns roles, refs, and accessible names (~200-400 tokens) and is deterministic and cheap. Screenshots, pixel coordinates, vision, and OCR are a scoped last resort for canvas only, never the default.

1. Graduation is the goal, not perpetual agent runs. Once a flow is stable, promote the run to a durable scripted tests/*.spec.ts with role-based locators. An agent that has been green for two weeks should become a fast, free regression test — keep the agent for exploration, not for guarding a settled path.

Fit: Intent-Driven vs Scripted

The decision is per-flow, not per-project. Run risk-based-testing first if you need the risk map; this table is the routing rule once you have it.

The rule, stated plainly: keep stable critical paths scripted and pinned; point intent-driven agents at fast-changing UI and exploratory smoke. Do not move everything to the agent — it is slower, costlier, and non-deterministic, and not every test should be agentic.

The Interaction Model (accessibility-tree-first)

Playwright MCP is not computer-use with screenshots and pixel coordinates. It is accessibility-tree-first:

1. browser_navigate to the seeded entry URL.
2. browser_snapshot returns the accessibility tree — each interactive element as a role, a stable ref, and its accessible name (from ARIA/labels). ~200-400 tokens.
3. The agent picks an element by ref and calls browser_click or browser_type.
4. browser_wait_for waits on text appearing/disappearing — never a fixed sleep.
5. Re-browser_snapshot after the DOM changes; assert against that tree.

Why not screenshots: the snapshot is token-efficient (thousands of tokens cheaper than an image), deterministic (text refs, not fuzzy pixel matching), and needs no vision model or OCR. Feeding screenshots as the primary input makes the run slower, pricier, and flakier. browser_take_screenshot is for human evidence only, never as the assertion input.

See references/setup.md for the MCP registration, the full tool table, and the goal prompt.

Determinism: making a run trustworthy in CI

A run that passes once and fails the next with no app change is not yet a test. The fix is never "just retry" or bumping temperature for "smarter" exploration — that adds variance. Pin the variables instead:

Avoid: temperature: 0.7 or 1 for exploration, retry-until-pass loops, waitForTimeout sleeps, and screenshot-based assertions. Each one hides flakiness rather than removing it. Full harness config in references/setup.md.

Success Assertion: the Oracle (where agents fail silently)

This is the sharpest failure mode: the agent reports success while stuck on the login page, because "page loaded / no error / looks good" was accepted as success and the LLM was allowed