Synthetic Monitoring

<objective> Synthetic monitoring runs scripted tests against production on a schedule, 24/7. It catches outages, performance degradation, and broken flows before real users report them — at 3 AM when traffic is zero, probes are the only thing checking your app works. A login page that always returns 200 but never authenticates passes a naive uptime check; a synthetic probe that asserts the dashboard loads catches it. This skill covers probe design, alerting integration, SLA validation, multi-region execution, and the runbook discipline that keeps a 3 AM page actionable. </objective>

Quick Route

Discovery Questions

Check .agents/qa-project-context.md first. If it exists, use it as context and skip questions already answered there. Each answer changes the probe set, the alerting config, or the SLA math.

Critical flows (decides which probes you write):

• What are the 5-10 most important user journeys (login, search, checkout, signup, core workflow)? These become your probe list.
• Which flows, if broken, cause immediate revenue loss or churn? These get the shortest frequency and page on-call.
• Are there flows that break silently (data sync, background jobs, webhooks)? Silent failures need probes most — no user reports them.
• Do you have documented SLAs/SLOs for availability and response time? They set the downtime budget and the alert thresholds.

Current monitoring (decides where the gaps are):

• What exists today (uptime checks, APM, error tracking, dashboards)? Avoid duplicating; find the gap.
• Are there gaps between what monitoring covers and what users experience? That gap is where probes earn their keep.
• How do you learn about production issues today (alerts, user reports, social media)? If it's user reports, detection time is your first metric to fix.
• What was the last outage, and how long before it was detected? Sets the detection-time target probes must beat.

Infrastructure (decides regions and CDN assertions):

• Is the app served from multiple regions or one? Multi-region serving needs multi-region probing.
• Are there CDN, caching, or edge layers that could mask origin failures? If yes, probes must assert on x-cache/origin headers.
• Do third-party dependencies (payment, auth, email) have their own monitoring? Their status pages feed your runbooks.
• What alerting systems exist (PagerDuty, OpsGenie, Slack, email)? Decides the routing config.

Test accounts (decides data safety):

• Do dedicated synthetic test accounts exist in production? Without them, probes pollute real data and analytics.
• Can test accounts be excluded from analytics, billing, and email campaigns? If not, probe traffic skews every downstream number.
• Is there API access for programmatic account setup and data cleanup? Decides whether create-then-delete probes are viable.

Core Principles

1. Synthetic tests validate the user experience continuously

RUM tells you what happened; synthetic tells you what is happening right now, whether or not real users are active. At 3 AM when traffic is zero, synthetic probes are the only thing checking the app works. Complement RUM, never replace it: synthetic covers known paths with predictable inputs, RUM discovers the creative ways real users break things.

2. Keep probes simple and fast

A probe that takes 2 minutes and touches 15 pages is not a probe — it is an E2E suite running in production. Probes are short (per-probe wall-clock budget under 30 seconds), focused (one critical path each), and stable (retries: 0, zero flakiness tolerance). The 30-second ceiling is a hard budget: a slow probe that "still passes" is masking a degradation users feel.

3. Alert on trends, not single failures

A single probe failure is noise — network blips and DNS hiccups cause them constantly. Two consecutive failures are a signal; failures from 2+ regions confirm it is not local. Configure consecutive-failure and multi-region thresholds, or alert fatigue teaches the team to ignore the pager.

4. Probes must not affect production data

Probes run every few minutes, 24/7. Even small side effects (a created record, an incremented counter) compound. Probes must be non-destructive, created data cleaned up immediately, and synthetic traffic excluded from analytics, billing, and — critically — from SLO/error-budget math itself, or probes inflate your own reliability numbers.

5. Assert on the goal, not the status code

A login page that returns 200 but never authenticates is broken. A search page that returns 200 with zero results is broken. Probes assert that the user can accomplish their goal — data loads, auth succeeds, results appear — not merely that the page returns a 2xx.

Probe Design

Design probes around critical user journeys, not infrastructure components. Users do not care if your load balancer is healthy — they care if they can log in and use the product.

Probe implementations

Three probe shapes cover most needs: a login flow (Playwright browser), an API health check (status + auth + latency budget), and a search probe (fill query → submit → assert on results, not just status). Keep each to one critical path, a tight timeout, and retries: 0. See references/probe-implementations.md for runnable login-flow, API-health, and search probes plus the environment-aware config.

Non-destructive probe patterns

Safe patterns:
  - Read-only operations: GET requests, page loads, searches
  - Sandbox transactions: payment in test mode, email to internal addresses
  - Create-then-delete: create a draft, verify, delete immediately
  - Synthetic flag: operations tagged as synthetic, excluded from processing

Dangerous patterns (avoid):
  - Creating real orders, tickets, or user-facing records
  - Triggering real notifications (email, SMS, push)
  - Modifying shared resources (config, permissions, settings)
  - Operations that cannot be automatically cleaned up

Dedicated test accounts

Account requirements:
  - Clearly identifiable: email contains "synthetic" or "monitor"
  - Flagged in database with an is_synthetic flag (is_synthetic = true)
  - Excluded from analytics, billing, email campaigns, support queues
  - Excluded from RUM and SLO/error-budget pipelines (synthetic traffic is not real traffic)
  - Pre-populated with stable test data that probes can rely on
  - Credentials in a secrets manager (process.env), rotated quarterly
  - Separate accounts per concurrent probe (avoid state conflicts)

Platform Options