Testing In Production

<objective> Production is the only environment that is production. Every other environment is an approximation — staging never replicates real data volume, traffic, or third-party quirks. This skill covers how to validate quality in production safely: controlled blast radius, automated rollback, guardrail metrics, and smoke tests that catch problems before users do. The recurring failure it prevents: shipping to 100% of users with no flag to flip, no baseline to compare against, and no tested way back. </objective>

Quick Route

Discovery Questions

Check .agents/qa-project-context.md first. If it exists, use it as context and skip questions already answered there.

Feature flag system:

• Do you have a feature flag platform? (LaunchDarkly, Statsig — now part of OpenAI, GrowthBook, Unleash, Flagsmith, Harness FME — formerly Split, custom, none)
• How are flags managed? (Dashboard, config file, environment variables)
• Can flags target specific users, percentages, or segments?
• How many active flags exist today? Is there a cleanup process?

Rollout capability:

• Can you deploy to a subset of traffic? (Canary infrastructure, weighted routing, feature flags)
• How long does a deployment take? How long does a rollback take?
• Do you have blue-green or rolling deployments?
• Can you route traffic by region, user cohort, or percentage?

Monitoring maturity:

• What observability is in place? (APM, logging, error tracking, metrics)
• Do you have dashboards for error rate, latency, and business metrics?
• Are alerts configured with appropriate thresholds?
• Can you compare metrics between canary and baseline in real time?

Production access and safety:

• Who has production access? Is there an approval process?
• Are there dedicated test accounts in production?
• Can you run operations in production without affecting real user data?
• Is there a production incident response process?

Core Principles

1. Production is the final test environment

Staging approximates production. It does not replicate production's data volume, traffic patterns, third-party integrations, infrastructure quirks, or user behavior. Testing in production is not reckless — it is realistic. The question is not whether to test in production, but how to do it safely.

2. Safety through blast radius control

Every production test must answer: "If this goes wrong, how many users are affected?" The answer must be as small as possible. Feature flags, canary deploys, and traffic splitting exist to shrink the blast radius from 100% to 1% or less.

3. Always have a tested rollback plan

Before any production test begins, the rollback mechanism must be identified, tested, and fast. "Disable the flag" is a good rollback plan. "Redeploy the previous version" is acceptable. "We'll figure it out" is not a plan. A rollback you have never fired is a hypothesis, not a plan — see Verification for how to prove it works.

4. Monitoring is a prerequisite, not a nice-to-have

You cannot test in production without monitoring. If you cannot measure error rates, latency, and business metrics in real time, you cannot detect problems. Fix monitoring gaps before adding production tests.

5. Production tests must be non-destructive

Production tests must never corrupt real user data, send real notifications to real users, charge real payment methods, or create side effects that require manual cleanup. Synthetic accounts, test flags, and isolated resources are mandatory.

Feature Flag Testing

Feature flags are the safest mechanism for production testing. They decouple deployment from release and provide instant rollback.

Test with flags ON and OFF

Every flagged feature needs tests in both states. The flag-off path is the rollback path and must work flawlessly — use setFeatureFlag(name, true|false, { userId: TEST_USER_ID }) to drive both. See references/patterns.md for the full ON/OFF test pair.

Flag lifecycle testing

Flags are not just on or off. They transition through states, and each transition must be validated.

Flag lifecycle:
  Created → Targeting internal users → Canary (1%) → Partial (10-50%) → Full (100%) → Cleanup (removed)

Test at each stage:
  - Internal: Feature works for internal accounts, hidden from external
  - Canary: Metrics are comparable between flag-on and flag-off cohorts
  - Partial: No performance degradation at scale
  - Full: All user segments work correctly
  - Cleanup: Code with flag removed behaves identically to flag-on

Stale flag cleanup

Flags left in code become technical debt. Run a weekly CI job that queries the flag provider for flags that are 100% rolled out and older than 14 days. These are candidates for code cleanup — remove the flag branching logic and retain only the enabled path. Removing the flag without removing the dead code is half a cleanup.

Flag combination testing

When multiple flags interact, test the combinations that matter. Do not test all 2^N combinations — focus on flags that affect the same user flow (e.g. new-checkout, express-pay, discount-engine-v2 all touch checkout). Pick the critical combinations: all-new, a representative mixed state, and all-legacy. See references/patterns.md for the combination-test loop.

Progressive Rollout

Vendor-native canary analysis. Before hand-rolling the rollout-policy YAML below, check whether your platform already does it: LaunchDarkly Guarded Rollouts (auto-monitored progressive rollouts with metric-based auto-rollback; uses a frequentist sequential-testing analysis model since early 2026), Statsig Auto-tune, Argo Rollouts AnalysisRun, Flagger, Harness Continuous Verification. If you have one, prefer it — the integration with your metrics and rollback mechanics is cheaper than maintaining a custom analysis loop. AI feature rollout is its own pattern: model variant + prompt as a flag value, with cost guardrails and a kill switch. LaunchDarkly AI Configs / AgentControl (AI Configs rebranded under the AgentControl umbrella, announced May 2026) is the documented path for shipping LLM features behind progressive rollout. See release-readiness for the full pattern.

Canary stages: 1% to 100%

A structured rollout with explicit promotion criteria at each stage.

Automated promotion and rollback

Define machine-checkable conditions for advancing between stages — hold_duration plus metric conditions (error_rate_5xx < 0.5%, latency_p95 < 500ms, crash_rate == 0). Automatic rollback fires when guardrails are breached, with no human approval needed: error_rate_5xx > 2x_baseline for 5m, latency_p99 > 3x_baseline for 5m, crash_rate > 0.1% for 2m, each notifying on-call. Gate on error-budget burn rate, not only raw multipliers, so slow burns that still blow the SLO are caught. See references/rollout-policy.md for the full promotion YAML, rollback triggers, and SLO-gate config.

Verify the rollback fired correctly