Chaos Engineering

<objective> Chaos engineering is the discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions. It is not random destruction -- it is hypothesis-driven, controlled experimentation that reveals weaknesses before they cause outages. A retry that "works in the demo" silently double-charges customers when the payment API times out; the only way to know is to inject the timeout and watch. </objective>

Quick Route

Discovery Questions

Check .agents/qa-project-context.md first. If it exists, use it as context and skip questions already answered there.

Environment and readiness:

• Where will chaos experiments run? (Pre-production only, production with approval, never production)
• What is the team's monitoring maturity? Can you detect problems in real time?
• Has the team practiced incident response? Is there a runbook?
• Is there executive buy-in for chaos engineering? (Important for production experiments)

Architecture:

• What is the architecture? (Monolith, microservices, serverless, hybrid)
• What are the critical dependencies? (Database, cache, message queue, third-party APIs)
• Are there single points of failure? (Single database, single region, no redundancy)
• What redundancy and failover mechanisms exist?

Current resilience practices:

• Do services have health checks? What do they check?
• Are there circuit breakers, retry logic, or timeout configurations?
• What happens when a dependency is unavailable? (Graceful degradation, hard failure, unknown)
• Have you experienced unexpected outages? What failed?

Team and culture:

• Is the team comfortable with controlled failure? (Anxiety is normal and should be addressed)
• Who would be the chaos engineering champion? (Needs someone to own the practice)
• What is the appetite for starting? (Start small or dive in)

Core Principles

1. Hypothesis-driven: define expected behavior before injecting

Every chaos experiment starts with a hypothesis: "We believe that if [failure X occurs], the system will [expected behavior Y]." Without a hypothesis, you are just breaking things. The hypothesis names concrete steady-state metrics (baseline metrics) — error rate, latency, throughput — and the bound each may move to.

Example hypothesis: "We believe that if the primary database becomes unavailable, the application will serve cached data for read requests and queue write requests for up to 5 minutes without user-visible errors. Blast radius: staging, one service. Steady-state baseline: error rate <0.1%, P95 latency <300ms."

2. Start small: one service, controlled blast radius

The first chaos experiment should not be "shut down production." It should be "add 200ms latency to one non-critical service in staging." Increase scope gradually as confidence and tooling mature.

3. Monitoring is a prerequisite

If you cannot detect problems in real time, you cannot safely inject failures. Chaos experiments without monitoring are just outages with extra steps. Verify dashboards, alerts, and on-call processes before running any experiment.

4. Game days build muscle memory

Running chaos experiments in automated pipelines is valuable, but game days -- scheduled sessions where the team runs experiments together and practices response -- build the human skills that matter during real incidents.

Chaos Experiment Workflow

Every chaos experiment follows this five-step process.

Step 1: Define steady state hypothesis

Identify the metrics that define "normal" and predict what should happen during the experiment.

Experiment: Database failover
Steady state:
  - Error rate: < 0.1%
  - P95 latency: < 300ms
  - Successful orders per minute: > 50

Hypothesis: When the primary database fails over to the replica,
  - Error rate will spike to < 2% for < 30 seconds
  - P95 latency will increase to < 1s for < 60 seconds
  - No orders will be permanently lost
  - The application will recover without manual intervention

Step 2: Introduce the variable

Inject the failure in a controlled way with a clear scope and duration.

Injection:
  Target: primary database (PostgreSQL)
  Method: block TCP port 5432 on the primary instance
  Scope: single database instance
  Duration: 60 seconds
  Blast radius: staging environment only (first run)

  Abort conditions:
    - Error rate > 10% for > 2 minutes
    - Any data corruption detected
    - Manual abort by experiment owner

Step 3: Observe

During the experiment, monitor all relevant metrics in real time. Assign observers to specific dashboards.

Observation assignments:
  - Engineer A: application error rate and latency dashboard
  - Engineer B: database metrics (connections, replication lag, failover status)
  - Engineer C: application logs (search for database connection errors)
  - Engineer D: business metrics (order count, payment processing)

Step 4: Analyze recovery and data integrity

After the experiment, analyze what happened versus what was expected.

Analysis checklist:
  - Did the system behave as hypothesized? (Y/N, with details)
  - How long was the impact? (Expected vs. actual duration)
  - Were any errors visible to users?
  - Was any data lost or corrupted?
  - Did monitoring and alerting detect the problem correctly?
  - How long before alerts fired?
  - What was the recovery time?

Step 5: Fix and iterate

Document findings, fix resilience gaps, and schedule a re-run to verify the fix.

Findings document:
  Experiment: Database failover (2026-03-20)
  Hypothesis: Confirmed / Partially confirmed / Disproved
  Recovery time: 45s (expected vs actual: expected <10s, actual 45s)
  Data integrity: no rows lost; 3 writes returned 500 instead of queueing

  Findings:
    - Connection pool did not detect stale connections for 45 seconds (expected: <10s)
    - Retry logic worked correctly for read operations
    - Write operations returned 500 errors for 38 seconds (expected: queued)

  Action items (every one has an owner and a due date — no item is deferred):
    - [ ] Configure connection pool health checks — assigned to @maria, due 2026-03-31
    - [ ] Implement write queue with 5-minute buffer — assigned to @dan, due 2026-04-02
    - [ ] Re-run experiment after fixes deployed (re-run scheduled 2026-04-03);
          specific metrics to check on re-run: stale-connection detection <10s,
          zero write 500s, error rate <2%

Failure Injection Types

Network failures

See references/fault-injection.md for the tc netem latency/packet-loss commands and the toxiproxy latency config.

Service failures