CI CD Integration

<objective> A 20-minute serial suite on every push destroys developer velocity; a green pipeline that retries flaky tests three times hides the race condition until it ships. This skill produces CI/CD pipelines that run the right tests at the right trigger, shard them across runners, store traces and reports as evidence, quarantine flaky tests instead of masking them, and gate merges on real coverage numbers. Use this skill when the question is about running tests in a pipeline, not writing them. </objective>

Discovery Questions

Check .agents/qa-project-context.md first — if it exists, use it and skip anything already answered there (especially team_maturity and existing CI conventions). Then:

1. Which CI platform? GitHub Actions, GitLab CI, CircleCI, Jenkins? This skill ships templates for GitHub Actions and GitLab CI.
2. What test types need to run? Unit, integration, E2E, visual, performance? Each has different resource and timing needs.
3. What is the current CI duration? Over 10 minutes means parallelism and sharding are mandatory, not optional.
4. How many developers push per day? High-frequency teams need aggressive concurrency cancellation and caching.
5. What triggers should run which tests? Not every push needs a full E2E suite — map triggers to suites before writing YAML.

Calibrate to team maturity

Set team_maturity in .agents/qa-project-context.md; pick the matching pipeline shape:

• startup — one job: lint + unit + one E2E smoke on PR. Fast feedback over completeness.
• growing — separate jobs for unit, integration, E2E. Parallelization, artifact uploads, result publishing, flaky quarantine.
• established — full matrix: sharded E2E, multi-environment promotion gates, perf and security scans, deploy-gated checks, SLA-backed pipelines.

Core Principles

1. Fast feedback: right tests at the right time. Unit tests on every push (under 2 min). E2E on PRs (under 10 min). Full suite on merge and nightly. The trigger-to-suite map below is the contract.
2. Parallel first: shard tests across workers. A 20-minute serial suite becomes 5 minutes across 4 shards. Always worth the runner cost.
3. Artifacts are evidence. Every run stores traces, screenshots, coverage, and HTML reports. Without artifacts, a CI failure is an undebuggable "reproduce locally" cycle.
4. Flaky tests need quarantine, not retries. Retrying hides the problem — the test passes on retry, the report is green, the race condition persists. Move flaky tests to a non-blocking job, track them, fix the root cause.
5. Quality gates get stricter toward production. Define what must pass at each stage; PR gate is fast and cheap, deploy gate is comprehensive.
6. Read thresholds from config, not from bash. Let the test runner enforce coverage via its own coverageThreshold/thresholds and exit non-zero. Scraping percentages out of stdout with regex is fragile across runner versions.

Pipeline Architecture

Push to branch:   lint+types (30s) → unit (1-2m)
PR opened:        + integration (2-3m) → E2E sharded (5-8m) → merge report
Merge to main:    full E2E ∥ visual ∥ perf budget → deploy (OIDC)
Nightly (cron):   full suite + npm audit + axe a11y + flaky quarantine

What runs when

GitHub Actions

For complete copy-paste workflow files (unit, sharded Playwright E2E, full pipeline, nightly, PR gate), see references/github-actions-templates.md.

Action versions (June 2026)

Pin to the current major and let Dependabot bump them. The actions/* family runs on the Node 24 runner; Node 20 is deprecated on GH-hosted runners.

For supply-chain-sensitive pipelines, pin third-party actions (dorny, marocchino, slackapi, knapsack) to a full-length commit SHA with a version comment, and let Dependabot update the SHA: uses: dorny/test-reporter@<40-char-sha> # v3.0.0. First-party actions/* are lower risk; tags are acceptable there.

Key concepts

Concurrency groups cancel wasted runs when a branch gets multiple pushes:

concurrency:
  group: tests-${{ github.ref }}
  cancel-in-progress: true

Matrix sharding across runners:

strategy:
  fail-fast: false
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: npx playwright test --shard=${{ matrix.shard }}/4

Caching browsers so they aren't re-downloaded every run:

- uses: actions/setup-node@v6
  with: { node-version: 22, cache: npm }

- name: Cache Playwright browsers
  id: playwright-cache
  uses: actions/cache@v5
  with:
    path: ~/.cache/ms-playwright
    key: playwright-${{ runner.os }}-${{ hashFiles('package-lock.json') }}

- name: Install Playwright browsers
  if: steps.playwright-cache.outputs.cache-hit != 'true'
  run: npx playwright install --with-deps chromium

Artifacts for reports and traces, and merging sharded reports into one HTML report — see references/github-actions-templates.md (E2E workflow). The merge job uses actions/download-artifact@v7 with pattern: test-results-* then npx playwright merge-reports --reporter=html.

Smarter sharding at scale

Past 10–15 shards, naïve hash-based splitting wastes runner time on uneven shards. Use a timing-aware balancer:

• `knapsack-pro` — timing-data based, supports Playwright/Jest/Cypress/RSpec; distributes by historical duration.
• CloudBees Smart Tests (formerly Launchable) — ML prioritization + Test Impact Analysis; runs only the tests likely to fail for the diff.
• Datadog Test Optimization — TIA + flake management; shard-balancing by historical time.
• Trunk Flaky Tests — flake-aware quarantine + retry budgeting.

Before reaching for a paid balancer: Playwright's --shard already distributes by file and balances on duration from prior runs. To inspect or feed custom timing data, dump it yourself — npx playwright test --reporter=json | jq '[.suites[].specs[] | {file: .file, duration: .tests[].results[].duration}]'. For Jest, jest-slow-test-reporter surfaces the slowest specs so you can split or fix them.

For self-hosted runners on Kubernetes, use Actions Runner Controller (arc-runner-set / gha-runner-scale-set) — Helm-installed, auto-scales runner pods per workflow. Replaces the deprecated runner-deployment CRD.

Required status checks

Protect main in Settings → Branches → Branch protection rules: enable "Require status checks to pass before merging," add lint, unit-tests, and e2e (all shards) as required checks, and enable "Require branches to be up to date."

GitLab CI

For the full pipeline, see references/gitlab-ci-template.md. Key points:

• Stages [validate, test, e2e, deploy]; node:22-alpine for lint/unit, mcr.microsoft.com/playwright:v1.60.0-noble for E2E (keep this pinned to your installed @playwright/test minor).