Test Coverage

Test Coverage Analysis

A reference manual for choosing, applying, and interpreting test-coverage analyses on an existing test suite.

This skill is a knowledge reference, not a procedure. It does not tell you when to write tests or which test types to design — that is the job of design-testing-strategy. It tells you, once tests exist, which mechanical signal best measures what those tests do (and do not) exercise, and how to read that signal honestly.

What Coverage Analysis Is

Test coverage analysis is the post-hoc measurement of how thoroughly a test suite exercises a software artifact along one or more axes. It answers the question "what did my tests actually touch?" — for some specific definition of "touch."

The word "coverage" is overloaded. It can mean any of:

• Structural / code coverage — which lines, statements, branches, conditions, or paths in the source code were executed (measured by instrumentation).
• Mutation coverage — what proportion of deliberately-injected source faults the test suite detects (measured by re-running the suite against mutated code).
• Requirements / feature coverage — which acceptance criteria, user stories, or specification clauses have at least one verifying test (measured by traceability).
• API / integration coverage — which endpoints, methods, status codes, contract interactions, and schema fields are exercised (measured by request/response inspection).
• Specification-domain coverage — equivalence classes, boundary values, parameter combinations, state transitions, error paths (measured by analyzing test inputs against a model).

Category correction: coverage is not a test type

Mutation testing, MC/DC, branch coverage, RTM linkage, contract coverage, and schema coverage are measurements about an existing test suite. They are not test types in the way unit, integration, e2e, contract, or smoke tests are.

• "Should I write a unit test or a mutation test?" is a malformed question. The correct framing is: "I already have unit/integration tests; should I additionally run mutation analysis against them?"
• Mutation tools generate variants of the source and re-execute the existing suite. They produce a score, not new tests.
• MC/DC and branch coverage are reports computed from instrumented runs of the existing suite.
• RTM linkage is a property of test metadata (tags, IDs), not a separate execution.

If a "test strategy" places mutation testing alongside unit / integration / e2e, that strategy has confused what to test with how to measure the tests. The two questions are orthogonal.

The asymmetry principle

Low coverage is strong evidence of weak testing. High coverage is weak evidence of strong testing.

Coverage is necessary-but-not-sufficient. A test can execute a line without asserting anything meaningful; 100% line coverage is routinely achievable with zero assertions (thinkinglabs.io, codeintelligently.com). Use coverage as a tripwire, not a trophy. Once a coverage percentage becomes a target, it ceases to be a good metric (Goodhart's law applied to testing; see Optivem Journal).

What coverage analysis is NOT

• NOT a measure of test quality. Lines can execute without assertions.
• NOT a measure of correctness. Coverage proves the test ran, not that it would have failed on a bug.
• NOT a synonym for "well tested". Mutation testing routinely refutes 100%-coverage-with-no-assertions suites.
• NOT a substitute for risk-based test selection per ISO/IEC/IEEE 29119.
• NOT a target. Treat as a floor and a trend, never as the goal itself.

Per-Type Structure

Every coverage type in this skill is documented in the same six sub-fields, in this order:

1. Definition — what it measures.
2. What it does NOT measure — its limits / blind spots.
3. Typical tools — per ecosystem.
4. When to use vs skip — applicability heuristics.
5. Targets / thresholds & pitfalls — defensible numeric ranges (always with the risk caveat) and common gaming patterns.
6. Cost-benefit ROI — order-of-magnitude cost vs the signal you actually buy.

Scan any section by these headings.

Structural / Code Coverage

Measured by instrumenting the compiled or interpreted program and recording which structural elements (lines, statements, branches, conditions, paths) the test suite executes.

Line / Statement Coverage

• Definition. Percentage of source-code lines (or statements) executed at least once.
• What it does NOT measure. Whether branches were taken in both directions. Whether assertions verified the result. Whether boundary values were tested. Multiple statements on one line distort the metric (Metridev).
• Typical tools.

• When to use vs skip. Always-on; cost is near-zero (a CI flag). Never use as a quality goal in itself.
• Targets / thresholds & pitfalls. 70–85% is typical for general-purpose code (Qt blog). Apply only with the risk caveat (see Risk-Based Interpretation below). Tests with no assertions still count lines as covered. Single-line if (x) doA(); else doB(); shows 100% statement coverage with only one branch exercised. Snapshot-only tests inflate numbers without verifying behavior.
• Cost-benefit ROI. Very high — cost near-zero, value is a tripwire on regression in test reach.

Branch / Decision Coverage

• Definition. Percentage of decision branches (true/false outcomes of if, while, for, ?:, switch cases) executed.
• What it does NOT measure. Compound-condition independence (A && B taken true might never test A=true, B=false). Order of evaluation. Loop iteration counts. Assertion strength.
• Typical tools. Same as line coverage; enable with --branch (coverage.py), branch mode (Istanbul is branch-aware by default), JaCoCo reports branches natively. Strictly stronger than line/statement coverage (Graph AI).
• When to use vs skip. Default for any non-trivial logic. Prefer branch over line as the primary structural metric.