Test Harness

Test the Claude Code harness — the hooks, skills, settings, and CLAUDE.md that steer an agent — as the assembled machine it ships as. vigiles gives three tiers, cheapest first; this skill picks the right one, writes the test, and runs it.

The guiding rule: start at the cheapest tier that can answer the question, and climb only when it genuinely can't. Two of the three tiers need no model and no API key, so they run on every commit for free — reach for the paid real-model tier only when the question actually requires a real model.

Step 0 — Pick the tier (the judgment call)

Match what you're testing to the cheapest tier that can answer it:

Most harness questions — block/allow, wired-in, context-landed — never need a model. Only "does the model trigger / behave differently" needs the eval tier.

⚠️ A trigger-rate of 0% on EVERY prompt is a wiring bug until proven otherwise. It reads like a verdict on the description, and three separate setup mistakes produce it: a bare id in fired where the namespaced <plugin>:<skill> is required; pluginDir where a loose .claude/skills needs `skillsDir`; and a missing `fixture`, since a run starts in an empty directory and a prompt about a file that isn't there is one the model is right to decline. Rule all three out before reporting it. (A partial rate is a real number — don't second-guess it.)

Don't tune against a cheaper model until you've checked it's actually a floor. compareContainment(weak, strong) answers that: it reports prompts that fired on the weak model but NOT the strong one, and each one means the weak model is not a lower bound but a _different router_. Prompts that fired only on the strong model are expected and are not a failure. Measured once (21 skills, 84 prompts, haiku vs sonnet): 3 weak-only, and one skill higher on haiku — so containment is not established, which is why the floor stays.

If the unit and deterministic tiers can both answer it, prefer unit: it's faster and reaches events the deterministic mock can't drive.

Step 0.4 — Observing a run, and what it costs

Two questions have their own references — open the one you need, don't guess:

• "What did the run actually DO?" — which tools it called, whether it stayed inside its declared allowed-tools, what it wrote, how to record a call without executing it → `references/observing-a-run.md`
• "Is this free, sub-priced, or does it need a container?" — the three buckets, and what to tell the user after a paid run → `references/cost-and-expectations.md`

Never say "we'll test it" without settling the second one first.

Step 1 — Ensure vigiles is installed

Check whether vigiles is a dependency (package.json), and install it as a dev dependency if not:

npm i -D vigiles    # or: pnpm add -D vigiles / yarn add -D vigiles

The deterministic tier additionally needs the claude CLI on PATH (no API key): npm i -g @anthropic-ai/claude-code. The eval tier needs model auth. If the claude CLI is missing, you can still write and run unit-tier tests.

Step 2 — Locate the harness surface to test

Find what the project actually ships, in this order:

1. .claude/settings.json / .claude/settings.local.json — inline hooks.
2. .claude-plugin/plugin.json — a plugin manifest (hooks, skills, agents, mcpServers).
3. hooks/hooks.json — the plugin hooks convention (e.g. obra/superpowers).
4. skills/<name>/SKILL.md, agents/<name>.md, commands/<name>.md.

Pick one concrete thing to pin down — a specific PreToolUse hook, a specific SessionStart injection, a specific skill.

Step 3 — Write the test for the chosen tier

Per-tier skeletons, and the one mistake that silently swallows failures (a hand-rolled runner eats stderr) → `references/writing-tests.md`

Read it before writing the file — the skeleton differs per tier, and the runner warning has cost real debugging time.

Step 4 — Run it

In a runner (node:test / vitest / jest) the tests are plain async functions. Or use the zero-setup CLI, which discovers and runs the files:

npx vigiles test                 # *.harness.{mjs,ts} — unit + deterministic, no API key
npx vigiles eval --trials=6      # *.eval.{mjs,ts} — real model (local / nightly, not CI)

Unit-tier runHook tests need no claude and always run — write and run them even with no claude installed. A tier that genuinely can't run reports a loud ⊘ SKIPPED (tallied separately, never a fake ✓); a standalone script emits one via skip(reason) from vigiles. A skip passes by default, but in a CI job that asserts the capability is present, run `vigiles test --no-skip` so a skipped tier fails — a green-with-skips is untested surface. Keep unit + deterministic tests in CI (free); run evals locally or on a schedule with auth.

CI — don't hand-write the steps

These tiers belong in CI, and there is a published Action for it. Run vigiles init: it writes .github/workflows/vigiles.yml, wiring the Action (zernie/vigiles@v1) for the jobs