Terraform Skill

Terraform Release and Provisioner Safety

Prevent a valid-looking Terraform workflow from publishing unvalidated bytes or widening a change's blast radius. Keep the user's business outcome and the actual mutation surface ahead of plan counts, green wrappers, or process completeness.

Route the task

• Release, shared gateway, saved plan, staging receipt, or production promotion: read release-safety-and-environment-parity.md.
• A second environment, environment retirement, shared data backend, DNS ownership, state, or snapshots: read multi-env-isolation.md.
• Pre-deploy checks or a validator: read pre-deploy-validation.md.
• Fresh instance or empty data disk: read zero-to-deploy-checklist.md.
• One known provisioner symptom: use the matching pattern below.

Operating contract

1. Use the repository's canonical wrapper when it has one. Do not bypass it with a raw Terraform, SSH, SCP, helper-script, or console path because a gate rejects the planned release.
2. Prefer provider resources, image baking, cloud-init, or configuration management. HashiCorp recommends exhausting purpose-built alternatives because Terraform cannot model provisioner side effects predictably. When a provisioner remains necessary, make its artifact, target, lock, validation, and readback explicit.
3. Inventory every resource or recovery tool that can write the target runtime. A validator attached to only one writer does not protect another writer of the same shared service.
4. Give staging and production one required-key schema. Let values differ; never let a key be required in one environment and optional, defaulted, absent, or allowed-empty in another.
5. Validate the exact candidate bytes, Compose-rendered environment, and immutable runtime image before the first live write or restart. Keep post-deploy checks too: they detect damage but cannot prevent the first bad mutation.
6. Treat a saved plan as an executable artifact. Bind it to reviewed source/artifact identity and apply that exact file. A successful apply is not a staging receipt; record promotion evidence only after every required live verifier succeeds.
7. Require an explicit production decision at the last reversible point. A deadline, PLAN_DIGEST, CONFIRM_*, or agent inference is not production authorization.
8. Stop after the requested result is verified. Do not turn a single-service fix into full-stack drift reconciliation, recovery redesign, or unrelated hardening.

Provisioner traps (symptom → fix)

Use these incident-derived symptom patterns to choose the next falsifying check. Confirm the current source and runtime before promoting a historical cause into the present diagnosis.

docker: not found in remote-exec

cloud-init still installing Docker when provisioner SSHs in.

provisioner "remote-exec" {
  inline = [
    "cloud-init status --wait",
    "command -v docker >/dev/null || { echo 'FATAL: Docker not ready'; exit 1; }",
  ]
}

rsync: connection unexpectedly closed in local-exec

Do not infer a universal Terraform limitation from this symptom. A second SSH client can lose to the target's connection budget, SSH policy, or a competing deploy. Keep local-exec local: package an immutable artifact there, then use a Terraform-managed upload or a purpose-built deploy system. Give every apply a unique remote staging path; never share /tmp/src.tar.gz across concurrent applies.

provisioner "local-exec" {
  command = "tar czf /tmp/src-${self.id}.tar.gz --exclude=node_modules --exclude=.git -C ${path.module}/../../.. myproject"
}
provisioner "file" {
  source      = "/tmp/src-${self.id}.tar.gz"
  destination = "/tmp/src-${self.id}.tar.gz"
}
provisioner "remote-exec" {
  inline = ["tar xzf /tmp/src-${self.id}.tar.gz -C /data/ && rm -f /tmp/src-${self.id}.tar.gz"]
}

macOS BSD tar: --exclude must come BEFORE the source argument.

cloud-init status shows "running" forever

apt-get -y does not suppress debconf dialogs. Packages like iptables-persistent block on TTY prompts.

- |
    echo iptables-persistent iptables-persistent/autosave_v4 boolean true | debconf-set-selections
    echo iptables-persistent iptables-persistent/autosave_v6 boolean true | debconf-set-selections
    DEBIAN_FRONTEND=noninteractive apt-get install -y iptables-persistent

Known offenders: iptables-persistent, postfix, mysql-server, wireshark-common.

EACCES: permission denied in container logs, container Restarting

Host volume dirs are root-owned; container runs as non-root (uid 1001). Fix before docker compose up:

mkdir -p /data/myapp/data /data/myapp/logs
chown -R 1001:1001 /data/myapp/data /data/myapp/logs

Find UID: grep adduser.*-u or USER in Dockerfile.

Provisioner fails but no diagnostic output

Keep fail-fast behavior; attach diagnostics to failure instead of disabling set -e. Otherwise an early failed command can be overwritten by a later green health check.

provisioner "remote-exec" {
  inline = [
    "set -eu",
    "trap 'rc=$?; if [ $rc -ne 0 ]; then docker logs myapp --tail 20 2>&1 || true; docker ps --format \\\"table {{.Names}}\\\\t{{.Status}}\\\" || true; fi; exit $rc' EXIT",
    "docker compose up -d",
    "sleep 15",
    "docker ps --filter name=myapp --format '{{.Status}}' | grep -q healthy || exit 1",
  ]
}

Container Restarting — database tables missing

DB migrations not in provisioner. PostgreSQL docker-entrypoint-initdb.d only runs on empty data dir. Explicitly create DB + run migrations:

# After postgres healthy:
docker exec pg psql -U postgres -tc "SELECT 1 FROM pg_database WHERE datname='mydb'" | grep -q 1 \
  || docker exec pg psql -U postgres -c "CREATE DATABASE mydb;"

# Idempotent migrations:
for f in migrations/*.sql; do
  VER=$(basename $f)
  APPLIED=$($PSQL -tAc "SELECT 1 FROM schema_migrations WHERE version='$VER'" | tr -d ' ')
  [ "$APPLIED" = "1" ] && continue
  { echo 'BEGIN;'; cat $f; echo 'COMMIT;'; } | $PSQL
  $PSQL -tAc "INSERT INTO schema_migrations(version) VALUES ('$VER') ON CONFLICT DO NOTHING"
done

Compose uses an unexpected value despite .env

Compose interpolation gives the invoking shell higher precedence than --env-file or project .env. An old exported value can therefore override the reviewed environment silently. Inspect what Compose actually used; unset ambient overrides when the env file is meant to be authoritative.

# Inspect interpolation inputs and the rendered model.
docker compose --env-file .env config --environment
docker compose --env-file .env config --format json > compose.rendered.json

# Make the reviewed env file authoritative for this key.
env -u DOCKER_WITH_PROXY_MODE docker compose --env-file .env build

TLS handshake fails: Invalid format for Authorization header

Caddy's Cloudflare DNS module expects a scoped API Token through Bearer authentication. Do not infer credential type, validity, or permissions from length/prefix alone. Verify the token with Cloudflare's official endpoint, then exercise the exact zone operation or provider path required by the release.

curl -fsS https://api.cloudflare.com/client/v4/user/tokens/verify \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  | jq -e '.success == true and .result.status == "active"' >/dev/null

If the credential is absent or wrong, create a least-privilege API Token through Cloudflare's current dashboard/API flow and grant only the zones/operations the provider needs. Follow the official creation contract rather than copying permission-group IDs that may drift: <https://developers.cloudflare.com/fundamentals/api/get-started/create-token/>.

TLS fails on staging but works on production — hardcoded domains