Clickstack Otel Collector

Set up an OpenTelemetry collector for Managed ClickStack

This skill wires an OpenTelemetry collector into a Managed ClickStack service running on ClickHouse Cloud, sends rich synthetic telemetry through it, and confirms the data is actually visible in ClickStack. It uses `clickhousectl` for all cloud and SQL operations.

Scope. This skill supports two paths, chosen in Step 0:

1. Deploy a new collector locally. You can do this two ways: individual `docker` commands, or a `docker compose` file (recommended, fewer commands and one file to start/stop). Make the user aware of both up front and let them pick in Step 0; do not assume plain docker. Either way runs the ClickStack distribution of the collector, preconfigured for Managed ClickStack.
2. Configure your own existing collector by adding the ClickHouse exporter configuration. We give you the exact config to drop in; you reload your collector. Use this if you already run a collector in a gateway role.

A full Kubernetes deployment (Helm, secrets in K8s Secrets) is out of scope here; the config we generate in path 2 can be applied to a collector running anywhere.

The end state is:

• A dedicated hyperdx_ingest SQL user on the target service, with exactly the grants the collector needs (it creates the otel.* schema on first write).
• A collector forwarding logs, traces, and metrics into the otel database on the service, either the new local ClickStack collector or your existing one.
• Rich synthetic telemetry across several services, severities, span statuses, and metric types, so ClickStack's Search, Service Map, and dashboards have something real to show.
• The service confirmed awake, and the user walked through the ClickStack onboarding in the Cloud console so they can actually see their data.

Secrets (the OTLP auth token and the SQL password) are generated locally, written once to a 0600 env file, and passed to Docker via --env-file. They are never pasted into the chat, never passed with docker run -e, and never echoed back after creation.

Follow these steps in order. Each step depends on state established by the previous one.

Step 0: Choose your path

Ask the user two short questions before doing anything else, because they determine which later steps run.

Question 1: Do you already have an OpenTelemetry collector running in a gateway role?

• No, set one up for me. -> the new-collector path. Continue to Question 2.
• Yes, I have one. -> the existing-collector path. Skip Question 2 (it does not apply), and in Step 6 you will configure their collector rather than deploy a new one.

Question 2 (new-collector path only): Run the collector with individual Docker commands, or a Docker Compose file?

• Docker Compose (recommended). Fewer commands, one file to start and stop, easiest to re-run. Best if docker compose is available.
• Individual Docker commands. Use if Compose is not installed or you prefer explicit commands.

Record the answers as COLLECTOR_PATH (new or existing) and, for the new path, DEPLOY_MODE (compose or run). Refer back to them in Step 6 and Step 7.

Step 1: Batch the permissions up front

Coding agents prompt for approval the first time they see each shell command. To avoid interrupting the user every few steps, ask them once, up front, to allowlist the command prefixes below (the "always allow for this project / session" option in their agent). There are no destructive operations and nothing targets anything outside this project or their ClickHouse Cloud service.

Tell the user, in your own words: "If your agent supports it, choose 'always allow' for each of these the first time it asks. The whole run is read-only against your machine except for the collector container, and write operations against ClickHouse are limited to creating the ingest user and the `otel` schema."

If the user is on the existing-collector path and does not want to run the optional telemetry check, you can drop docker and curl from the list.

Two approvals are semantic, not prefix-based, so allowlisting won't pre-clear them. Warn the user to expect these and approve them explicitly when they appear:

• The clickhousectl install in Step 3 uses curl … | sh, which many agent sandboxes flag as "downloading and running untrusted code" regardless of any curl allowlist rule.
• The CREATE USER / GRANT in Step 5 may be flagged as "modifying shared production infrastructure," again independent of the clickhousectl prefix rule.

Neither is solved by the table above; they are one-time, intentional, and safe to approve.

Then continue.

Step 2: Confirm the target service and lay down the secrets file

The user's prompt contains a service identifier, either a service ID (UUID) or a service name. Treat that value as SERVICE_REF.

Create a working directory and a `0600` env file that will hold all configuration and secrets for this run. The key names match exactly what the collector image reads, so this same file is passed straight to docker run --env-file (or referenced by Compose) in Step 6. Write it under a tight umask so the secret is never briefly world-readable:

WORKDIR="${WORKDIR:-$HOME/clickstack-otel-collector}"
mkdir -p "$WORKDIR" && chmod 700 "$WORKDIR"
ENV_FILE="$WORKDIR/collector.env"

# Generate secrets WITHOUT printing them; write straight into a private file.
( umask 177
  {
    echo "SERVICE_REF=$SERVICE_REF"
    echo "OTLP_AUTH_TOKEN=$(openssl rand -hex 32)"
    echo "CLICKHOUSE_USER=hyperdx_ingest"
    echo "CLICKHOUSE_PASSWORD=$(openssl rand -hex 24)Aa1-"
    echo "HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=otel"
  } > "$ENV_FILE"
)
chmod 600 "$ENV_FILE"
ls -l "$ENV_FILE"

Two things about these values matter and are easy to get wrong:

• Key names are exact. The collector reads CLICKHOUSE_USER, CLICKHOUSE_PASSWORD, CLICKHOUSE_ENDPOINT, and HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE. Store the SQL password under CLICKHOUSE_PASSWORD (not a custom name); if it is missing, the collector starts with an empty password and dies with code: 516, Authentication failed.
• The password charset is constrained from three directions at once. ClickHouse Cloud rejects passwords without at least one uppercase character and one special character, so a plain hex string fails at CREATE USER. At the same time, the collector's migration tool embeds the password in a connection URL, so @, :, /, ?, #, and % corrupt it (symptom: code: 516 at startup even though the password is "correct"). The recipe above is random hex (lowercase + digits) plus the suffix Aa1-, which adds the required uppercase, a digit, and a URL-unreserved special character (-). The OTLP token has no such rules (it is just a bearer token), so plain hex is fine for it.

The env file uses bare `KEY=VALUE` lines with no quotes: Docker's --env-file does not do shell parsing, so any quotes you add become part of the value.

On the existing-collector path the OTLP_AUTH_TOKEN is not used by your collector (auth on your receiver is your own setup); it is generated only so the same file works if you later switch