Cargo Observability

Cargo CLI — Observability

Alerts. An alert is a scheduled threshold check. On every cron tick it measures a scope (what to watch), compares the measured value against a threshold (the breach condition), and on breach fires actions — each as its own run — and records an event. This is the proactive counterpart to cargo-diagnostics: diagnostics explains a failure after you notice it; an alert tells you the moment a metric crosses a line.

Everything lives under one CLI domain:

cargo-ai observability alert   …   # the alert CRUD + preview surface
cargo-ai observability event   …   # an alert's firing history

Bootstrap

Already signed in (cargo-ai whoami returns a workspace)? Skip to the next section.

npm install -g @cargo-ai/cli            # no global install? prefix every command with `npx @cargo-ai/cli`
cargo-ai login --email you@company.com  # emailed code, no browser; creates the account on first use
                                        # alternatives: --oauth (browser) · --token <api-token> (CI)
cargo-ai whoami                         # confirm the active workspace before any write

Every command prints JSON to stdout; failures exit non-zero with {"errorMessage": "..."}. Anything that creates a run or a batch is async — pass --wait-until-finished or poll the matching get. Alerts are guarded by observability:read / observability:write permissions. If a create/update/remove returns a permission error, the token lacks observability:write — use an admin token or have one granted (`../cargo-workspace-management/SKILL.md`). When the full skill bundle is installed, `../cargo/references/prerequisites.md` adds the CLI version pin, token scopes, and the admin-only surface.

The three moving parts of every alert

The scope and threshold are a matched pair — a metric can only be computed over the scopes that produce it (e.g. errorRate needs telemetry, freshness needs a model). The full compatibility matrix, every metric's meaning and units, and every scope filter field are in [`references/scopes-and-thresholds.md`](references/scopes-and-thresholds.md) — read it before writing a --scope/--threshold pair you haven't used before.

The golden rule: preview before you create

alert preview evaluates a scope + threshold right now, without firing actions or writing an event. It returns the value the alert would measure and whether that value breaches — so you calibrate the threshold against reality instead of guessing, and you confirm the scope/threshold pairing is even valid before committing it to a schedule.

cargo-ai observability alert preview \
  --scope '{"kind":"runs","workflowUuid":"<uuid>","statuses":["error"]}' \
  --threshold '{"metric":"errorRate","operator":"gte","value":10}' \
  --window-minutes 1440          # last 24h; default 60

• outcome: "computed" → { value, total, failed, isBreached }. Set your threshold from value.
• outcome: "empty" → the window had nothing to measure (see the empty-vs-zero rule in references/alert-lifecycle.md).
• outcome: "notComputed" → { errorMessage }. A bad SQL query, a deleted model, or an invalid scope/threshold pairing all land here — fix it before creating.

--window-minutes only shapes the window for telemetry scopes (spans/runs/records). A model is measured as it stands right now; a query scope windows itself in its SQL.

Always preview first. It is free, it is the only way to size a threshold correctly, and it catches an invalid pairing before it becomes a schedule that writes an error event every tick.

Commands

All commands output JSON. Reads need a token with observability:read; create/update/remove need observability:write (an admin token has both; a plain member token may not — see Bootstrap above).

Create an alert

cargo-ai observability alert create \
  --name "CRM sync error rate" \
  --description "Page when the HubSpot sync starts failing" \
  --cron "*/30 * * * *" \
  --scope '{"kind":"runs","workflowUuid":"<workflow-uuid>","statuses":["error"]}' \
  --threshold '{"metric":"errorRate","operator":"gte","value":10}' \
  --actions '[{"kind":"agent","agentUuid":"<agent-uuid>","config":{"message":"{{alert.name}} breached: {{event.value}}% errors. {{alert.url}}"}}]'

• --cron — 5-field cron or @every <interval> (e.g. @every 30m), always UTC, at most once a minute. The UI presets bottom out at 30 minutes; go tighter only with reason (every tick scans ClickHouse and can fire paid runs).
• --disabled — create it paused (evaluate nothing until you update --enabled true).
• --folder <uuid> — file it under a folder (from cargo-workspace-management).
• --actions — optional. Omit for a silent alert. Each entry is a configured action: unlike orchestration action execute, which carries no config at all, an alert action requires one — that is where the templated message lives. The config is templated against the firing context ({{alert.*}}, {{event.*}}) — see `references/alert-lifecycle.md` for the full variable list. Each action's target (agentUuid/toolUuid/connectorUuid) is validated to exist in the workspace at create time.

List, get, update, remove

cargo-ai observability alert list                      # all alerts, each with its lastEvent
cargo-ai observability alert get <uuid>                # one alert + its lastEvent

cargo-ai observability alert update --uuid <uuid> \
  --enabled false                                      # pause it (true/false — must be literal)
cargo-ai observability alert update --uuid <uuid> \
  --threshold '{"metric":"errorRate","operator":"gte","value":20}'   # raise the bar
cargo-ai observability alert update --uuid <uuid> \
  --description none                                    # "none" clears; --folder none unfiles

cargo-ai observability alert remove <uuid>

--enabled is strict: only the literal true or false are accepted — --enabled yes is rejected rather than silently disabling the alert. On update, any flag you omit is left unchanged; --description none / --folder none are the explicit "clear it" spellings.

Inspect firing history

cargo-ai observability event list <alertUuid>          # latest evaluation events, newest first

Each event carries status (healthy / unhealthy / error), the measured value, a snapshot of the scope/threshold/actions as they were when it fired (the alert can change afterwards), runUuids (the runs the actions spawned — feed these to cargo-diagnostics or orchestration run get), the evaluation window, and errorMessage for error events. unhealthy = breached and fired; error = the metric could not be computed.

How evaluation actually works