Aws Observability

AWS Observability

Overview

Domain expertise for AWS observability across metrics, logs, and traces, for two products that share the CloudWatch name but are separate services with separate control planes, data models, and APIs:

Enabling Omni does not replace CloudWatch; log groups, metrics, and alarms keep working, and most customers use both.

Works best with the AWS MCP server — enables running CLI commands, querying CloudWatch, and validating configurations directly. All guidance also works with standard AWS CLI access.

Note: Reference files contain specific runtime versions, quota values, and feature matrices that may change. When precision matters (e.g. deploying to production, choosing a runtime, or checking a quota), confirm values against current AWS documentation rather than relying solely on the values in these files.

Step 0 — CloudWatch or CloudWatch Omni?

Decide this before routing. The natural wording ("set up an alert for high latency", "build a dashboard", "query my logs") does not say which product the customer means.

1. The customer names the product — "Omni", "Application Observability", "Agent Observability", a Space, Domain, Dataset, access grant, spaceId, cloudwatch-omni, Omni SQL, PromQL, views, context graph, evaluators → Omni. "Log Insights", "log group", "metric namespace", "CloudWatch alarm", "metric alarm", "composite alarm", "anomaly alarm", "X-Ray", "Application Signals", "canary", "CloudTrail", "Dynamic Instrumentation" → CloudWatch. A bare "alarm" (or "alert") with no other product signal is ambiguous — fall through to rule 3 and probe: a Space → Omni alert (alerts.md); no Space → CloudWatch alarm (cloudwatch/alarms.md). Exception: a "PromQL alarm" is a CloudWatch alarm on OTel metrics (cloudwatch/alarms.md) — "alarm" wins over "PromQL".
2. Knowledge or how-to question ("what is an Omni alert", "does Omni have an API", "how do alerts differ from alarms") → answer from the reference files directly. Do not probe the account, and do not divert to the other product. Whether a capability exists is a fact about the product, not the account. concepts.md carries the full feature-equivalence matrix.
3. Request that must act on live data, and the wording is ambiguous → probe the target Region before acting:

   aws___call_aws → aws cloudwatch-omni list-domains
   aws___call_aws → aws cloudwatch-omni list-spaces        # scope to the target Region

• A Domain and a Space exist in that Region → Omni.
• No Space → CloudWatch: alarms → cloudwatch/alarms.md, dashboards → cloudwatch/dashboards.md, queries → cloudwatch/log-insights.md, metrics → cloudwatch/metrics.md. If the customer explicitly asked for Omni and has no Space, that is a first-time setup — see step 4.
• The probe itself errors ("not yet supported", unknown service, endpoint does not resolve) → the CLI/SDK model in use lacks cloudwatch-omni. That is not evidence Omni is absent and must not be reported as "Omni is unavailable". The customer's installed AWS CLI/SDK most likely predates the service. If the request carried any Omni signal, have them upgrade (AWS CLI v2 reinstall or brew upgrade awscli; pip install -U boto3 botocore) and re-run aws cloudwatch-omni list-domains — exact steps in programmatic-access.md; never substitute a CloudWatch or X-Ray command for an Omni request. If the request carried no Omni signal, do not block on the upgrade: proceed on the CloudWatch path (the pre-Omni default) and mention the upgrade only in passing.
• Still inconclusive → ask the customer.

1. First-time Omni setup — creating a Domain or Space, granting access, provisioning ingestion, forwarding log groups into the Dataset, connecting Slack, or instrumenting an application or AI agent so traces reach a Space → STOP and route to the `setting-up-cloudwatch-observability` skill. This skill covers a Space that already has data. An instrumentation / ADOT / OTel-collector request that names neither Application Signals, ServiceEvents, or the amazon-cloudwatch-observability add-on nor Omni or a Space is ambiguous — probe list-spaces in the target Region: a Space → route to the setting-up-cloudwatch-observability skill's application-instrumentation reference (plain ADOT SDK, no add-on); no Space → cloudwatch/application-signals-onboarding.md.

A Space is one per account per Region — probe the Region the request targets. An Omni query against the wrong Region returns an empty result that is easily misread as "no data".

Routing — CloudWatch (references/cloudwatch/)