Querying Aws Cloudwatch

Query AWS CloudWatch System Tables

Overview

Works best with the AWS MCP server for sandboxed execution and audit logging. All commands below use the AWS CLI and work in any environment with configured AWS credentials.

The CloudWatch Logs S3 Tables integration exports log data as Apache Iceberg tables in the AWS-managed aws-cloudwatch table bucket. This enables SQL analysis via Amazon Athena and correlation of log data with non-CloudWatch data (S3 metadata, business tables, etc.). Available at no additional storage charge beyond CloudWatch ingestion pricing.

Decision Tree

Supported Data Sources

The following data sources are available through the S3 Tables integration. Each data source has a namespace pattern used in SQL queries. Not all AWS vended data sources may be available in all Regions; check the CloudWatch console Data Sources tab for current availability.

Note: This table lists the 24 most commonly queried data sources. The integration supports 43+ AWS vended data sources in total. Use list-namespaces on the aws-cloudwatch bucket to discover all available data sources in your account. Namespace patterns follow the convention <service>__<type>.

Common Tasks

1. Check If Configured

# Check if the aws-cloudwatch table bucket exists
aws s3tables list-table-buckets --region <REGION> \
  --query "tableBuckets[?name=='aws-cloudwatch']"

• Empty result → integration not enabled. Guide user through setup.
• Bucket exists but no namespaces → integration enabled but no log data yet (only captures events after association).

List available tables:

aws s3tables list-namespaces --table-bucket-arn arn:aws:s3tables:<REGION>:<ACCOUNT>:bucket/aws-cloudwatch --region <REGION>

aws s3tables list-tables --table-bucket-arn arn:aws:s3tables:<REGION>:<ACCOUNT>:bucket/aws-cloudwatch --namespace <NAMESPACE> --region <REGION>

2. Enable / Configure

Create integration:

aws observabilityadmin create-s3-table-integration \
  --region <REGION> \
  --encryption '{"SseAlgorithm": "aws:kms", "KmsKeyArn": "<KMS_KEY_ARN>"}' \
  --role-arn <SERVICE_ROLE_ARN>

Associate a specific data source (recommended):

aws logs associate-source-to-s3-table-integration \
  --region <REGION> \
  --integration-arn <INTEGRATION_ARN> \
  --data-source '{"name": "<source-name>", "type": "<source-type>"}'

Associate all data sources (wildcard):

⚠️ Warning: Wildcard association delivers all current and future data sources to S3 Tables. Use specific associations for tighter control over what log data lands in queryable tables.

aws logs associate-source-to-s3-table-integration \
  --region <REGION> \
  --integration-arn <INTEGRATION_ARN> \
  --data-source '{"name": "*", "type": "*"}'

For IAM requirements (service role trust policy, permissions policy, condition keys), see Security Considerations below.

3. Verify Permissions for Querying

Requires:

• S3 Tables federated catalog registered in Glue (s3tablescatalog)
• Lake Formation SELECT + DESCRIBE grants on the table (or IAM-only mode in supported regions)
• Athena execution permissions

Grant access:

aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=<ROLE_ARN> \
  --resource '{"Table": {"CatalogId": "<ACCOUNT>:s3tablescatalog/aws-cloudwatch", "DatabaseName": "<NAMESPACE>", "Name": "<TABLE>"}}' \
  --permissions DESCRIBE SELECT \
  --region <REGION>

4. Query

Query syntax:

"s3tablescatalog/aws-cloudwatch"."<namespace>"."<table>"

Constraints:

• You MUST ALWAYS run get-tables on the target namespace and include the command in your response before writing any SQL query — schemas vary by data source. Never skip this step even if you already know the likely schema. Run get-tables once on the target namespace (one call returns all tables + columns + types + descriptions):

  aws glue get-tables --catalog-id "<ACCOUNT>:s3tablescatalog/aws-cloudwatch" --database-name "<namespace>" --region <REGION>

• You MUST confirm workgroup and output location before executing
• You MUST inform user that only logs received after association are available (no backfill)

Example — VPC Flow Logs rejected traffic:

SELECT srcaddr, dstaddr, dstport, protocol, packets, bytes
FROM "s3tablescatalog/aws-cloudwatch"."amazon_vpc__flow"."<table>"
WHERE action = 'REJECT'
ORDER BY bytes DESC
LIMIT 50;

Example — WAF blocked requests:

SELECT timestamp, action, terminatingRuleId, httpSourceId
FROM "s3tablescatalog/aws-cloudwatch"."aws_waf__logs"."<table>"
WHERE action = 'BLOCK'
ORDER BY timestamp DESC
LIMIT 50;

Example — correlate VPC Flow Logs with S3 object metadata:

SELECT f.srcaddr, f.dstaddr, f.bytes, j.key, j.record_type
FROM "s3tablescatalog/aws-cloudwatch"."amazon_vpc__flow"."<table>" f
JOIN "s3tablescatalog/aws-s3"."b_<bucket>"."journal" j
  ON f.srcaddr = j.source_ip_address
WHERE j.record_type = 'CREATE'
  AND f.action = 'ACCEPT';

Key Behaviors

• No backfill — only new log events after association are delivered to S3 Tables
• Retention follows log group — when log group retention expires, data is removed from the table