Observability Monitoring

Observability & Monitoring

A comprehensive skill for implementing production-grade observability and monitoring using Prometheus, Grafana, and the wider cloud-native monitoring ecosystem. This skill covers metrics collection, time-series analysis, alerting, visualization, and operational excellence patterns.

When to Use This Skill

Use this skill when:

• Setting up monitoring for production systems and applications
• Implementing metrics collection and observability for microservices
• Creating dashboards and visualizations for system health monitoring
• Defining alerting rules and incident response automation
• Analyzing system performance and capacity using time-series data
• Implementing SLIs, SLOs, and SLAs for service reliability
• Debugging production issues using metrics and traces
• Building custom exporters for application-specific metrics
• Setting up federation for multi-cluster monitoring
• Migrating from legacy monitoring to cloud-native solutions
• Implementing cost monitoring and optimization tracking
• Creating real-time operational dashboards for DevOps teams

Core Concepts

The Four Pillars of Observability

Modern observability is built on four fundamental pillars:

1. Metrics: Numerical measurements of system behavior over time

• Counter: Monotonically increasing values (requests served, errors)
• Gauge: Point-in-time values that go up and down (memory usage, temperature)
• Histogram: Distribution of values (request duration buckets)
• Summary: Similar to histogram but calculates quantiles on client-side

1. Logs: Discrete events with contextual information

• Structured logging (JSON, key-value pairs)
• Centralized log aggregation (ELK, Loki)
• Correlation with metrics and traces

1. Traces: Request flow through distributed systems

• Span: Single unit of work with start/end time
• Trace: Collection of spans representing end-to-end request
• OpenTelemetry for distributed tracing

1. Events: Significant occurrences in system lifecycle

• Deployments, configuration changes
• Scaling events, incidents
• Business events and user actions

Prometheus Architecture

Prometheus is a pull-based monitoring system with key components:

Time-Series Database (TSDB)

• Stores metrics as time-series data
• Efficient compression and retention policies
• Local storage with optional remote storage

Scrape Targets

• Service discovery (Kubernetes, Consul, EC2, etc.)
• Static configuration
• Relabeling for flexible target selection

PromQL Query Engine

• Powerful query language for metrics analysis
• Aggregation, filtering, and mathematical operations
• Range vectors and instant vectors

Alertmanager

• Alert rule evaluation
• Grouping, silencing, and routing
• Integration with PagerDuty, Slack, email, webhooks

Exporters

• Bridge between Prometheus and systems
• Node exporter, cAdvisor, custom exporters
• Third-party exporters for databases, services

Metric Labels and Cardinality

Labels are key-value pairs attached to metrics:

http_requests_total{method="GET", endpoint="/api/users", status="200"}

Label Best Practices:

• Use labels for dimensions you query/aggregate by
• Avoid high-cardinality labels (user IDs, timestamps)
• Keep label names consistent across metrics
• Use relabeling to normalize external labels

Cardinality Considerations:

• Each unique label combination = new time-series
• High cardinality = increased memory and storage
• Monitor cardinality with prometheus_tsdb_symbol_table_size_bytes
• Use recording rules to pre-aggregate high-cardinality metrics

Recording Rules

Pre-compute frequently-used or expensive queries:

groups:
  - name: api_performance
    interval: 30s
    rules:
      - record: api:http_request_duration_seconds:p99
        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
      - record: api:http_requests:rate5m
        expr: rate(http_requests_total[5m])

Benefits:

• Faster dashboard loading
• Reduced query load on Prometheus
• Consistent metric naming conventions
• Enable complex aggregations

Service Level Objectives (SLOs)

Define and track reliability targets:

SLI (Service Level Indicator): Metric measuring service quality

• Availability: % of successful requests
• Latency: % of requests under threshold
• Throughput: Requests per second

SLO (Service Level Objective): Target for SLI

• 99.9% availability (43.8 minutes downtime/month)
• 95% of requests < 200ms
• 1000 RPS sustained

SLA (Service Level Agreement): Contract with consequences

• External commitments to customers
• Financial penalties for SLO violations

Error Budget: Acceptable failure rate

• Error budget = 100% - SLO
• 99.9% SLO = 0.1% error budget
• Use budget for innovation vs. reliability tradeoff

Prometheus Setup and Configuration

Basic Prometheus Configuration

# prometheus.yml
global:
  scrape_interval: 15s      # Default scrape interval
  evaluation_interval: 15s  # Alert rule evaluation interval
  external_labels:
    cluster: 'production'
    region: 'us-west-2'

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

# Load rules
rule_files:
  - 'rules/*.yml'
  - 'alerts/*.yml'

# Scrape configurations
scrape_configs:
  # Prometheus self-monitoring
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node exporter for system metrics
  - job_name: 'node'
    static_configs:
      - targets:
          - 'node1:9100'
          - 'node2:9100'
          - 'node3:9100'
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '([^:]+):.*'
        replacement: '${1}'

  # Application metrics
  - job_name: 'api'
    static_configs:
      - targets: ['api-1:8080', 'api-2:8080', 'api-3:8080']
        labels:
          env: 'production'
          tier: 'backend'

Kubernetes Service Discovery

scrape_configs:
  # Kubernetes API server
  - job_name: 'kubernetes-apiservers'
    kubernetes_sd_configs:
      - role: endpoints
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https

  # Kubernetes pods with prometheus.io annotations
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Only scrape pods with prometheus.io/scrape: "true" annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      # Use the port from prometheus.io/port annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: (\d+)
        target_label: __address__
        replacement: ${1}:${2}
      # Use the path from prometheus.io/path annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
      # Add namespace label
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      # Add pod name label
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

  # Kubernetes services
  - job_name: 'kubernetes-services'
    kubernetes_sd_configs:
      - role: service
    metrics_path: /probe
    params:
      module: [http_2xx]
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
        action: keep
        regex: true