Prometheus Cardinality Troubleshooter

Prometheus Cardinality Troubleshooter

You are an expert in diagnosing live Prometheus cardinality problems. When a user reports a Prometheus performance, memory, or cost issue that smells like cardinality, use this guide to triage systematically.

This skill is diagnostic and operational. For schema design and prevention, route to prometheus-label-strategy.

Before You Remediate: The One Rule

Under pressure, the tempting move is to labeldrop the high-cardinality label at scrape time. Do not. You cannot remove, at scrape time, any label that makes a series unique — not pod, not instance, not anything that distinguishes one real series from another. It looks like it stops the bleeding; it actually breaks the data:

• Counter resets from different series get merged → rate() and increase() return garbage, often absurdly high values.
• Multiple samples land on the same series per scrape → duplicate-sample / out-of-order errors and inflated DPM, not reduced.
• The breakage is silent (no config error) and leaves no evidence in the data of where it went wrong. Weeks later someone asks "why is my DPM so high / why is rate() absurd?" and there's nothing to point to.

The only safe remediations are:

1. *Drop an entire unwanted metric* (action: drop on __name__) — you're discarding the whole metric, not merging distinct series.
2. Fix the source — stop the application emitting the bad label (the real fix for unbounded path, user_id, etc.).
3. Adaptive Metrics — for structural cardinality on series you can't fix at the source. It aggregates correctly (counter-reset-aware, audited, reversible). This is the right way to reduce the cost of a label like pod. Route to adaptive-metrics.

Everywhere below that says "drop a label," read it through this rule: drop whole metrics, fix the source, or use Adaptive Metrics — never labeldrop a distinguishing label.

Symptom → Likely Cause

Step 1: Active Series Triage

Get the headline number

# Total active series in the local Prometheus
prometheus_tsdb_head_series

# Or for Mimir / Grafana Cloud Metrics (per tenant)
cortex_ingester_memory_series{user="<tenant>"}

Compare to recent history:

# Growth over the last 7 days
deriv(prometheus_tsdb_head_series[7d]) * 86400

A growth rate > a few % per day on a stable application set is a red flag.

Use the TSDB status endpoint

Prometheus exposes a built-in cardinality breakdown:

curl -s http://prometheus:9090/api/v1/status/tsdb | jq

Returns:

• seriesCountByMetricName — top metrics by series count
• labelValueCountByLabelName — top labels by unique value count
• memoryInBytesByLabelName — top labels by memory footprint
• seriesCountByLabelValuePair — top label-value pairs by series count

This is usually the fastest path to "which metric / which label is the problem."

For Grafana Cloud:

# Same endpoint, authenticated against the per-tenant Mimir
curl -s -u "<user>:<token>" \
  "https://prometheus-prod-XX.grafana.net/api/prom/api/v1/status/tsdb" | jq

Step 2: Read the Output

Top metrics by series count

"seriesCountByMetricName": [
  { "name": "http_request_duration_seconds_bucket", "value": 184320 },
  { "name": "go_gc_duration_seconds",               "value": 80 },
  ...
]

Heuristics:

• A histogram (_bucket) at the top is almost always the answer — those have a 14× multiplier (bucket count + 3). The fix is usually reducing the labels on the underlying histogram at the source (in instrumentation code), not stripping them at scrape and not touching the buckets themselves.
• A metric in the top 5 you don't recognize → grep the codebase for it; it's likely a new feature flag or a debug metric that shipped to prod
• The same metric showing up under multiple variants (_total, _count, _sum) — that's a histogram or summary, count all variants together for the true impact

Top labels by unique value count

"labelValueCountByLabelName": [
  { "name": "url",       "value": 84210 },
  { "name": "trace_id",  "value": 41000 },
  { "name": "pod",       "value": 1820 }
]

Red flags:

• Any label with >10K unique values is almost certainly a bug. The only exceptions are intentional per-target labels in massive fleets.
• trace_id, request_id, session_id, query, email, path, url — these should never be labels. They belong in exemplars, logs, or traces.
• pod with thousands of values — see Churn diagnosis; recent churn often inflates this number

Step 3: Per-Metric Drill-Down

Once you've identified a suspect metric, find which label is responsible.

Count distinct label values per label, for one metric

# How many unique values does each label have on this metric?
count by (__name__) (
  count by (__name__, label_name_here) (
    http_request_duration_seconds_bucket
  )
)

Repeat per label, or use the helper:

# Via the Prometheus HTTP API
curl -s "http://prometheus:9090/api/v1/labels?match[]=http_request_duration_seconds_bucket" | jq -r '.data[]' | \
  while read label; do
    count=$(curl -s "http://prometheus:9090/api/v1/label/${label}/values?match[]=http_request_duration_seconds_bucket" | jq '.data | length')
    echo "${count}  ${label}"
  done | sort -rn | head -20

Find the top label values for one label

# Top 20 path values for http_requests_total
topk(20,
  count by (path) (http_requests_total)
)

If you see UUIDs, hashes, timestamps, or numeric IDs in the top values → that label has unbounded values from the source.

Per-metric series count, grouped

# Series-per-instance breakdown — if uneven, one instance is misbehaving
sum by (job, instance) ({__name__=~"my_metric.*"})

Step 4: Recent Change Diff

If the cardinality fire started recently, the cause is almost always a recent change. Diff what's there now against what was there before.

List of metrics, current vs. yesterday

Via Grafana Cloud cardinality dashboard, or:

# Current metrics
group by (__name__) ({__name__!=""})

# Compare to last week (offset)
group by (__name__) ({__name__!=""} offset 7d)

Diff externally. A new metric near the top of seriesCountByMetricName that wasn't there a week ago → that's your offender.

Correlate with deploys

# Active series correlated with build_info
prometheus_tsdb_head_series
# Overlay with:
changes(app_build_info[1d])