Dt Obs Kubernetes

Infrastructure Kubernetes

Monitor and analyze Kubernetes infrastructure using Dynatrace DQL. Query cluster resources, monitor workload health, analyze pod placement, optimize costs, and assess security posture.

When to Use This Skill

• Monitoring Kubernetes cluster health and capacity
• Analyzing pod and container resource utilization
• Investigating pod failures, OOMKills, evictions, or crash loops
• Debugging degraded deployments, stuck rollouts, or node pressure
• Optimizing Kubernetes resource costs
• Assessing security posture and compliance
• Troubleshooting workload scheduling and placement
• Auditing ingress routing and network policies

Knowledge Base Structure

Core Monitoring (Start Here)

1. Cluster Inventory → references/cluster-inventory.md - Clusters, namespaces, resource distribution
2. Node Monitoring - Node capacity, CPU/memory usage, pod density
3. Pod Monitoring - Pod CPU, memory, lifecycle events
4. Workload Monitoring - Deployment, StatefulSet, DaemonSet resources

Advanced Topics

1. Configuration Analysis → references/labels-annotations.md - Parse k8s.object, labels, annotations
2. Scheduling & Placement → references/pod-node-placement.md - Node selectors, affinity, taints, HA
3. Cost Optimization - Right-sizing, waste detection, efficiency scoring
4. Security & Compliance - Privileged containers, security contexts

Key Concepts

Entity Types

Workloads: K8S_DEPLOYMENT, K8S_STATEFULSET, K8S_DAEMONSET, K8S_JOB, K8S_CRONJOB, K8S_HORIZONTALPODAUTOSCALER Infrastructure: K8S_CLUSTER, K8S_NAMESPACE, K8S_NODE, K8S_POD Configuration: K8S_SERVICE, K8S_CONFIGMAP, K8S_SECRET, K8S_PERSISTENTVOLUMECLAIM, K8S_PERSISTENTVOLUME, K8S_INGRESS, K8S_NETWORKPOLICY

Note: HPA, Job, CronJob, and configuration entity types are listed for inventory queries only. For HPA scaling analysis see references/workload-health.md; for PVC/PV see references/pv-pvc.md; for Ingress/NetworkPolicy see the corresponding reference files.

Query Types

smartscapeNodes - Query K8s entities (current state, no time range needed):

smartscapeNodes K8S_POD
| filter k8s.namespace.name == "production"
| fields k8s.cluster.name, k8s.pod.name

timeseries - Monitor metrics over time (always specify a from: range):

timeseries cpu = sum(dt.kubernetes.container.cpu_usage),
  by: {k8s.pod.name, k8s.namespace.name},
  from: now()-1h
| fieldsAdd avg_cpu = arrayAvg(cpu)

timeseries returns each metric as a time-bucket array. To collapse it to a scalar in a downstream fieldsAdd, use arrayAvg(series) or arraySum(series) — calling avg() or sum() on a series field outside the timeseries {} block is not supported.

fetch logs - Analyze log events:

fetch logs
| filter k8s.namespace.name == "production" and loglevel == "ERROR"

Core Fields

• k8s.cluster.name, k8s.namespace.name, k8s.pod.name, k8s.node.name
• k8s.workload.name, k8s.workload.kind, k8s.container.name
• k8s.object - Full JSON configuration for deep inspection
• tags[label] - Access labels and annotations

`k8s.workload.kind` values (always lowercase in DT — not Pascal-case as in the K8s API): "deployment", "statefulset", "daemonset", "replicaset", "job", "cronjob"

`k8s.object` availability:

Available Metrics

CPU: dt.kubernetes.container.cpu_usage, cpu_throttled, limits_cpu, requests_cpu Memory: dt.kubernetes.container.memory_working_set, limits_memory, requests_memory Operations: dt.kubernetes.container.restarts, oom_kills Node: dt.kubernetes.node.pods_allocatable, cpu_allocatable, memory_allocatable, dt.kubernetes.pods

Aggregation rule for requests/limits: Always use sum() — never avg() — when aggregating requests_cpu, limits_cpu, requests_memory, or limits_memory. avg() returns a per-container average and silently undercounts multi-container pods and multi-replica workloads. For DaemonSets (one pod per node), this error is proportional to cluster size.

Entity Disambiguation

K8S_POD vs CONTAINER: these are different entity types in Dynatrace.

• `K8S_POD` — K8s-native entities with k8s.object JSON, scheduling state, conditions, and K8s metrics. Use this skill.
• `CONTAINER` — Host-level container inventory (image, lifetime, host assignment). Use dt-obs-hosts skill instead.

The smartscape edge is CONTAINER --(is_part_of)--> K8S_POD. To reach containers from a pod, traverse backward:

smartscapeNodes K8S_POD
| filter k8s.namespace.name == "<namespace>"
| traverse edgeTypes: {is_part_of}, targetTypes: {CONTAINER}, direction: backward, fieldsKeep: {id}
| fields k8s.cluster.name, k8s.namespace.name, k8s.pod.name, container.id=id

Service → K8S_POD Correlation

No direct smartscape edge exists between SERVICE and K8S_POD. The correlation key is the shared dimension k8s.workload.name. See Service → Pod Drill-Down in references/pod-debugging.md for the full two-step pattern.

Common Workflows

1. Cluster Health Check

List all clusters:

smartscapeNodes K8S_CLUSTER
| fields k8s.cluster.name, k8s.cluster.version, k8s.cluster.distribution

Check node capacity:

timeseries {
  current_pods = avg(dt.kubernetes.pods),
  max_pods = avg(dt.kubernetes.node.pods_allocatable)
}, by: {k8s.node.name, k8s.cluster.name},
from: now()-1h
| fieldsAdd pod_capacity_pct = (arrayAvg(current_pods) / arrayAvg(max_pods)) * 100
| filter pod_capacity_pct > 80

Identify pods in non-Running state:

smartscapeNodes K8S_POD
| parse k8s.object, "JSON:config"
| fieldsAdd phase = config[status][phase]
| filter not(in(phase, {"Running", "Succeeded"}))
| fields k8s.cluster.name, k8s.namespace.name, k8s.pod.name, phase

Succeeded is a healthy terminal phase for completed Job pods — excluding it avoids false positives. Adjust if you specifically want to audit completed Jobs.

2. Resource Optimization

Pod-level — find over-provisioned pods (usage < 30%):

timeseries {
  cpu_usage = sum(dt.kubernetes.container.cpu_usage),
  cpu_requests = sum(dt.kubernetes.container.requests_cpu)
}, by: {k8s.pod.name, k8s.namespace.name, k8s.cluster.name},
from: now()-7d
| fieldsAdd usage_pct = (arrayAvg(cpu_usage) / arrayAvg(cpu_requests)) * 100
| filter usage_pct < 30 and arrayAvg(cpu_requests) > 0

Workload-level — aggregate across all replicas (required for correct DaemonSet accounting):

timeseries {
  cpu_usage = sum(dt.kubernetes.container.cpu_usage),
  cpu_requests = sum(dt.kubernetes.container.requests_cpu)
}, by: {k8s.workload.name, k8s.workload.kind, k8s.namespace.name, k8s.cluster.name},
from: now()-7d
| fieldsAdd
    avg_usage = arrayAvg(cpu_usage),
    avg_requests = arrayAvg(cpu_requests)
| fieldsAdd usage_pct = (avg_usage / avg_requests) * 100
| filter usage_pct < 30 and avg_requests > 0
| sort usage_pct asc

Use sum() for both cpu_usage and cpu_requests at every aggregation level. A DaemonSet running on 50 nodes has 50× the per-pod request total; avg() would report 1/50th of the true reserved capacity.

Identify containers without limits:

smartscapeNodes K8S_POD
| parse k8s.object, "JSON:config"
| expand container = config[spec][containers]
| fieldsAdd
    container_name = container[name],
    cpu_limit = container[resources][limits][cpu],
    memory_limit = container[resources][limits][memory]
| filter isNull(cpu_limit) or isNull(memory_limit)

3. Troubleshooting Pod Issues