Dt Obs Tracing

Application Tracing Skill

Overview

Distributed traces in Dynatrace consist of spans - building blocks representing units of work. With Traces in Grail, every span is accessible via DQL with full-text searchability on all attributes. This skill covers trace fundamentals, common analysis patterns, and span-type specific queries.

Use Cases

1. Investigate Slow Requests

• Goal: Find and diagnose requests exceeding a latency threshold
• Trigger: "slow requests", "high latency", "p99 response time", "find traces over 5 seconds"
• Done: List of slow traces with duration, endpoint, service, and trace IDs for drilldown

2. Analyze Request Failures

• Goal: Identify failed requests, failure reasons, and exception patterns
• Trigger: "failed spans", "HTTP 500 errors", "exception analysis", "failure rate by service"
• Done: Failure breakdown by reason (HTTP code, exception, gRPC status) with exemplar traces

3. Map Service Dependencies

• Goal: Understand service-to-service communication patterns and external API calls
• Trigger: "service dependencies", "what services does X call", "outgoing HTTP calls"
• Done: Dependency map showing call counts, latency, and error rates between services

Core Concepts

Understanding Traces and Spans

Spans represent logical units of work in distributed traces:

• HTTP requests, RPC calls, database operations
• Messaging system interactions
• Internal function invocations
• Custom instrumentation points

Span kinds:

• span.kind: server - Incoming call to a service
• span.kind: client - Outgoing call from a service
• span.kind: consumer - Incoming message consumption call to a service
• span.kind: producer - Outgoing message production call from a service
• span.kind: internal - Internal operation within a service

Root spans: A request root span (request.is_root_span == true) represents an incoming call to a service. Use this to analyze end-to-end request performance.

Key Trace Attributes

Essential attributes for trace analysis:

Service Context

Spans reference services via Smartscape node IDs and the detected service name dt.service.name which is also present on every span.

fetch spans
| summarize spans=count(), by: { dt.smartscape.service, dt.service.name }

Node functions:

• getNodeName(dt.smartscape.service) - Adds dt.smartscape.service.name field with the human-readable service name
• getNodeField(dt.smartscape.service, "attribute_name") - Access specific node attributes

📖 Learn more: See Entity Lookups for advanced entity selectors, infrastructure correlation, and hardware analysis.

Sampling and Extrapolation

One span can represent multiple real operations due to:

• Aggregation: Multiple operations in one span (aggregation.count)
• ATM (Adaptive Traffic Management): Head-based sampling by agent
• ALR (Adaptive Load Reduction): Server-side sampling
• Read Sampling: Query-time sampling via samplingRatio parameter

When to extrapolate: Always extrapolate when counting actual operations (not just spans). Use the multiplicity factor:

fetch spans
| fieldsAdd sampling.probability = (power(2, 56) - coalesce(sampling.threshold, 0)) * power(2, -56)
| fieldsAdd sampling.multiplicity = 1 / sampling.probability
| fieldsAdd multiplicity = coalesce(sampling.multiplicity, 1)
                         * coalesce(aggregation.count, 1)
                         * dt.system.sampling_ratio
| summarize operation_count = sum(multiplicity)

📖 Learn more: See Sampling and Extrapolation for detailed formulas and examples.

Common Query Patterns

Basic Span Access

Fetch spans and explore by type:

fetch spans | limit 1

Explore spans by function and type:

fetch spans
| summarize count(), by: { span.kind, code.namespace, code.function }

Request Root Filtering

List request root spans (incoming service calls):

fetch spans
| filter request.is_root_span == true
| fields trace.id, span.id, start_time, response_time = duration, endpoint.name
| limit 100

Service Performance Summary

Analyze service performance with error rates:

fetch spans
| filter request.is_root_span == true
| summarize
    total_requests = count(),
    failed_requests = countIf(request.is_failed == true),
    avg_duration = avg(duration),
    p95_duration = percentile(duration, 95),
    by: {dt.service.name}
| fieldsAdd error_rate = (failed_requests * 100.0) / total_requests
| sort error_rate desc

Trace ID Lookup

Find all spans in a specific trace:

fetch spans
| filter trace.id == toUid("abc123def456")
| fields span.name, duration, dt.service.name

Performance Analysis

Response Time Percentiles

Calculate percentiles by endpoint:

fetch spans
| filter request.is_root_span == true
| summarize {
    requests=count(),
    avg_duration=avg(duration),
    p95=percentile(duration, 95),
    p99=percentile(duration, 99)
  }, by: { endpoint.name }
| sort p99 desc

💡 Best practice: Use percentiles (p95, p99) over averages for performance insights.

Slow Trace Detection

Find requests exceeding a threshold:

fetch spans, from:now() - 2h
| filter request.is_root_span == true
| filter duration > 5s
| fields trace.id, span.name, dt.service.name, duration
| sort duration desc
| limit 50

Duration Buckets with Exemplars

fetch spans, from:now() - 24h
| filter http.route == "/api/v1/storage/findByISBN"
| summarize {
    spans=count(),
    trace=takeAny(record(start_time, trace.id))
  }, by: { bin(duration, 10ms) }
| fields `bin(duration, 10ms)`, spans, trace.id=trace[trace.id], start_time=trace[start_time]

Performance Timeseries

Extract response time as timeseries:

fetch spans, from:now() - 24h
| filter request.is_root_span == true
| makeTimeseries {
    requests=count(),
    avg_duration=avg(duration),
    p95=percentile(duration, 95),
    p99=percentile(duration, 99)
  }, by: { endpoint.name }

📖 Learn more: See Performance Analysis for advanced patterns and timeseries techniques.

Failure Investigation

Failed Request Summary

Summarize failures by service:

fetch spans
| filter request.is_root_span == true
| summarize
    total = count(),
    failed = countIf(request.is_failed == true),
  by: { dt.service.name }
| fieldsAdd failure_rate = (failed * 100.0) / total
| sort failure_rate desc

Failure Reason Analysis

Breakdown by failure detection reason:

fetch spans
| filter request.is_failed == true and isNotNull(dt.failure_detection.results)
| expand dt.failure_detection.results
| summarize count(), by: { dt.failure_detection.results[reason] }

Failure reasons:

• http_code - HTTP response code triggered failure
• grpc_code - gRPC status code triggered failure
• exception - Exception caused failure
• span_status - Span status indicated failure
• custom_rule - Custom failure detection rule matched

HTTP Code Failures

Find failures by HTTP status code:

fetch spans