Otel Instrumentation

OpenTelemetry Instrumentation for Honeycomb

SDK setup, custom spans, attributes, span events, sampling, and layered telemetry. For conceptual foundations (why wide events matter, how attributes connect to investigation), see the observability-fundamentals skill.

OTLP Configuration and SDK Setup

Every OTel SDK needs these environment variables to send data to Honeycomb:

Required Environment Variables

Base configuration:

OTEL_SERVICE_NAME=your-service-name
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io
OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=YOUR_API_KEY"

Optional but recommended:

# Protocol selection (default: http/protobuf)
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf  # or grpc

# Signal-specific endpoints (override base endpoint for specific signals)
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.honeycomb.io/v1/traces
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.honeycomb.io/v1/metrics

For metrics (preferred): Use modern OTLP metrics and native datapoints. Use dataset hints to confirm the destination type (metrics or events). Authenticate with:

OTEL_EXPORTER_OTLP_METRICS_HEADERS="x-honeycomb-team=YOUR_API_KEY"

Protocol Selection

OTEL_EXPORTER_OTLP_PROTOCOL determines the wire format and transport:

• http/protobuf (default, recommended) — HTTP with protobuf encoding
• grpc — gRPC with protobuf encoding
• http/json — HTTP with JSON encoding (larger payload, slower)

Use http/protobuf unless you have specific infrastructure requirements for gRPC.

Signal-Specific Endpoints

By default, OTel SDKs append /v1/traces and /v1/metrics to OTEL_EXPORTER_OTLP_ENDPOINT. Use signal-specific endpoint vars to override:

• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT — full URL for traces (including /v1/traces)
• OTEL_EXPORTER_OTLP_METRICS_ENDPOINT — full URL for metrics (including /v1/metrics)

Useful when routing signals to different backends or using non-standard endpoints.

Common Pitfalls

Silent auth failure: The OTLP exporters need the x-honeycomb-team header to authenticate. Without it, Honeycomb silently rejects requests — no error, no data. Set OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=YOUR_API_KEY" or pass headers programmatically. If loading the key from .env, ensure dotenv runs before SDK init.

Metrics: Prefer modern OTLP metrics and native datapoints. Dataset hints identify the destination type (metrics or events), so do not add x-honeycomb-dataset by default. Use that header only when hints or configuration require legacy routing to a named event dataset. Traces do not need it; they route by service.name.

For the env var values, language-specific dependencies, and setup code (Go, Python, Node.js, Java, Ruby, .NET, Rust), see ${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/sdk-setup-by-language.md.

Custom Instrumentation

Adding Attributes to Existing Spans (Highest Impact)

Add business context to auto-instrumented spans — no new spans needed. Get the current span from context and call SetAttributes (Go), set_attribute (Python), or setAttribute (Node.js) with user, tenant, business, and deployment context.

Creating Custom Spans

Wrap important business operations for visibility in the trace waterfall. Use tracer.Start(ctx, "operation-name") (Go), tracer.start_as_current_span("operation-name") (Python), or tracer.startActiveSpan("operation-name", callback) (Node.js).

For full code examples in all languages, consult ${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/custom-instrumentation.md.

When to Create a Span

Not every function needs a span. Two questions determine whether a span is worth creating:

1. Is it interesting? — Does the work meaningfully impact performance (latency or failures) for the overall request?
2. Is it aggregable? — If you group this span by name and attributes, will it produce useful trends and comparisons?

Common mistakes:

• Too many spans: A trace with millions of 2ms spans is far too detailed and rarely actionable. Roll them up — combine into a single span, or capture the detail as an attribute on the parent span instead.
• Too few spans: Collapsing hours of work into a single opaque handler leaves you guessing about where time is spent.
• Test spans left in: Spans named test-span, debug-span, or similar are artefacts that pollute the dataset. Remove any span created solely to verify tracing is working before finishing.

When in doubt, prefer attributes on existing spans over creating new child spans.

Timing Attributes (measure sub-operations without child spans)

Record important sub-operation durations as attributes on the parent span. These are easier to query than child spans and work directly with BubbleUp.

// Go: time auth and record on the existing span
span := trace.SpanFromContext(r.Context())
authStart := time.Now()
user, err := authenticate(r)
span.SetAttributes(attribute.Float64("auth.duration_ms", float64(time.Since(authStart).Milliseconds())))

# Python: time auth and record on the existing span
span = trace.get_current_span()
auth_start = time.monotonic()
user = authenticate(request)
span.set_attribute("auth.duration_ms", (time.monotonic() - auth_start) * 1000)

Exception telemetry: event details plus span-level dimensions

Use the Logs API for new exception events. Emit the record while the relevant span is active and include the standard exception fields (exception.type, exception.message, exception.stacktrace, and exception.escaped when applicable), an ERROR severity, and event.name="exception". Set the span status to ERROR separately when the operation failed.

In Honeycomb, a trace-correlated exception log is rendered in the trace as a span_event annotation and carries trace.trace_id and trace.parent_id. Its full exception.* payload remains on the log-derived event; it is not hoisted onto the containing span. Search the exception event row, then follow its trace ID to inspect the surrounding trace.

Use low-cardinality span attributes for aggregation and alerting:

• error=true and the span status indicate operation failure.
• exception.slug is a static, greppable identifier for the error site.
• An optional error category is safer for GROUP BY than full exception messages.

Logs-API exception event: event.name=exception, body=exception, meta.signal_type=log
Legacy span-event exception: name=exception, meta.signal_type=trace
Both may have: meta.annotation_type=span_event

record_exception / RecordError remain compatibility APIs for existing SDKs and code, but do not use them as the only new guidance when Logs API support is available. They can