Agent Observability Trace Rca

Backend

Detection — At the start of every invocation, before taking any action, determine which backend to use:

1. If the user passed --backend pup anywhere in their invocation → use pup mode immediately, regardless of whether MCP tools are present. Skip steps 2–4.
2. Check whether MCP tools are present in your active tool list. The canonical signal is whether mcp__datadog-llmo-mcp__list_llmobs_evals appears in your available tools.
3. If MCP tools are present → use MCP mode throughout. Call MCP tools exactly as named in this skill's workflow sections.
4. If MCP tools are absent → check whether pup is executable: run pup --version via Bash. A JSON response containing "version" confirms pup is available.
5. If pup responds → use pup mode throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.
6. If neither is available → stop and tell the user: > "Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs') or install pup."

--backend pup is accepted anywhere in the invocation arguments and is stripped before passing remaining args to the skill logic.

pup invocation rules:

• Invoke via Bash: pup llm-obs <subcommand> [flags]
• pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in [{"type": "text", "text": "<json>"}]).
• If pup returns an auth error, tell the user to run pup auth login and stop.
• Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).
• Time flags: pup accepts bare duration strings (1h, 7d, 30m) and RFC3339 timestamps. Do not use now--prefixed strings — strip the prefix when converting from a skill --timeframe argument: now-7d → 7d, now-24h → 24h, now-30d → 30d.
• --summary on pup llm-obs spans search strips payload fields to essential metadata only. Use it in bulk/search phases where content is not needed.

Invocation ID: At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., 3a9f1c2b). Keep it constant for the entire invocation.

Intent tagging: On every MCP tool call, prefix telemetry.intent with skill:agent-observability-trace-rca[<inv_id>] — followed by a description of why the tool is being called. On the first MCP tool call only, use skill:agent-observability-trace-rca:start[<inv_id>] — instead (note the :start suffix). Example first call: skill:agent-observability-trace-rca:start[3a9f1c2b] — Phase 0: discover configured evals for task-cruncher to infer analysis mode

Agent Observability Trace RCA — Root Cause Analysis from Production LLM Traces

Diagnose why an LLM application is failing by searching production traces and walking the span tree from symptom to root cause. The skill automatically selects the best analysis mode based on available signals:

The mode is announced (never asked) in the first user-facing output with a one-line override hint.

Methodology

Resolve → Search → Observe → Open Coding → Axial Coding → Root Cause Navigation → Recommendations

Usage

What's wrong with <ml_app> over the last <timeframe>
Why is <ml_app> failing today
Analyze eval failures for <eval_name> on <ml_app>
Look at the errors on <ml_app> over the last <timeframe>
Root-cause low scores on <eval_name>

Inputs

If neither ml_app nor eval_name is provided, ask the user.

Available Tools

Eval discovery & overview

Trace & span exploration

Key get_llmobs_span_content patterns

How to use search_llmobs_spans

Always include `@ml_app:"<ml_app>"` in the `query` string — the structured `ml_app` parameter is unreliable and can return spans from other apps. Do not rely on the structured parameter alone.

Useful query fragments — combine with space (AND):

Dedicated params (span_kind, root_spans_only, ml_app) work alongside query, but query takes precedence over tags.

Parallelization rules

1. `get_llmobs_span_details`: Group span_ids by trace_id, chunk each trace's span_ids into batches of at most 20. Issue ALL chunks for a page in a single message.
2. `get_llmobs_span_content`: Each call is independent — always issue ALL in a single message.
3. `get_llmobs_trace` / `find_llmobs_error_spans` / `get_llmobs_agent_loop`: Parallelize across different traces in a single message.