Agent Observability Session Classify

Backend

Detection — At the start of every invocation, before taking any action, determine which backend to use:

1. If the user passed --backend pup anywhere in their invocation → use pup mode immediately, regardless of whether MCP tools are present. Skip steps 2–4.
2. Check whether MCP tools are present in your active tool list. The canonical signal is whether mcp__datadog-llmo-mcp__search_llmobs_spans appears in your available tools.
3. If MCP tools are present → use MCP mode throughout. Call MCP tools exactly as named in this skill's workflow sections.
4. If MCP tools are absent → check whether pup is executable: run pup --version via Bash. A JSON response containing "version" confirms pup is available.
5. If pup responds → use pup mode throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.
6. If neither is available → stop and tell the user: > "Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs,rum') or install pup."

--backend pup is accepted anywhere in the invocation arguments and is stripped before passing remaining args to the skill logic.

pup invocation rules:

• Invoke via Bash: pup llm-obs <subcommand> [flags]
• pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in [{"type": "text", "text": "<json>"}]).
• If pup returns an auth error, tell the user to run pup auth login and stop.
• Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).
• Time flags: pup accepts bare duration strings (1h, 7d, 30m) and RFC3339 timestamps. Do not use now--prefixed strings — strip the prefix when converting from a skill --timeframe argument: now-7d → 7d, now-24h → 24h, now-30d → 30d.
• --summary on pup llm-obs spans search strips payload fields to essential metadata only. Use it in bulk/search phases where content is not needed.

pup mode notes by entry mode:

• session_id mode: Steps 1–3 and Step 5 work fully. Step 4 (RUM) uses pup rum aggregate --user-email EMAIL instead of analyze_rum_events — see Tool Reference. Step 4b (audit trail) is pup-native and queries the active user's own org via OAuth.
• trace_id mode: Full parity with MCP mode.
• ml_app mode: Option A (aggregate_spans) is unavailable in pup — skip it and proceed directly to Option B.

Invocation ID: At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., 3a9f1c2b). Keep it constant for the entire invocation.

Intent tagging: On every MCP tool call, prefix telemetry.intent with skill:agent-observability-session-classify[<inv_id>] — followed by a description of why the tool is being called. On the first MCP tool call only, use skill:agent-observability-session-classify:start[<inv_id>] — instead (note the :start suffix). Example first call: skill:agent-observability-session-classify:start[3a9f1c2b] — Step 1: enumerate turn root spans for session abc-123

Skill: eval-session-classify

Classification skill for Datadog Agent Observability sessions and traces. Produces a satisfaction verdict (yes / partial / no) with a brief reasoning string. Designed to be called standalone or by eval-pipeline.

Inputs

If none of `session_id`, `trace_id`, or `ml_app` is provided → stop immediately and return:

{
  "error": "missing_input",
  "detail": "Provide one of: session_id, trace_id, or ml_app."
}

Phase 0 — Mode Detection

• session_id provided → session_id mode → go to Session Mode
• trace_id provided (no session_id) → trace_id mode → go to Trace Mode
• ml_app provided (no session_id, no trace_id) → ml_app mode → go to ML App Mode
• Nothing provided → emit error (see above)

Output Format

Default (compact) — used in all modes unless the caller requests verbose:

verdict: yes | partial | no
reason: <one sentence>

Verbose — full markdown report (see end of each mode section). Request verbose by including verbose: true in input or asking for a detailed report.

In ml_app mode, a summary table is always appended after the per-trace compact blocks.

Content Retrieval Cascade

Reading conversation content follows this cascade across all modes. Run the cascade for every turn root span identified in Step 1 (session mode) or Step T1 (trace mode) before forming a verdict.

Completeness gate: do not proceed to RUM (session mode) or classification (any mode) until the cascade has run for every turn. Previews from search_llmobs_spans (input.preview / output.preview, truncated to ~200 chars) do NOT satisfy this gate — they identify turn boundaries, nothing more. Only actual content from get_llmobs_agent_loop, expand_llmobs_spans + get_llmobs_span_content, or get_llmobs_span_content(field="input"/"output") counts. Short sessions (≤ 20 turns): run C1/C2 for every turn — no selection, no "key turns" heuristic. Long sessions (> 20 turns): run C1/C2 for the first 5 and last 5 turns, plus any turn whose input.preview or output.preview flags an anomaly (error, correction loop, repeated intent, unexpected tool). Scan all remaining turns' previews for anomalies before skipping them.

Parallelism: issue all N get_llmobs_agent_loop calls in a single message — one call per turn, all in the same batch. For an N-turn session this means N simultaneous C1 calls. Do not process turns sequentially and do not form partial verdicts before all results are in hand. Move to C2 for a turn only when C1 returns iterations: [] for that turn.

C1 — get_llmobs_agent_loop(trace_id, agent_span_id)

The richest source: full system prompt, user message, tool call arguments + results, assistant response, and token economics per iteration. Attempt this first for every agent span.

get_llmobs_agent_loop(
  trace_id           = "<TRACE_ID>",
  span_id            = "<AGENT_SPAN_ID>",
  from               = "now-90d",
  to                 = "now",
  max_content_length = 2000
)

• `iterations: []` and `timeline: null` → the app's LLM spans go through an intermediate workflow layer (e.g. get_answer_from_model_step) rather than as direct children of the agent span. Fall through to C2.
• 404 → span ID does not resolve in the trace store. Fall through to C2.
• Content fields contain `<REDACTED_INPUT>` or `<MASKED_STREAMING_RESPONSE>` → IO tracing is disabled by a feature flag. Structure (iteration count, tool names, token usage) is still useful — note it. Fall through to C2 for actual content.

When a C1 result is too large for context and gets persisted to disk (common on long sessions): do NOT try to Read the full file — it will exceed the token limit. Run a targeted Python extraction to build a structured per-turn summary:

import json
with open('<persisted_path>') as f:
    data = json.loads(json.load(f)[0]['text'])
iters = data['iterations']
for it in iters:
    tcs = [tc['name'] for tc in it.get('tool_calls', [])]
    content = it.get('content', '') or ''