Agent Observability Eval Pipeline

Backend

Detection — At the start of every invocation, before taking any action, determine which backend to use:

1. If the user passed --backend pup anywhere in their invocation → use pup mode immediately, regardless of whether MCP tools are present. Skip steps 2–4.
2. Check whether MCP tools are present in your active tool list. The canonical signal is whether mcp__datadog-llmo-mcp__search_llmobs_spans appears in your available tools.
3. If MCP tools are present → use MCP mode throughout. Call MCP tools exactly as named in the sub-skill workflow sections.
4. If MCP tools are absent → check whether pup is executable: run pup --version via Bash. A JSON response containing "version" confirms pup is available.
5. If pup responds → use pup mode throughout. Each sub-skill carries its own Tool Reference appendix with the full MCP→pup mapping.
6. If neither is available → stop and tell the user: > "Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs') or install pup."

--backend pup is accepted anywhere in the invocation arguments. Strip it from args before passing to sub-skills, but carry the pup-mode decision forward — every sub-skill must also operate in pup mode for the entire pipeline run.

Sub-skill backend propagation: The backend detected at startup applies to all sub-skills invoked across the six phases. Do not re-detect per phase. Announce once at startup:

• MCP mode: "(Running in MCP mode — all features available.)"
• pup mode: "(Running in pup mode — pup commands used throughout. All features available.)"

Invocation ID: At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., 3a9f1c2b). Keep it constant for the entire invocation.

Intent tagging: On every MCP tool call, prefix telemetry.intent with skill:agent-observability-eval-pipeline[<inv_id>] — followed by a description of why the tool is being called. On the first MCP tool call only, use skill:agent-observability-eval-pipeline:start[<inv_id>] — instead (note the :start suffix). Example first call: skill:agent-observability-eval-pipeline:start[3a9f1c2b] — Precheck: verify ml_app has traces in the last 7 days

Agent Observability Eval Pipeline — Classify → RCA → Eval Bootstrap → Dataset → Experiment → Analyze

A deterministic, six-phase guided pipeline for an already-instrumented ml_app owner. Each phase has the same envelope — a banner that names the entity being produced, an explanation of its purpose, the action (a sub-skill call or a small executable step), and a checkpoint. You always know where you are.

[Precheck] verify ml_app, project, backend, credentials, output dir
   ↓
[Phase 1: Classify ml_app traces]        entity: ml_app, trace, span
   ↓
[Phase 2: Root cause analysis]           entity: failure mode, root cause
   ↓
[Phase 3: Bootstrap evaluators]          entity: evaluator, LLM judge
   ↓                                     (stop here with --stop-after eval-bootstrap
   ↓                                      for the classic eval-pipeline behavior)
[Phase 4: Create + publish dataset]      entity: dataset record, published dataset
   ↓                                     (executes: LLMObs.create_dataset)
[Phase 5: Generate + run experiment]     entity: experiment, task, evaluator, run
   ↓                                     (executes: python <generated_file>)
   ↓                                     (in-phase review beat before run)
[Phase 6: Analyze experiment]            entity: metric, comparison, recommendation

This skill is pure orchestration plus pedagogy — no new analytical logic. The work happens inside the sub-skills (agent-observability-session-classify, agent-observability-trace-rca, agent-observability-eval-bootstrap, agent-observability-experiment-bootstrap, agent-observability-experiment-analyzer). What this skill adds is the deterministic envelope: every phase has the same shape, the same checkpoint contract, and the same entity-explanation banner — so the user gets a consistent, narrated experience regardless of how they phrased the original request.

Usage

/agent-observability-eval-pipeline <ml_app> [--project-name <name>] [--timeframe <window>] [--trace-limit <N>]
                                [--format py|ipynb] [--evaluator-style function|class|remote]
                                [--offline-evaluators | --online-evaluators | --data-only]
                                [--start-at classify|rca|eval-bootstrap|dataset|experiment|analyze]
                                [--stop-after classify|rca|eval-bootstrap|dataset|experiment|analyze]
                                [--classification-summary <path>] [--rca-report <path>]
                                [--dataset-file <path>] [--dataset-name <name>]
                                [--experiment-file <path>] [--experiment-id <uuid> | --experiment-url <url>]
                                [--app-root <path>] [--env-file <path>] [--output-dir <dir>]
                                [--backend pup]

Arguments: $ARGUMENTS

Inputs