Agent Observability Eval Bootstrap

Backend

Detection — At the start of every invocation, before taking any action, determine which backend to use:

1. If the user passed --backend pup anywhere in their invocation → use pup mode immediately, regardless of whether MCP tools are present. Skip steps 2–4.
2. Check whether MCP tools are present in your active tool list. The canonical signal is whether mcp__datadog-llmo-mcp__list_llmobs_evals appears in your available tools.
3. If MCP tools are present → use MCP mode throughout. Call MCP tools exactly as named in this skill's workflow sections.
4. If MCP tools are absent → check whether pup is executable: run pup --version via Bash. A JSON response containing "version" confirms pup is available.
5. If pup responds → use pup mode throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.
6. If neither is available → stop and tell the user: > "Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs') or install pup."

--backend pup is accepted anywhere in the invocation arguments and is stripped before passing remaining args to the skill logic.

pup invocation rules:

• Invoke via Bash: pup llm-obs <subcommand> [flags]
• pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in [{"type": "text", "text": "<json>"}]).
• If pup returns an auth error, tell the user to run pup auth login and stop.
• Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).
• Time flags: pup accepts bare duration strings (1h, 7d, 30m) and RFC3339 timestamps. Do not use now--prefixed strings — strip the prefix when converting from a skill --timeframe argument: now-7d → 7d, now-24h → 24h, now-30d → 30d.
• --summary on pup llm-obs spans search strips payload fields to essential metadata only. Use it in bulk/search phases where content is not needed.

Invocation ID: At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., 3a9f1c2b). Keep it constant for the entire invocation.

Intent tagging: On every MCP tool call, prefix telemetry.intent with skill:agent-observability-eval-bootstrap[<inv_id>] — followed by a description of why the tool is being called. On the first MCP tool call only, use skill:agent-observability-eval-bootstrap:start[<inv_id>] — instead (note the :start suffix). Example first call: skill:agent-observability-eval-bootstrap:start[3a9f1c2b] — Phase 0: map existing eval coverage for task-cruncher

Eval Bootstrap — Generate Evaluators from Production Traces

Given a sample of production LLM traces, analyze input/output patterns and quality dimensions, then propose a ready-to-use evaluator suite. Four output modes — online evaluators are the default; SDK code, the JSON spec, and the dataset-emit mode are produced on request:

• `publish` (default) — propose online LLM-judge evaluators, then — only after you confirm the suite — write them to Datadog via create_or_update_llmobs_evaluator as disabled drafts (enabled: false). Nothing is created until you confirm at the Phase 2 checkpoint, and nothing scores any spans until you enable it in the UI — the skill never auto-publishes a live evaluator. Once you enable a draft, it runs automatically on matching production spans, traces, or sessions (no dataset, no task function). The skill auto-classifies each proposed evaluator as span-scoped, trace-scoped, or session-scoped based on what the judgment requires (a per-LLM-call tone check vs. an agent goal completion that needs the whole trace vs. user satisfaction across a whole multi-trace conversation) — you accept or override the classification at that checkpoint. Session-scoped evaluators are only proposed when the app's spans carry a session_id (verified by a probe in Phase 1).
• `sdk_code` (on request — `--sdk-code`, or ask after a publish run) — Python .py file using the Datadog Evals SDK (BaseEvaluator / LLMJudge) for offline experiments.
• `data_only` (on request — `--data-only`) — self-contained JSON spec, framework-agnostic.
• `emit_dataset` (on request — `--emit-dataset <path>`) — sample production traces and write a DatasetRecordRaw[] JSON file shaped for LLMObs.create_dataset(records=...). Skips evaluator proposal and generation entirely — this mode produces a dataset, not evaluators. Used by agent-observability-eval-pipeline (Phase 4) to seed an experiment dataset from production behavior.

After a publish run, if the user wants the same suite as offline code or a portable spec, they just ask — the skill regenerates the already-confirmed suite in sdk_code / data_only mode without re-exploring (see "On-request code generation" in Phase 3). The emit_dataset mode is independent of the evaluator workflow and never re-uses a prior proposal — it always re-samples traces.

Usage

/eval-bootstrap <ml_app> [--timeframe <window>] [--sdk-code | --data-only | --emit-dataset <path>] [--trace-limit <N>]

Arguments: $ARGUMENTS

Inputs

If ml_app is missing, ask the user before proceeding. With no mode flag, the skill defaults to `publish` — it proposes online evaluators and, only after you confirm, creates them as disabled drafts (it never auto-enables them). If more than one of --sdk-code, --data-only, --emit-dataset is supplied, error out and ask which mode the user wants.

Available Tools