Error Discovery

Error Discovery Skill

You are running an interactive error analysis session. The user has a dataset (JSONL, CSV, JSON, etc.) of LLM outputs or traces and wants to discover failure modes by reviewing samples.

This skill is meant for interactive sessions only.

This skill has two parts:

• This file covers phases 1 through 4. You read the data, design the UI, build it, and select samples.
• review-loop.md covers phase 5. You run the interactive review session.

Read this file first to build everything. Once the app is running and the human starts reviewing, follow review-loop.md.

Progress updates

Each phase takes time, especially building the interface. Tell the user what you are doing at each step. Before starting a phase, say what you are about to do and why. When a step finishes, say what you did and what comes next. For example: "Reading 10 sample records to understand the data shape", "Building the HTML app with three views: article, map, and progress", "Clustering on 5 features to select diverse samples." Do not go silent for long stretches.

Phase 1: Understand the domain and data

Before building anything, study the dataset thoroughly.

1a: Read and inventory the data

Load the file. Examine 5 to 10 records across the distribution. For each record, identify:

• All fields, their types, and what they represent
• Which fields are the primary content the human needs to judge
• Which fields are metadata (context for understanding, but not the thing being judged)
• Which fields vary across records and which are constant

1b: Identify the content structure

The data could take many forms. Determine which pattern fits:

• Single text field, e.g., article, summary, email, essay, translation
• Input/output pair, e.g., prompt + completion, question + answer, instruction + result
• Multi-turn trace, e.g., a sequence of messages with different roles (system, human, assistant, tool calls, tool results, thinking/reasoning blocks). This is common for agent traces, chatbot logs, and agentic workflows.
• Code, e.g., source files, patches, diffs, with or without surrounding context
• Structured output, e.g., JSON, function calls, extracted entities, classifications
• Composite, e.g., a task description + an agent trace + a final output

For multi-turn traces specifically, identify:

• What roles or authors exist (system, user, assistant, tool, thinking, etc.)
• Whether there are tool call / tool result pairs
• Whether there are thinking/reasoning blocks
• What the logical grouping of turns is, e.g., one "step" = thinking + tool call + tool result

1c: Identify dimensions of variation

Think about what differs between data points and within each data point. You will use these dimensions to decide the visual design.

Between data points (vary across records):

• Metadata, e.g., topic, model, difficulty, source, label, task type
• Structural, e.g., length, number of turns, number of tool calls
• Outcome, e.g., success/failure, score

Within each data point (vary across parts of one record):

• Author/role of each segment (human vs agent vs system vs tool)
• Content type of each segment (natural language vs code vs JSON vs thinking)
• Importance/relevance of each segment (boilerplate system prompt vs the actual response)

1d: Think about what "bad" means

Think of a few plausible failure categories in this domain. Expect the human to discover most of them during review.

Phase 2: Design the visual encoding

Before writing any code, design how every dimension of variation maps to a visual property. Use Gestalt principles and information visualization fundamentals.

Core Gestalt principles to apply

• Similarity (color, shape). Viewers perceive things that share a visual property as related. Use this for categorical dimensions. Give different message roles different colors, and different labels different badge colors.
• Proximity (spacing). Viewers perceive things that are close together as grouped. Use this for logical units. Place turns in a conversation step close together, and separate steps by more space.
• Common region (containers, backgrounds). Viewers perceive things inside a shared boundary as grouped. Use this for multi-part content, e.g., a tool call and its result share a container.
• Figure/ground (opacity, contrast). Important content should be high-contrast (figure). Less important content should recede (ground). Use this for emphasis. Show boilerplate at low opacity.

Visual encoding rules

Map each dimension of variation to exactly one visual channel. Do not use the same channel for two different things.

Color hue. Use for the categorical dimension with the most important distinction.

• For multi-turn traces, use color for message role. Each role (human, assistant, system, tool, thinking) gets its own distinct hue at full saturation. Pick 3 to 5 easily distinguishable hues.
• For single-content, use color for label or source category.
• Do not use color hue for quantitative data.

Opacity / saturation. Use to show whether the human needs to read this content carefully.

• Reduce opacity only for content the human genuinely does not need to read, e.g., repeated boilerplate that is identical across records, verbose tool schemas, auto-generated headers. Ask yourself: "if I removed this, would the human miss anything?"
• Do not mute content by role. System messages, tool results, and thinking blocks can all contain the actual bug. Mute only specific content that is redundant or mechanical, regardless of which role produced it.
• Normal content gets full opacity.

Spacing. Use for hierarchical structure.

• Tight spacing (4 to 8px) between items within a logical group, e.g., consecutive messages in one "step" of an agent trace.
• Medium spacing (16 to 24px) between logical groups, e.g., between steps.
• Large spacing (32 to 48px) between major sections.

Typography. Use for content type.

• Natural language prose: proportional font, normal size.
• Code: monospace font, slightly smaller.
• Metadata: small, muted, compact.
• Thinking/reasoning: italic or a distinct font treatment to signal "internal."

Border / container. Use for grouping related parts.

• Put a tool call and its tool result in a shared container with a subtle border.
• Separate an input/output pair with a clear divider, or place them side by side.

Structural outlier flags. Show in the header only, not inline.

• Pre-compute dataset-level averages for key structural features (length, heading count, paragraph count, etc.).
• For each record, flag dimensions where it is a clear statistical outlier (e.g., top/bottom 10%).
• Show these as small, compact badges in the header, e.g., "6 headings (more than 89%)" or "shorter than 95%."
• Keep it minimal. Most records should have zero or one flag. Do not annotate inline content.

Phase 3: Build the review interface

Architecture

• Python HTTP server (stdlib http.server, no dependencies):
• GET / serves the HTML app
• GET /api/samples returns the current sample set
• POST /api/samples lets the agent push new samples
• GET /api/annotations returns the current annotations
• POST /api/annotations lets the app save annotations on every change
• GET /api/graph returns the 2D projection of all records for the cluster map
• GET /api/patterns returns the agent's current failure mode taxonomy
• POST /api/patterns lets the agent push the updated taxonomy
• GET /api/suggestions returns agent-suggested annotations
• POST /api/suggestions lets the agent push suggestions
• On-disk files in an error_discovery_data/ directory:
• samples.json, annotations.json, graph.json, patterns.json, suggestions.json
• The HTML app auto-saves to the server on every annotation. It polls for new samples and suggestions.

HTML app structure

Three views, toggled from the top bar: