AI Bug Triage

<objective> A hybrid pipeline for bug classification, deduplication, and ticket generation. Deterministic fingerprinting handles deduplication (what LLMs are bad at); LLM handles explanation, severity assessment, and ticket writing (what LLMs are good at).

Key reframe: The LLM is best at explaining and routing, not deduplication. Teach agents to DESIGN the pipeline, not BE the pipeline. </objective>

Discovery Questions

Check .agents/qa-project-context.md first — it carries tech stack, component mapping, and known flaky areas that improve classification accuracy. Use it and skip anything already answered there. Then clarify:

1. What is the failure source?

• CI pipeline logs (GitHub Actions, GitLab CI, Jenkins, CircleCI)
• Test framework output (Playwright, Jest, pytest, Vitest)
• Production error monitoring (Sentry, Datadog, Bugsnag)
• Manual bug reports from QA or users

1. What is the ticket destination?

• Jira, Linear, GitHub Issues, Azure DevOps, Shortcut
• What fields are required? (component, severity, priority, labels)
• What workflows exist? (triage board, auto-assignment rules)

1. What is the deduplication scope?

• Same test run? Same sprint? Same release? All time?
• Do you already have fingerprinting? What is the current duplicate rate?

1. What approval workflow is needed?

• Auto-create tickets with human review?
• Suggest tickets for human approval before creation?
• Auto-close duplicates? (dangerous -- require approval)

1. What historical data exists?

• Past bug reports with resolution data?
• Flaky test history? Known environment issues?
• Component ownership mapping?

Core Principles

1. Deterministic first, LLM second. Use stable, reproducible fingerprinting for deduplication and clustering. Use LLM only for tasks requiring understanding: severity classification, root cause hypothesis, and human-readable ticket writing.

1. Normalize before comparing. Raw CI logs are full of timestamps, port numbers, process IDs, and random suffixes that make identical failures look different. Strip all noise before fingerprinting.

1. Fingerprints are anchored to stable elements. Exception type, top stack frames, test name, error message template, and URL pattern are stable. Timestamps, request IDs, and ephemeral ports are not.

1. Human approval before destructive actions. Auto-closing a ticket as duplicate or auto-merging reports requires human confirmation. False deduplication wastes more time than manual triage.

1. Classification drives routing. The value of triage is not the label itself but the routing decision it enables: which team, what priority, what SLA.

1. Track triage accuracy. Measure how often auto-classification matches human judgment. Below 85% accuracy, the pipeline needs tuning.

The Pipeline

CI Log / Error Report
  │
  ▼
Step 1: NORMALIZE
  Strip timestamps, process IDs, ports, random suffixes, ANSI codes
  │
  ▼
Step 2: EXTRACT STABLE ANCHORS
  Exception type, top N stack frames, test name, error message template, URL pattern
  │
  ▼
Step 3: HASH CANONICAL FORM
  Deterministic fingerprint from ordered anchors
  │
  ▼
Step 4: CLUSTER NEAR-DUPLICATES
  Similarity scoring for non-identical but related failures
  │
  ▼
Step 5: LLM CLASSIFY
  Severity, component, suspected root cause, failure category
  │
  ▼
Step 6: LLM GENERATE TICKET
  Title, description, repro steps, evidence, suggested assignee
  │
  ▼
Step 7: HUMAN APPROVAL
  Review before create/close/merge

Step 1: Normalize

Strip noise that makes identical failures look different.

Normalization rules (apply in order):

1. Strip ANSI color codes:        \x1b\[[0-9;]*m → ""
2. Strip timestamps:              \d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}[.\d]*Z? → "<TIMESTAMP>"
3. Strip UUIDs:                   [0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12} → "<UUID>"
4. Strip process IDs:             pid[=: ]\d+ → "pid=<PID>"
5. Strip port numbers:            :\d{4,5}(?=[\s/,)\]]|$) → ":<PORT>"
6. Strip temp file paths:         /tmp/[^\s]+ → "<TMPPATH>"
7. Strip memory addresses:        0x[0-9a-f]{8,16} → "<ADDR>"
8. Strip random suffixes:         [-_][a-z0-9]{6,8}(?=\.) → "<RAND>"
9. Strip request IDs:             (?:request[_-]?id|trace[_-]?id|correlation[_-]?id)[=: ]["']?[a-zA-Z0-9-]+ → "<REQ_ID>"
10. Collapse whitespace:          \s+ → " "

Example:

Before: 2025-03-22T14:32:01.456Z [pid=42891] Error: Connection refused at 127.0.0.1:54321
        request_id=abc-123-def-456
After:  <TIMESTAMP> [pid=<PID>] Error: Connection refused at 127.0.0.1:<PORT>
        <REQ_ID>

Rule 5 strips the port but not the literal loopback IP — 127.0.0.1 stays in the fingerprint. That's fine for same-host failures, but two runners that bind different hosts (e.g. 127.0.0.1 vs 0.0.0.0) will split into separate fingerprints. If you run heterogeneous hosts, add a rule to normalize bind addresses too.

Step 2: Extract Stable Anchors

From the normalized log, extract elements that identify the failure regardless of environment or timing.

Anchor types (in priority order):

Extraction rules:

• Keep function names but strip line numbers (they change with edits)
• Keep URL paths but strip query parameters and IDs in paths (/api/orders/<ID>)
• Keep error message structure but replace dynamic values with placeholders
• Keep test file and test name exactly as-is

Step 3: Hash Canonical Form

Create a deterministic fingerprint from the extracted anchors.

Algorithm:

1. Sort anchors alphabetically by type
2. Concatenate: exception_type + "|" + message_template + "|" + top_frames + "|" + test_name
3. SHA-256 hash the concatenated string
4. Take first 16 hex characters as fingerprint

Fingerprint properties:

• Same failure always produces same fingerprint (deterministic)
• Different failures produce different fingerprints (collision-resistant)
• Minor log format changes do not change fingerprint (stable)
• Fingerprint is short enough for Jira labels and GitHub tags

Example:

Anchors:
  exception_type: "TypeError"
  message_template: "Cannot read property 'vendorId' of undefined"
  top_frames: "processOrder|groupByVendor|checkout"
  test_name: "checkout.spec.ts > multi-vendor checkout"

Canonical: "TypeError|Cannot read property 'vendorId' of undefined|processOrder|groupByVendor|checkout|checkout.spec.ts > multi-vendor checkout"
Fingerprint: a3f8b2c1e9d04567

Step 4: Cluster Near-Duplicates

Exact fingerprint matching catches identical failures. Similarity scoring catches related failures that differ slightly (same root cause, different manifestation).

Similarity dimensions:

Clustering threshold: similarity score > 0.75 = likely duplicate, suggest merge.

Human review required for:

• Scores between 0.60 and 0.75 (ambiguous)
• First occurrence of a new fingerprint (no history to compare)
• Failures in components with known intermittent issues