Judge With Debate

judge-with-debate

<task> Evaluate solutions through multi-agent debate where independent judges analyze, challenge each other's assessments, and iteratively refine their evaluations until reaching consensus or maximum rounds. </task>

<context> This command implements the Multi-Agent Debate pattern for high-quality evaluation where multiple perspectives and rigorous argumentation improve assessment accuracy. Unlike single-pass evaluation, debate forces judges to defend their positions with evidence and consider counter-arguments.

Key benefits:

• Structured evaluation - Meta-judge produces tailored rubrics and criteria before judging begins
• Multiple perspectives - Three independent judges reduce individual bias
• Evidence-based debate - Judges defend positions with specific evidence from the solution and evaluation specification
• Iterative refinement - Up to 3 debate rounds drive convergence on accurate scores
• Shared specification - Meta-judge runs once; all judges across all rounds share the same evaluation specification

</context>

Pattern: Debate-Based Evaluation

This command implements iterative multi-judge debate:

Phase 0: Setup
         mkdir -p .specs/reports
                  |
Phase 0.5: Dispatch Meta-Judge
         Meta-Judge (Opus)
              |
         Evaluation Specification YAML
              |
Phase 1: Independent Analysis (3 judges in parallel)
         +- Judge 1 -> {name}.1.md -+
Solution +- Judge 2 -> {name}.2.md -+-+
         +- Judge 3 -> {name}.3.md -+ |
                                      |
Phase 2: Debate Round (iterative)     |
    Each judge reads others' reports  |
         |                            |
    Argue + Defend + Challenge        |
    (grounded in eval specification)  |
         |                            |
    Revise if convinced --------------+
         |                            |
    Check consensus                   |
         +- Yes -> Final Report       |
         +- No -> Next Round ---------+

Process

Setup: Create Reports Directory

Before starting evaluation, ensure the reports directory exists:

mkdir -p .specs/reports

Report naming convention: .specs/reports/{solution-name}-{YYYY-MM-DD}.[1|2|3].md

Where:

• {solution-name} - Derived from solution filename (e.g., users-api from src/api/users.ts)
• {YYYY-MM-DD} - Current date
• [1|2|3] - Judge number

Phase 0.5: Dispatch Meta-Judge

Before independent analysis, dispatch a meta-judge agent to generate a tailored evaluation specification. The meta-judge runs ONCE and produces rubrics, checklists, and scoring criteria that ALL judges will use across ALL rounds.

Meta-judge prompt template:

## Task

Generate an evaluation specification yaml for the following evaluation task. You will produce rubrics, checklists, and scoring criteria that multiple judge agents will use to evaluate the solution through independent analysis and multi-round debate.

CLAUDE_PLUGIN_ROOT=`${CLAUDE_PLUGIN_ROOT}`

## User Prompt
{task description - what the solution was supposed to accomplish}

## Context
{Any relevant context about the solution being evaluated}

## Artifact Type
{code | documentation | configuration | etc.}

## Evaluation Mode
Multi-judge debate with consensus-seeking across rounds

## Instructions
Return only the final evaluation specification YAML in your response.
The specification should support both independent analysis and debate-based refinement.

Dispatch:

Use Task tool:
  - description: "Meta-judge: generate evaluation specification for {solution-name}"
  - prompt: {meta-judge prompt}
  - model: opus
  - subagent_type: "sadd:meta-judge"

Wait for the meta-judge to complete and extract the evaluation specification YAML from its output before proceeding to Phase 1.

Phase 1: Independent Analysis

Launch 3 independent judge agents in parallel (Opus for rigor):

1. Each judge receives:

• Path to solution(s) being evaluated
• The meta-judge's evaluation specification YAML
• Task description

1. Each produces independent assessment saved to .specs/reports/{solution-name}-{date}.[1|2|3].md
2. Reports must include:

• Per-criterion scores with evidence
• Specific quotes/examples supporting ratings
• Overall weighted score
• Key strengths and weaknesses

Key principle: Independence in initial analysis prevents groupthink.

Prompt template for initial judges:

You are Judge {N} evaluating a solution independently against an evaluation specification produced by the meta judge.

CLAUDE_PLUGIN_ROOT=`${CLAUDE_PLUGIN_ROOT}`

## Solution
{path to solution file(s)}

## Task Description
{what the solution was supposed to accomplish}

## Evaluation Specification

{meta-judge's evaluation specification YAML}

## Output File
.specs/reports/{solution-name}-{date}.{N}.md

## Instructions

Follow your full judge process as defined in your agent instructions!

Additional instructions:
1. Read the solution thoroughly
2. For each criterion from the evaluation specification:
   - Find specific evidence (quote exact text)
   - Score on the defined scale
   - Justify with concrete examples
3. Calculate weighted overall score
4. Write comprehensive report to {output_file}

Add to report beginning `Done by Judge {N}`

Dispatch each judge:

Use Task tool:
  - description: "Judge {N}: independent analysis of {solution-name}"
  - prompt: {judge prompt with evaluation specification YAML}
  - model: opus
  - subagent_type: "sadd:judge"

Phase 2: Debate Rounds (Iterative)

For each debate round (max 3 rounds):

Launch 3 debate agents in parallel:

1. Each judge agent receives:

• Path to their own previous report (.specs/reports/{solution-name}-{date}.[1|2|3].md)
• Paths to other judges' reports (.specs/reports/{solution-name}-{date}.[1|2|3].md)
• The original solution
• The meta-judge's evaluation specification YAML

1. Each judge:

• Identifies disagreements with other judges (>1 point score gap on any criterion)
• Defends their own ratings with evidence from the solution and evaluation specification
• Challenges other judges' ratings they disagree with
• Considers counter-arguments
• Revises their assessment if convinced

1. Updates their report file with new section: ## Debate Round {R}
2. After they reply, if they reached agreement move to Phase 3: Consensus Report

Key principle: Judges communicate only through filesystem - orchestrator doesn't mediate and don't read reports files itself, it can overflow your context.

Prompt template for debate judges:

You are Judge {N} in debate round {R}.

CLAUDE_PLUGIN_ROOT=`${CLAUDE_PLUGIN_ROOT}`

## Your Previous Report
{path to .specs/reports/{solution-name}-{date}.{N}.md}

## Other Judges' Reports
Judge 1: .specs/reports/{solution-name}-{date}.1.md
...

## Task Description
{what the solution was supposed to accomplish}

## Solution
{path to solution}

## Evaluation Specification

{meta-judge's evaluation specification YAML}

## Output File
.specs/reports/{solution-name}-{date}.{N}.md (append to existing file)

## Instructions

Follow your full judge process as defined in your agent instructions!

Additional debate instructions:
1. Read your previous assessment from {your_previous_report}
2. Read all other judges' reports
3. Identify disagreements (where your scores differ by >1 point)
4. For each major disagreement:
   - State the disagreement clearly
   - Defend your position with evidence from the solution and evaluation specification
   - Challenge the other judge's position with counter-evidence
   - Consider whether their evidence changes your view
5. Update your report file by APPENDING debate round section
6. Reply whether you reached agreement, and with which judge. Include revisited scores and criteria scores.

CRITICAL: