Skill Judge

Skill Judge

Evaluate Agent Skills against official specifications and patterns derived from 17+ official examples.

Core Philosophy

What is a Skill?

A Skill is NOT a tutorial. A Skill is a knowledge externalization mechanism.

Traditional AI knowledge is locked in model parameters. To teach new capabilities:

Traditional: Collect data → GPU cluster → Train → Deploy new version
Cost: $10,000 - $1,000,000+
Timeline: Weeks to months

Skills change this:

Skill: Edit SKILL.md → Save → Takes effect on next invocation
Cost: $0
Timeline: Instant

This is the paradigm shift from "training AI" to "educating AI" — like a hot-swappable LoRA adapter that requires no training. You edit a Markdown file in natural language, and the model's behavior changes.

The Core Formula

Good Skill = Expert-only Knowledge − What Claude Already Knows

A Skill's value is measured by its knowledge delta — the gap between what it provides and what the model already knows.

• Expert-only knowledge: Decision trees, trade-offs, edge cases, anti-patterns, domain-specific thinking frameworks — things that take years of experience to accumulate
• What Claude already knows: Basic concepts, standard library usage, common programming patterns, general best practices

When a Skill explains "what is PDF" or "how to write a for-loop", it's compressing knowledge Claude already has. This is token waste — context window is a public resource shared with system prompts, conversation history, other Skills, and user requests.

Tool vs Skill

Tools define capability boundaries — without bash tool, model can't execute commands. Skills inject knowledge — without frontend-design Skill, model produces generic UI.

The equation:

General Agent + Excellent Skill = Domain Expert Agent

Same Claude model, different Skills loaded, becomes different experts.

Three Types of Knowledge in Skills

When evaluating, categorize each section:

The art of Skill design is maximizing Expert content, using Activation sparingly, and eliminating Redundant ruthlessly.

Evaluation Dimensions (120 points total)

D1: Knowledge Delta (20 points) — THE CORE DIMENSION

The most important dimension. Does the Skill add genuine expert knowledge?

Red flags (instant score ≤5):

• "What is [basic concept]" sections
• Step-by-step tutorials for standard operations
• Explaining how to use common libraries
• Generic best practices ("write clean code", "handle errors")
• Definitions of industry-standard terms

Green flags (indicators of high knowledge delta):

• Decision trees for non-obvious choices ("when X fails, try Y because Z")
• Trade-offs only an expert would know ("A is faster but B handles edge case C")
• Edge cases from real-world experience
• "NEVER do X because [non-obvious reason]"
• Domain-specific thinking frameworks

Evaluation questions:

1. For each section, ask: "Does Claude already know this?"
2. If explaining something, ask: "Is this explaining TO Claude or FOR Claude?"
3. Count paragraphs that are Expert vs Activation vs Redundant

D2: Mindset + Appropriate Procedures (15 points)

Does the Skill transfer expert thinking patterns along with necessary domain-specific procedures?

The difference between experts and novices isn't "knowing how to operate" — it's "how to think about the problem." But thinking patterns alone aren't enough when Claude lacks domain-specific procedural knowledge.

Key distinction: | Type | Example | Value | |------|---------|-------| | Thinking patterns | "Before designing, ask: What makes this memorable?" | High — shapes decision-making | | Domain-specific procedures | "OOXML workflow: unpack → edit XML → validate → pack" | High — Claude may not know this | | Generic procedures | "Step 1: Open file, Step 2: Edit, Step 3: Save" | Low — Claude already knows |

What counts as valuable procedures:

• Workflows Claude hasn't been trained on (new tools, proprietary systems)
• Correct ordering that's non-obvious (e.g., "validate BEFORE packing, not after")
• Critical steps that are easy to miss (e.g., "MUST recalculate formulas after editing")
• Domain-specific sequences (e.g., MCP server's 4-phase development process)

What counts as redundant procedures:

• Generic file operations (open, read, write, save)
• Standard programming patterns (loops, conditionals, error handling)
• Common library usage that's well-documented

Expert thinking patterns look like:

Before [action], ask yourself:
- **Purpose**: What problem does this solve? Who uses it?
- **Constraints**: What are the hidden requirements?
- **Differentiation**: What makes this solution memorable?

Valuable domain procedures look like:

### Redlining Workflow (Claude wouldn't know this sequence)
1. Convert to markdown: `pandoc --track-changes=all`
2. Map text to XML: grep for text in document.xml
3. Implement changes in batches of 3-10
4. Pack and verify: check ALL changes were applied

Redundant generic procedures look like:

Step 1: Open the file
Step 2: Find the section
Step 3: Make the change
Step 4: Save and test

The test:

1. Does it tell Claude WHAT to think about? (thinking patterns)
2. Does it tell Claude HOW to do things it wouldn't know? (domain procedures)

A good Skill provides both when needed.

D3: Anti-Pattern Quality (15 points)

Does the Skill have effective NEVER lists?

Why this matters: Half of expert knowledge is knowing what NOT to do. A senior designer sees purple gradient on white background and instinctively cringes — "too AI-generated." This intuition for "what absolutely not to do" comes from stepping on countless landmines.

Claude hasn't stepped on these landmines. It doesn't know Inter font is overused, doesn't know purple gradients are the signature of AI-generated content. Good Skills must explicitly state these "absolute don'ts."

Expert anti-patterns (specific + reason):

NEVER use generic AI-generated aesthetics like:
- Overused font families (Inter, Roboto, Arial)
- Cliched color schemes (particularly purple gradients on white backgrounds)
- Predictable layouts and component patterns
- Default border-radius on everything

Weak anti-patterns (vague, no reasoning):

Avoid making mistakes.
Be careful with edge cases.