Evaluation Methodology

Evaluation Methodology

This document is the authoritative reference for how PluginEval measures plugin and skill quality. It covers the three evaluation layers, all ten scoring dimensions, the composite formula, badge thresholds, anti-pattern flags, Elo ranking, and actionable improvement tips.

Related: Full rubric anchors

The Three Evaluation Layers

PluginEval stacks three complementary layers. Each layer produces a score between 0.0 and 1.0 for each applicable dimension, and later layers override or blend with earlier ones according to per-dimension blend weights.

Layer 1 — Static Analysis

Speed: < 2 seconds. No LLM calls. Deterministic.

The static analyzer (layers/static.py) runs six sub-checks directly against the parsed SKILL.md:

These six sub-checks feed directly into six of the ten final dimensions (via STATIC_TO_DIMENSION mapping). The remaining four dimensions — output_quality, scope_calibration, robustness, and part of triggering_accuracy — receive no static contribution and rely entirely on Layer 2 and/or Layer 3.

Anti-pattern penalty is applied multiplicatively to the Layer 1 score:

penalty = max(0.5, 1.0 − 0.05 × anti_pattern_count)

Each additional detected anti-pattern reduces the score by 5%, flooring at 50%.

Layer 2 — LLM Judge

Speed: 30–90 seconds. One or more LLM calls (Sonnet by default). Non-deterministic.

The eval-judge agent reads the SKILL.md and any references/ files, then scores four dimensions using anchored rubrics (see references/rubrics.md):

1. Triggering accuracy — F1 score derived from 10 mental test prompts
2. Orchestration fitness — Worker purity assessment (0–1 rubric)
3. Output quality — Simulates 3 realistic tasks; assesses instruction quality
4. Scope calibration — Judges depth and breadth relative to the skill's category

The judge returns a structured JSON object (no markdown fences) that the eval engine merges into the composite. When judges > 1, scores are averaged and Cohen's kappa is reported as an inter-judge agreement metric.

Layer 3 — Monte Carlo Simulation

Speed: 5–20 minutes. N=50 simulated Agent SDK invocations (default). Statistical.

Monte Carlo runs N real prompts through the skill and records:

• Activation rate — Fraction of prompts that triggered the skill
• Output consistency — Coefficient of variation (CV) across quality scores
• Failure rate — Error/crash fraction with Clopper-Pearson exact CIs
• Token efficiency — Median token count, IQR, outlier count

The Layer 3 composite formula:

mc_score = 0.40 × activation_rate
         + 0.30 × (1 − min(1.0, CV))
         + 0.20 × (1 − failure_rate)
         + 0.10 × efficiency_norm

where efficiency_norm = max(0, 1 − median_tokens / 8000).

Composite Scoring Formula

The final score is a weighted blend across all three layers for each dimension, then summed:

composite = Σ(dimension_weight × blended_dimension_score) × 100 × anti_pattern_penalty

Dimension Weights

Layer Blend Weights

Each dimension draws from different layers at different ratios. With all three layers active (--depth deep or certify):

At --depth standard (static + judge only), blends are renormalized to drop the Monte Carlo column. At --depth quick (static only), all weight falls on Layer 1.

Blended Score Calculation

For a given depth, the blended score for dimension d is:

blended[d] = Σ( layer_weight[d][layer] × layer_score[d][layer] )
             ─────────────────────────────────────────────────────
             Σ( layer_weight[d][layer] for available layers )

This normalization ensures that skipping Monte Carlo at standard depth doesn't artificially deflate scores.

Interpreting Dimension Scores

Each dimension score is a float in [0.0, 1.0]. The CLI converts it to a letter grade:

When reading a report, focus first on the lowest-graded dimension that has the highest weight. A D in triggering_accuracy (weight 0.25) costs far more than a D in ecosystem_coherence (weight 0.02).

Confidence intervals appear in the report when Layer 2 or Layer 3 ran. Narrow CIs (± < 5 points) indicate stable scores. Wide CIs suggest inconsistency — often caused by an ambiguous description or instructions that work for some prompt styles but not others.

Quality Badges

Badges require both a composite score threshold AND an Elo threshold (when Elo is available). The Badge.from_scores() logic checks composite first, then Elo if provided:

The Elo threshold is skipped when Elo has not been computed (i.e., at quick or standard depth without certify). A skill can earn a badge on composite score alone in those cases.

Anti-Pattern Flags

The static analyzer detects five anti-patterns. Each carries a severity multiplier that feeds into the penalty formula.

OVER_CONSTRAINED

Trigger: More than 15 occurrences of MUST, ALWAYS, or NEVER in the SKILL.md.

Problem: Overly prescriptive instructions reduce model flexibility, increase token overhead, and signal that the author is trying to micromanage every output rather than providing principled guidance.