Grade System

Grade System

You are applying substantive quality and reliability grading to Ane's MEL/SRHR system. Output is a Tier 1 working brief with letter grades, dimensional analysis, OECD-DAC adapted criteria, concrete risks, and recommendations for grade movement. Honest grading. Self-grading bias acknowledged openly.

When to use this

• Ane types /grade-system, /grade, or asks how good the system is
• Periodic quality read (suggested cadence: monthly, or after major bias-correction work)
• Before stakeholder communication where you need to characterise the system honestly
• After substantial system changes to assess whether they moved the grade

Mandatory self-grading bias note (always include in output)

You are both grading the system and operating inside it. Per the 2026-05-06 mel-system-bias-audit, the LLM-as-Judge effect makes you likely to favour outputs and architectures that match your own reasoning patterns. State this openly in the output. The grade is the system's internal calibration, not external validation. The structural test (community / external MA practitioner review) remains the test the system has not yet permitted to exist.

This note is non-negotiable. Skip it and the grade is misleading.

Workflow

Step 1 — Gather empirical evidence

Read in parallel:

• agent-improvements/ann-overlay.md (Active + Archived) — extract retrospective bullets with cost figures, harness state, and key learnings. Count sessions. Note any major incidents (e.g., DOCX truncation, em-dash regression).
• agent-improvements/coordination-log.md — friction events with status (OPEN / CLOSED).
• agent-improvements/cost-calibration-log.md — count rows with observed actuals vs not observed. Note ⚠️ over-band entries.
• agent-improvements/qa-rejection-log.md — count data rows and trend the verdicts; this is the live QA telemetry signal. An empty table means the improvement loop steers on anecdote — flag it as a risk. qa-disagreement-log.md is historical only: retired as a live counter 2026-07-30 (system-audit item #12), its 3-row Vi/Li elevation watch-trigger withdrawn as unmeasurable. Read its header note for the retirement rationale; do not count its rows as a watch-trigger. A table with rows in it can still be empty for trend purposes. The log's own protocol excludes any row tagged ⚠️ PILOT or ⚠️ SMOKE TEST from the rejection-rate trend, so count the trend-eligible rows separately and report that number, not the row count (grep -E '^\| 2026-' <log> | grep -vc -E '(PILOT|SMOKE TEST)'). On 2026-08-15 all ten rows carried an exclusion tag, so the declared measurable signal held zero data while the table looked populated. The external structural test remains open regardless.
• Audit history — do not glob for it, most of it has moved. agent-improvements/system-audit-*.md matches exactly one file, the 2026-04-28 architecture audit, which is still worth reading as the baseline but is not recent. agent-improvements/audit-drift-*.md matches a weekly series that was RETIRED on 2026-08-05 (Ane's decision; the harness carries its checks and /system-audit covers the judgement pass on demand), so its newest file is a closing entry, not a current reading. Recent audit findings live in agent-improvements/handoffs/ (for example system-audit-follow-ups.md, whose locked-decisions section exists nowhere else) and in agent-improvements/improve-system-runs.md. Run python scripts/handoff_status.py for the current set rather than guessing filenames.
• mel_wiki/wiki/index.md — page counts (frameworks, concepts, indicators, lenses).
• agent-improvements/agent_registry.md — count specialists; cross-reference with ~/.claude/agents/.
• Run python tests/run_tests.py for static state and python tests/run_tests.py --output for fixture coverage.

Step 2 — Apply per-component grades

Grade scale: A / A− / B+ / B / B− / C+ / C and below. Use evidence from Step 1, not vibes.

For each component: assign a grade with one-line rationale citing the evidence file.

Step 3 — Apply per-dimension grades

For each dimension: assign a grade. Each must be defensible from the evidence collected in Step 1. Do not give A grades without specific evidence; do not give C grades without specific gap.

Then answer two graph-shape questions explicitly. The orchestration is an agent graph (jobs, arrows, shared state files, checkers, human gates); fold both answers into the Reliability grade and raise any "no" as a Step 5 risk.

1. Is every checker separated from its writer? qa-reviewer and reader-position-reviewer carry no Write/Edit tools by design; verify that stays true, and that any newly added reviewer or QA skill also reviews without writing. A checker that can edit the product it reviews is a graph defect, whatever its prose quality.
2. Is each human gate placed where mistakes are expensive, not where convenient? Map where Ane's approval is requested (plan approval, pre-publish, pre-overwrite) against where an error would actually cost her (outward-facing sends, destructive writes, donor-visible content). A gate at a cheap step with none at an expensive step is a misplacement; the Run 4 post-decision-Q&A overrun was this symptom.

Step 4 — Apply OECD-DAC adapted (6 criteria)

Adapt the 2019 six-criteria framework to the system context: