Rewardkit

Help the user write task verifiers with Reward Kit. Reward Kit is a lightweight Python package that turns a directory of criteria files into a reward score. Each criterion is a Python function call or a TOML judge file; folders become separate rewards.

Setup in a Harbor task

Put criteria alongside test.sh in the task's tests/ directory:

tests/
├── test.sh
├── checks.py         # programmatic criteria
└── judge.toml        # optional LLM/agent judge

tests/test.sh:

#!/bin/bash
uvx --from 'harbor-rewardkit==0.2.*' rewardkit /tests

This runs all criteria in /tests/ against the workspace at /app and writes /logs/verifier/reward.json. Defaults match Harbor's conventions — no extra config needed.

If judge criteria need API keys, pass them through task.toml:

[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"

Ask whether Reward Kit should run in the agent's shared environment or in a separate verifier environment. Prefer a separate verifier environment when judge prompts, grading dependencies, API keys, or clean-room checks should not be available to the agent:

[environment]
network_mode = "no-network"   # Agent env baseline — offline during agent.run()

[verifier]
environment_mode = "separate"

[verifier.environment]
network_mode = "public"     # Verifier env baseline — LLM judge API calls
docker_image = "python:3.12-slim"

In shared mode, the verifier runs in the agent container and inherits [environment].network_mode. Put [verifier].network_mode only when verify() needs different network access than the agent phase (a phase override, not a baseline). If agent and verifier need different baselines without runtime switching, use environment_mode = "separate" and set [verifier.environment].network_mode.

Judge criteria that call external APIs need a public baseline or allowlist on the verifier environment. Programmatic checks that only read local files can use no-network.

In separate mode, tests/ is the verifier image build context and must provide /tests/test.sh at runtime; Harbor does not upload tests/ into the running verifier container.

Programmatic criteria

Call built-ins from any .py file in tests/:

import rewardkit as rk

rk.file_exists("output.txt")
rk.file_contains("output.txt", "hello")
rk.command_succeeds("python main.py", weight=2.0)
rk.json_key_equals("result.json", "status", "ok")

All criteria accept weight (default 1.0) and isolated (default False, runs in overlayfs so side effects don't leak).

Available built-ins

• Files: file_exists, file_not_exists, file_contains, file_contains_regex, file_matches, files_equal, diff_ratio
• Commands: command_succeeds, command_output_contains, command_output_matches, command_output_matches_regex (30s default timeout, optional cwd)
• Data: json_key_equals, json_path_equals, csv_cell_equals, xlsx_cell_equals (needs [office] extra), sqlite_query_equals
• HTTP: http_status_equals, http_response_contains
• Images: image_similarity, image_size_equals (needs [image] extra)
• Trajectory: trajectory_tool_used, trajectory_tool_not_used, trajectory_turn_count

For extras, install with uv tool install harbor-rewardkit[all].

Custom criteria

Use the @criterion decorator. First parameter is always workspace: Path. Returns bool or float:

from pathlib import Path
from rewardkit import criterion

@criterion
def has_valid_output(workspace: Path) -> bool:
    return (workspace / "output.txt").read_text().strip() != ""

Zero-parameter criteria auto-register. Criteria with extra args must be called via rk:

@criterion(description="output has at least {n} lines")
def has_n_lines(workspace: Path, n: int) -> bool:
    return len((workspace / "output.txt").read_text().splitlines()) >= n

rk.has_n_lines(10, weight=2.0)
rk.has_n_lines(50, weight=1.0)

For criteria shared across reward subdirs, define with shared=True in a root-level file and call from subdirs.

Judge criteria (LLM or agent-as-a-judge)

For subjective checks (quality, readability, edge cases), create a TOML file:

[judge]
judge = "anthropic/claude-sonnet-5"   # LiteLLM model string
files = ["/app/main.py"]

[[criterion]]
description = "Is the code correct?"
type = "binary"

[[criterion]]
description = "How readable is the code?"
type = "likert"
points = 5
weight = 2.0

Criterion types:

• binary — yes/no → 1.0 or 0.0
• likert — 1..points, normalized to [0, 1]
• numeric — min..max, normalized to [0, 1]
• rubric — 2 to 10 described levels forming a scale from worst to best; position sets the score

Agent judges

Agent judges shell out to a CLI and can explore the filesystem:

[judge]
judge = "claude-code"
model = "anthropic/claude-sonnet-5"
isolated = true

[[criterion]]
description = "Does the solution handle edge cases?"
type = "binary"

Slower and more expensive than LLM judges, but they can run commands and inspect files.

JEV judge

JEV is a new type of language model from TypeSafe. It answers each criterion with a probability or a rubric score and returns no reasoning, so it is fast and cheap. It needs the jev extra (harbor-rewardkit[jev]) and TYPESAFE_API_KEY.

[judge]
judge = "jev"
files = ["/app/answer.md"]

[[criterion]]
description = "Does the answer address the requested task?"

[[criterion]]
description = "How complete is the answer?"
type = "rubric"
levels = ["Omits the information", "Covers part of it", "Covers all of it"]

Binary criteria pass at a probability of 0.5 or higher. Only binary and rubric criteria and text files are supported; atif-trajectory and prompt_template are not. The task image needs CA certificates (ca-certificates on Debian and Ubuntu), because the TypeSafe SDK verifies TLS against the system trust store.

Useful [judge] options

timeout (default 300), reasoning_effort (low|medium|high), reference (path to reference solution), atif-trajectory (evaluate the agent's trajectory), weight, prompt_template (custom prompt with {criteria} placeholder).

Scoring aggregation (within one judge TOML)

[scoring]
aggregation = "all-pass"   # weighted-mean | weighted-sum | all-pass | any-pass | threshold | required-pass
threshold = 0.7             # only for threshold

Only affects how this file's own criteria combine. To aggregate across dimensions, see Aggregating dimensions.

Scoring config for programmatic files

Each .py file that registers criteria is an equal-weighted scoring component named after its filename stem. Files that only provide imports or shared criterion factories and register no checks are ignored. To change how criteria within a file combine, use [scoring.<stem>] in the same directory's reward.toml:

# tests/structure/reward.toml
[scoring.files_exist]       # configures files_exist.py
aggregation = "all-pass"

[scoring.behavior]          # configures behavior.py
aggregation = "threshold"
threshold = 0.75

Each entry takes the same aggregation values as a judge TOML. Unknown keys and stems that do not resolve to a criterion-bearing Python file raise.

Directories may be nested recursively. A non-root directory can aggregate its local Python files, local judges, and immediate child directories with one unnamed [[reward]] table:

# tests/correctness/reward.toml
[[reward]]
aggregation = "weighted-mean"
weights = { files = 2.0, behavior = 1.0 }

Membership is implicit. Child directories have weight 1.0 unless overridden; use filename stems for local Python files and judge TOMLs, and directory names for child groups. Without [[reward]], the directory defaults to weighted mean.

Multi-reward tasks

Put criteria in subdirectories — each becomes a separate reward:

tests/