Create Task

Guide the user through creating a new Harbor task end-to-end. Don't just dump commands — walk them through each decision, especially around the verifier (which is usually the hardest part).

Step 1: Scaffold the task

harbor task init "<org>/<task-name>"

Useful flags:

• --description "..."
• --author "Jane Doe <jane@example.com>" (repeat for multiple authors)
• --no-pytest — skip the pytest test template (use if planning Reward Kit or custom verifier)
• --no-solution — skip solution/ directory
• --metadata-template path.toml — pre-populate task.toml

Produces:

<task-name>/
├── instruction.md         # Task prompt for the agent
├── task.toml              # Config and metadata
├── environment/Dockerfile # Container definition
├── solution/solve.sh      # Reference solution (optional)
└── tests/test.sh          # Verifier script

If the user wants a multi-step task (ordered steps with per-step instructions, tests, and early stopping against a shared container), scaffold the single-step layout first, then convert to the steps/ layout described in the Multi-step tasks section below.

Step 2: Write instruction.md

This is the prompt the agent receives. Help the user write it clearly:

• State the goal concretely — what file to create, what behavior to produce
• Specify expected outputs — paths, formats, content
• Include constraints — language, tools, approach
• Don't leak the tests — describe what "done" looks like, not how you'll check it

Example (from the ssh-key-pair tutorial):

# SSH Key Pair Generation

Generate an SSH key pair in the files `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub`.

Don't make them password protected.

Step 3: Build the environment

Edit environment/Dockerfile to install dependencies the task needs. The agent works inside this container.

FROM ubuntu:24.04
WORKDIR /app

# Install what the task requires — NOT the solution
RUN apt-get update && apt-get install -y openssh-client && rm -rf /var/lib/apt/lists/*

For multi-container setups, use environment/docker-compose.yaml instead (note: most cloud sandbox providers only support Dockerfile).

Test the environment interactively before writing the solution or tests:

harbor task start-env -p "<task-path>" -e docker -a -i

This is usually where task authors realize something is missing from the Dockerfile.

Step 4: Decide how to verify

This is the most important decision. Ask the user: "How do you want to grade this task?" Then help them pick:

Also ask: "Should the verifier run in the same environment as the agent, or in a separate verifier environment?"

• Use the default shared environment when tests need to inspect the agent's full workspace, installed tools, or services.
• Use a separate verifier environment when grading code, dependencies, API keys, or OS requirements should stay hidden from the agent, or when verification should run from a clean image.

For a separate verifier container, tests/ is the verifier image build context and the image must provide /tests/test.sh (Linux) or /tests/test.bat (Windows). Harbor copies /logs/artifacts and configured artifacts into the verifier environment, not the agent's whole workspace.

[verifier]
environment_mode = "separate"

[verifier.environment]
docker_image = "ubuntu:24.04"

Option A: Reward Kit (recommended for most cases)

Use when the verifier has multiple criteria, needs partial credit, uses an LLM/agent judge, or would benefit from composable reusable checks. See the rewardkit skill.

Good fit signals:

• Multiple things to check (file exists + content correct + command works)
• Subjective quality dimensions (readability, correctness of prose)
• Want partial credit rather than pass/fail
• Want to compose built-ins like file_contains, command_succeeds, json_key_equals

tests/test.sh:

#!/bin/bash
uvx --from 'harbor-rewardkit==0.2.*' rewardkit /tests

Note: the package is named harbor-rewardkit but the executable is rewardkit, hence --from 'harbor-rewardkit==0.2.*' rewardkit. Running uvx harbor-rewardkit directly will fail.

Then add tests/checks.py and/or tests/judge.toml. Invoke the rewardkit skill to design the criteria.

Option B: pytest (good for deterministic unit-style checks)

Use when the verification is straightforward assertion-style Python. Default template if --no-pytest wasn't passed.

tests/test.sh:

#!/bin/bash
apt-get update && apt-get install -y curl
curl -LsSf https://astral.sh/uv/0.9.7/install.sh | sh
source $HOME/.local/bin/env

uvx --with pytest==8.4.1 pytest /tests/test_outputs.py

if [ $? -eq 0 ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

Example tests/test_outputs.py:

from pathlib import Path

def test_file_exists():
    assert (Path.home() / ".ssh" / "id_rsa").exists()

Option C: Custom shell

For simple single-command checks (e.g. a binary pass/fail from one command):

#!/bin/bash
if diff -q /app/output.txt /tests/expected.txt; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

Reward file format (all options)

• /logs/verifier/reward.txt — single number (usually 0 or 1)
• /logs/verifier/reward.json — {"accuracy": 0.95, "runtime_sec": 1.2} for multiple metrics

Always use absolute paths in `test.sh`.

Step 5: Write the solution

Write solution/solve.sh — a script that actually solves the task. The Oracle agent runs this to sanity-check that the task is solvable and the tests pass on a correct solution.

#!/bin/bash
ssh-keygen -t rsa -f ~/.ssh/id_rsa -N ""

Make it executable: chmod +x solution/solve.sh.

Step 6: Configure task.toml

Walk through the important fields:

[task]
name = "<org>/<task-name>"
version = "1.0.0"
description = "One-line description"
keywords = ["jax", "mnist", "rewardkit"]  # always populate — used for search/filtering

[metadata]
difficulty = "easy" | "medium" | "hard"
category = "programming" | "machine-learning" | "gpu" | ...
tags = ["..."]

[agent]
timeout_sec = 120.0       # How long the agent has

[verifier]
timeout_sec = 600.0       # How long tests have

[environment]
network_mode = "public"   # Baseline at env start (defaults to public)
cpus = 1                  # CPU cores
memory_mb = 2048          # RAM in MB
storage_mb = 10240        # Disk in MB

Always populate `keywords`. Pick 3–8 lowercase tokens covering the domain (language/framework/benchmark family), the verifier style (rewardkit, judge-grading, pytest), and any notable hardware (gpu). They're surfaced in harbor datasets list and registry search.

Network policy

Network access has three layers:

1. Baselines — set when an environment starts, restored between phases
2. Phase overrides — optional; only during agent.run() or verify()
3. Run-time merges — --allow-environment-host, --allow-agent-host on harbor run

Modes: public, no-network, or allowlist with allowed_hosts = ["pypi.org"] (exact hostnames, IPv4/IPv6 address literals or CIDR ranges, or leading wildcard hostnames, when supported by the selected environment; not URLs,