Arize Evaluator

Arize Evaluator Skill

`SPACE` — --space flags accept a space name (e.g., my-workspace) or a base64 space ID (e.g., U3BhY2U6...). Find yours with ax spaces list.

This skill covers designing, creating, and running evaluators on Arize — both LLM-as-judge (template) evaluators and code evaluators (deterministic, no LLM required). An evaluator defines the judge; a task is how you run it against real data.

Prerequisites

Proceed directly with the task — run the ax command you need. Do NOT check versions, env vars, or profiles upfront.

If an ax command fails, troubleshoot based on the error:

• command not found or version error → see references/ax-setup.md
• 401 Unauthorized / missing API key → run ax profiles show to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys
• Space unknown → run ax spaces list to pick by name, or ask the user
• LLM provider call fails (missing provider credentials) → run ax ai-integrations list --space SPACE to check for platform-managed credentials. If none exist, use the arize-ai-provider-integration skill — never ask the user to paste a provider key into chat.
• Security: Never read .env files or search the filesystem for credentials. Use ax profiles for Arize credentials and ax ai-integrations for LLM provider keys. Never ask the user to paste secrets into chat. For missing credentials, see references/ax-profiles.md.
• CRITICAL — Never fabricate evaluation results: If an evaluation task fails, is cancelled, or produces no scores, report the failure clearly and explain what went wrong. Do NOT perform a "manual evaluation," invent quality scores, estimate percentages, or present any agent-generated analysis as if it came from the Arize evaluation system. Instead suggest: (1) fix the identified issue and retry, (2) try running from the Arize UI, (3) verify integration credentials with ax ai-integrations list, (4) contact support at https://arize.com/support

Concepts

What is an Evaluator?

An evaluator is an LLM-as-judge definition. It contains:

Evaluators are versioned — every prompt or model change creates a new immutable version. The most recent version is active.

Code evaluators are the deterministic alternative — no AI integration or model, just Python. They run as a class subclassing CodeEvaluator, not a bare function, and have their own strict import-path and evaluate()-signature contract. Getting either wrong makes a run cancel silently at 0/0/0. See "Custom Python code evaluators" in references/cli-reference.md before writing one.

What is a Task?

A task is how you run one or more evaluators against real data. Tasks are attached to a project (live traces/spans) or a dataset (experiment runs). A task contains:

Data Granularity

Set --data-granularity when creating the evaluator (ax evaluators create-template-evaluator / create-code-evaluator), not on the task — it controls what unit of data that evaluator scores whenever it runs against a project task (not dataset/experiment tasks — those evaluate experiment runs directly). It defaults to span.

How trace and session aggregation works

For trace granularity, spans sharing the same context.trace_id are grouped together. Column values used by the evaluator template are comma-joined into a single string (each value truncated to 100K characters) before being passed to the judge model.

For session granularity, the same trace-level grouping happens first, then traces are ordered by start_time and grouped by attributes.session.id. Session-level values are capped at 100K characters total.

The {{conversation}} template variable

At session granularity, {{conversation}} is a special template variable that renders as a JSON array of {input, output} turns across all traces in the session, built from attributes.input.value / attributes.llm.input_messages (input side) and attributes.output.value / attributes.llm.output_messages (output side).

At span or trace granularity, {{conversation}} is treated as a regular template variable and resolved via column mappings like any other.

Note: For {{conversation}} to work, spans must carry attributes.session.id. See the arize-instrumentation skill for how to emit session.id from application code, including the force_flush() pattern required for Jupyter notebooks and short-lived scripts.

Multi-evaluator tasks

A task can contain evaluators at different granularities. At runtime the system uses the highest granularity (session > trace > span) for data fetching and automatically splits into one child run per evaluator. Per-evaluator query_filter in the task's evaluators JSON further narrows which spans are included (e.g., only tool-call spans within a session).

Basic CRUD

Full command reference for AI integrations, evaluators (template and code), and tasks — every flag, with examples — is in references/cli-reference.md. The workflows below include the commands you need inline.

Workflow A: Create an evaluator for a project