Langsmith Evaluator

<oneliner> Three core components: (1) Creating Evaluators - LLM-as-Judge, custom code; (2) Defining Run Functions - capture agent outputs/trajectories for evaluation; (3) Running Evaluations - locally with evaluate() or auto-run via uploaded evaluators. Python and TypeScript examples included. </oneliner>

<setup> Environment Variables

LANGSMITH_API_KEY=your-api-key                        # Alternative to `langsmith auth login`
LANGSMITH_ENDPOINT=https://api.smith.langchain.com   # SDK/CLI environment
LANGSMITH_PROJECT=your-project-name                   # Default for trace/run queries only
LANGSMITH_WORKSPACE_ID=your-workspace-id              # Optional: for org-scoped keys
OPENAI_API_KEY=your-openai-key                        # For locally executed OpenAI judges

Authenticate with a saved CLI profile (preferred):

langsmith auth login
langsmith auth info

Alternatively, set LANGSMITH_API_KEY. The hidden --api-key flag remains available for compatibility, but do not place keys directly in commands or logs. Use --profile or LANGSMITH_PROFILE when selecting among saved profiles.

IMPORTANT: LANGSMITH_PROJECT defaults trace and run queries; it does not choose an evaluator target. Evaluator creation and upload require an explicit --dataset or --project.

CLI and SDK authentication are separate: A CLI OAuth profile authenticates langsmith ... commands only. Python and TypeScript SDKs do not automatically consume it. Before using an SDK during evaluator CRUD, confirm that LANGSMITH_ENDPOINT, LANGSMITH_API_KEY, and, when required, LANGSMITH_WORKSPACE_ID target the same environment and workspace as langsmith auth info. Stop if the CLI and SDK target different environments.

Use a read-only SDK preflight before any SDK-assisted CRUD:

from langsmith import Client

client = Client()
client.read_dataset(dataset_name="My Dataset")

Python Dependencies

pip install langsmith langchain-openai python-dotenv

CLI Tool

curl -fsSL https://cli.langsmith.com/install.sh | sh
langsmith self-update

JavaScript Dependencies

npm install langsmith openai

</setup>

<cli_feedback> After an actual CLI failure, undocumented workaround, confusing output, or missing capability, submit one concise product-feedback note per distinct issue in the task. Do not report routine successes or low evaluation scores/application failures unrelated to the CLI.

CLI requirement: langsmith feedback requires LangSmith CLI v0.2.58 or later. Check the installed version with langsmith --version.

Check langsmith feedback --help for feedback <note> and --category; if unavailable, skip feedback without raw HTTP or unreleased builds. Use the existing authenticated profile, endpoint, and workspace. Feedback goes to LangSmith Cloud, including through the BYOC relay; skip standalone self-hosted. Respect user/organization restrictions and ask first if permission to send feedback is unclear.

Summarize expected versus observed CLI behavior and any workaround in your own words. Never send secrets, customer data, evaluator prompts, dataset contents, trace payloads, full stack traces, copied command output, raw arguments, environment-variable values, local paths, or resource identifiers. The CLI adds version/OS/architecture, but does not redact your note; skip it if it cannot be safely redacted.

Choose bug, feature-request, usability, documentation, or other. This is CLI product feedback, not an evaluator score or comment. Example shape only—do not submit unless actually encountered:

langsmith feedback --category feature-request --format json "Please support selecting a single evaluator by ID when display names collide."

Do not retry a failed or rate-limited feedback submission, switch credentials/endpoints to bypass a failure, or block the original task on feedback. </cli_feedback>

<crucial_requirement>

Golden Rule: Inspect Before You Implement

CRITICAL: Before writing ANY evaluator or extraction logic, you MUST:

1. Run your agent on sample inputs and capture the actual output
2. Inspect the output - print it, query LangSmith traces, understand the exact structure
3. Only then write code that processes that output

Output structures vary significantly by framework, agent type, and configuration. Never assume the shape - always verify first. Query LangSmith traces to when outputs don't contain needed data to understand how to extract from execution. </crucial_requirement>

<evaluator_format>

Offline vs Online Evaluators

Offline Evaluators (attached to datasets):

• Function signature: (run, example) - receives both run outputs and dataset example
• Use case: Comparing agent outputs to expected values in a dataset
• Upload with: --dataset "Dataset Name"

Online Evaluators (attached to projects):

• Function signature: (run) - receives only run outputs, NO example parameter
• Use case: Real-time quality checks on production runs (no reference data)
• Upload with: --project "Project Name"

CRITICAL - Return Format:

• Each evaluator returns ONE metric only. For multiple metrics, create multiple evaluator functions.
• Do NOT return {"metric_name": value} or lists of metrics - this will error.

CRITICAL - Local vs Uploaded Differences:

</evaluator_format>

<evaluator_types>

• LLM as Judge - Uses an LLM to grade outputs. Best for subjective quality (accuracy, helpfulness, relevance).
• Custom Code - Deterministic logic. Best for objective checks (exact match, trajectory validation, format compliance).

</evaluator_types>

<llm_judge>

LLM as Judge Evaluators

Use langsmith evaluator create-llm to create a server-managed LLM-as-judge run rule. It requires --model-config plus exactly one target (--dataset or --project). Supply either --prompt and --schema JSON files or a Prompt Hub reference with --hub-ref.

Native CLI CRUD for LLM-as-Judge Run Rules

The native lifecycle below manages LLM-as-judge evaluators attached to a dataset or project.

Create

langsmith evaluator create-llm \
  --name "Accuracy Judge" \
  --dataset "My Dataset" \
  --prompt prompt.json \
  --schema schema.json \
  --model-config model.json

Use --project instead of --dataset for an online evaluator. Use --hub-ref owner/prompt:latest instead of --prompt and --schema when the judge prompt is stored in Prompt Hub.

Model Configuration

--model-config requires a server-supported serialized model configuration. Never invent model.json from a local LangChain model constructor. Obtain it from a known-working LangSmith evaluator, the LangSmith evaluator UI, or another documented source for the target environment. Confirm that the target server allows the serialized model class before creating or replacing the evaluator.