Phoenix Evals
Build and run evaluators for AI/LLM applications using Phoenix.
Specification
- Skill ID
- arize-ai/phoenix/phoenix-evals
- Publisher
- arize-ai
- Repository
- phoenix
- Installs
- 0
- Files
- 38
- License
- Apache-2.0
- Requires
- Requires Phoenix server. Python skills need phoenix and openai packages; TypeScript skills need @arizeai/phoenix-client.
- Synced
- Sep 23, 2026
How to use it
Open any RiverX project, open the Skills panel in the chat, and search for this identifier. The files are fetched from the source repository at install time.
arize-ai/phoenix/phoenix-evalsInstalls these files- SKILL.md
- references/axial-coding.md
- references/common-mistakes-python.md
- references/error-analysis-multi-turn.md
- references/error-analysis.md
- references/evaluate-dataframe-python.md
- references/evaluators-code-python.md
- references/evaluators-code-typescript.md
- references/evaluators-custom-templates.md
- references/evaluators-llm-python.md
- references/evaluators-llm-typescript.md
- references/evaluators-overview.md
- references/evaluators-pre-built.md
- references/evaluators-rag.md
- references/experiments-datasets-python.md
- references/experiments-datasets-typescript.md
- references/experiments-overview.md
- references/experiments-running-python.md
- references/experiments-running-typescript.md
- references/experiments-synthetic-python.md
- references/experiments-synthetic-typescript.md
- references/filter-expressions.md
- references/fundamentals-anti-patterns.md
- references/fundamentals-model-selection.md
- references/fundamentals.md
- references/integrations-pytest.md
- references/integrations-vitest-jest.md
- references/observe-sampling-python.md
- references/observe-sampling-typescript.md
- references/observe-tracing-setup.md
- references/production-continuous.md
- references/production-guardrails.md
- references/production-overview.md
- references/setup-python.md
- references/setup-typescript.md
- references/validation-evaluators-python.md
- references/validation-evaluators-typescript.md
- references/validation.md
What this skill tells the agent
Phoenix Evals
Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.
Quick Reference
| Task | Files |
|---|---|
| Setup | setup-python, setup-typescript |
| Decide what to evaluate | evaluators-overview |
| Choose a judge model | fundamentals-model-selection |
| Use pre-built evaluators | evaluators-pre-built |
| Build code evaluator | evaluators-code-python, evaluators-code-typescript |
| Build LLM evaluator | evaluators-llm-python, evaluators-llm-typescript, evaluators-custom-templates |
| Batch evaluate DataFrame | evaluate-dataframe-python |
| Run experiment | experiments-running-python, experiments-running-typescript |
| Run evals in a test runner (CI gate) | integrations-pytest, integrations-vitest-jest |
| Create dataset | experiments-datasets-python, experiments-datasets-typescript |
| Generate synthetic data | experiments-synthetic-python, experiments-synthetic-typescript |
| Validate evaluator accuracy | validation, validation-evaluators-python, validation-evaluators-typescript |
| Export spans | observe-tracing-setup |
Write a span filter (SpanQuery().where) | filter-expressions |
| Sample traces for review | observe-sampling-python, observe-sampling-typescript |
| Analyze errors | error-analysis, error-analysis-multi-turn, axial-coding |
| RAG evals | evaluators-rag |
| Avoid common mistakes | common-mistakes-python, fundamentals-anti-patterns |
| Production | production-overview, production-guardrails, production-continuous |
Workflows
Starting Fresh: observe-tracing-setup → error-analysis → axial-coding → evaluators-overview
Building Evaluator: fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}
RAG Systems: evaluators-rag → evaluators-code- (retrieval) → evaluators-llm- (faithfulness)
Gating CI: evaluators-{code|llm}-{python|typescript} → integrations-{pytest|vitest-jest} → production-continuous
Production: production-overview → production-guardrails → production-continuous
Reference Categories
| Prefix | Description |
|---|---|
fundamentals-* | Types, scores, anti-patterns |
observe-* | Tracing, sampling |
error-analysis-* | Finding failures |
axial-coding-* | Categorizing failures |
evaluators-* | Code, LLM, RAG evaluators |
experiments-* | Datasets, running experiments |
integrations-* | Run evals from test runners (pytest, Vitest, Jest) as a CI gate |
validation-* | Validating evaluator accuracy against human labels |
production-* | CI/CD, monitoring |
Key Principles
| Principle | Action |
|---|---|
| Error analysis first | Can't automate what you haven't observed |
| Custom > generic | Build from your failures |
| Code first | Deterministic before LLM |
| Validate judges | >80% TPR/TNR |
| Binary > Likert | Pass/fail, not 1-5 |
| Invariants gate, signals trend | assert/expect hard invariants (CI red); log LLM-judge quality signals and gate the aggregate (acceptance criteria), not every case |
More from arize-ai
Use this skill in a project
Open any RiverX project, search the Skills panel for arize-ai/phoenix/phoenix-evals, and install it in one click.