Login
ruvnet/ruflo1 file

Cost Benchmark

Run the corpus benchmark — booster locally, optional Gemini/Sonnet/Opus baselines — and persist a verifiable measured-vs-claimed table

Use this skillView repositoryView on skills.sh
Specification
Skill ID
ruvnet/ruflo/cost-benchmark
Publisher
ruvnet
Repository
ruflo
Installs
0
Files
1
Synced
Sep 23, 2026
How to use it

Open any RiverX project, open the Skills panel in the chat, and search for this identifier. The files are fetched from the source repository at install time.

ruvnet/ruflo/cost-benchmarkInstalls these files
  • SKILL.md

What this skill tells the agent

Cost Benchmark

Runs scripts/bench.mjs against the structural+adversarial corpus and writes per-case + summary results to docs/benchmarks/runs/. This is the verification gate that backs every measurable claim in cost-booster-edit / cost-booster-route.

When to use

  • Before publishing a release — verify booster win rate didn't regress.
  • After expanding bench/booster-corpus.json — confirm new cases route correctly.
  • When auditing a "claimed upstream" tag — flip it to "verified" once the bench supports it.
  • On a cost question ("is Sonnet 4.6 cheaper than Opus 4.7 for these tasks?") — re-run with BENCH_ANTHROPIC=1.

Steps

  1. Run the bench from `v3/` (where agent-booster resolves):
   ( cd v3 && node ../plugins/ruflo-cost-tracker/scripts/bench.mjs )                  # booster only — free, ~85 ms
   ( cd v3 && BENCH_LLM_BASELINE=1 node ../plugins/ruflo-cost-tracker/scripts/bench.mjs ) # + Gemini 2.0 Flash (cheap)
   ( cd v3 && BENCH_LLM_BASELINE=1 BENCH_ANTHROPIC=1 \
        node ../plugins/ruflo-cost-tracker/scripts/bench.mjs )                          # + Sonnet 4.6 + Opus 4.7
  1. Inspect the markdown summary printed to stdout. The gate metric is winRate (Tier 1 cases). Adversarial cases are tracked separately as escalationRate.
  1. Persisted output lands at:
  • docs/benchmarks/runs/latest.json — pointer to the most recent run
  • docs/benchmarks/runs/<ISO-timestamp>.json — historical record
  1. Read it back in subsequent skills (e.g. cost-report step 2 reads latest.json for live tier-spend numbers).

Smoke gates

  • winRate ≥ 0.80 on Tier 1 cases (smoke step 23). Lower the threshold by editing scripts/smoke.sh.
  • escalationRate is reported but ungated — adversarial cases are diagnostic.

Env overrides

Env varDefaultPurpose
BENCH_LLM_BASELINEunset=1 runs the OpenAI-compat baseline
BENCH_LLM_MODELmodels/gemini-2.0-flashOverride the OpenAI-compat model
BENCH_LLM_BASE_URLGemini OpenAI shimOverride endpoint
BENCH_ANTHROPICunset=1 runs Anthropic baseline (Sonnet 4.6 + Opus 4.7)
BENCH_ANTHROPIC_MODELSclaude-sonnet-4-6,claude-opus-4-7Comma-separated Claude IDs
BENCH_OUTtimestamped fileOverride output path
BENCH_QUIET=1unsetSuppress markdown summary

API keys auto-pulled from gcloud secrets (GOOGLE_AI_API_KEY, ANTHROPIC_API_KEY); override with BENCH_LLM_API_KEY / BENCH_ANTHROPIC_API_KEY.

Cross-references

ADR-0002 §"Decision 1" / §"Riskiest assumption" · cost-booster-edit/SKILL.md (verification table consumes this skill's output) · cost-report/SKILL.md step 2 (reads runs/latest.json).

More from ruvnet

ruvnet/ruflo

Adr Create

Create a new Architecture Decision Record with sequential numbering and AgentDB registration

ruvnet/ruflo

Adr Index

Build or rebuild the ADR index + dependency graph by running scripts/import.mjs (handles v3-style and plugin-style ADR formats; one Bash call vs hundreds of MCP round-trips)

ruvnet/ruflo

Adr Review

Review code changes against accepted ADRs for compliance violations

Use this skill in a project

Open any RiverX project, search the Skills panel for ruvnet/ruflo/cost-benchmark, and install it in one click.

Use this skill