Codex Autoresearch Loop

Codex Autoresearch

Skill by ara.so — Daily 2026 Skills collection.

Codex Autoresearch is a Codex skill that runs an autonomous modify→verify→keep/revert loop on your codebase. You describe a measurable goal in one sentence; Codex confirms the plan, then iterates unattended — every improvement stacks in git, every failure reverts automatically — until interrupted or a cap is reached. Inspired by Karpathy's autoresearch concept, generalized beyond ML training to any software metric.

Installation

Option A — manual copy into your project:

git clone https://github.com/leo-lilinxiao/codex-autoresearch.git
cp -r codex-autoresearch your-project/.agents/skills/codex-autoresearch

Option B — Codex skill installer:

$skill-installer install https://github.com/leo-lilinxiao/codex-autoresearch

The skill lives at .agents/skills/codex-autoresearch/ inside your project. No config file is required before first use.

How to Activate

Open Codex in your project directory and prefix your goal with $codex-autoresearch:

$codex-autoresearch
I want to get rid of all `any` types in my TypeScript code

Codex will:

1. Scan the repo and infer scope, metric, verify command, and guard command.
2. Present a confirmation summary — reply go (or correct anything).
3. Run the loop unattended until you interrupt it or the goal is met.

You never write config. Codex infers everything.

Confirmation Flow

Before the loop starts Codex always shows what it found and asks you to confirm. Example exchange:

Codex: I found 47 `any` occurrences across src/**/*.ts.

       Confirmed:
       - Target: eliminate `any` types in src/**/*.ts
       - Metric: `any` count (current: 47), direction: lower
       - Verify: grep + tsc --noEmit as guard

       Need to confirm:
       - Run until all gone, or cap at N iterations?

       Reply "go" to start, or tell me what to change.

You:   Go, run overnight.

Codex: Starting — baseline: 47. Iterating until interrupted.

Up to five confirmation rounds are possible. After that, Codex proceeds.

The Loop (internals)

PHASE 0: Probe environment (CPU/GPU/RAM/toolchains), check for session resume
PHASE 1: Read context + lessons file from prior run (if any)

LOOP (forever or N times):
  1. Review current state, git history, results log, lessons
  2. Pick ONE hypothesis (apply perspectives, filter by environment)
     -- or N hypotheses if parallel mode is active
  3. Make ONE atomic change
  4. git commit (before verification)
  5. Run verify command  →  did the target metric improve?
     Run guard command   →  did anything else break?
  6. Improved → keep (extract lesson)
     Worse    → approved rollback strategy (git revert)
     Crashed  → fix or skip
  7. Log the result to results log
  8. Health check (disk, git, verify health)
  9. If 3+ discards → REFINE; 5+ → PIVOT; 2 PIVOTs → web search
 10. Repeat. Never stop. Never ask.

The loop runs unbounded unless you say Iterations: N during confirmation.

Dual-Gate Verification

Two commands serve distinct purposes:

Guard files are never modified by the loop.

Example verify + guard pair for a Python coverage run:

Verify: pytest --cov=src --cov-report=term 2>&1 | grep TOTAL | awk '{print $NF}'
Guard:  python -m mypy src --ignore-missing-imports

Example for TypeScript type cleanup:

Verify: grep -r "any" src --include="*.ts" | wc -l
Guard:  npx tsc --noEmit

Modes

Codex maps your sentence to one of seven modes automatically — you never pick a mode explicitly.

loop — iterate toward a measurable target (default)

$codex-autoresearch
Improve test coverage in src/ to at least 80%

$codex-autoresearch
Reduce bundle size — it's currently 2.3 MB, get it under 1 MB

plan — turn a vague goal into a validated loop config

$codex-autoresearch
I want to make our API faster but I don't know where to start

Codex will interview you (p95 latency vs throughput? which endpoint?) and produce a ready-to-run loop config.

fix — repair errors until count reaches zero

$codex-autoresearch
pytest is failing, 12 tests broken after the refactor — fix them all

debug — evidence-driven root-cause hunting

$codex-autoresearch
Our API returns 503 randomly under load, no idea why

Each iteration tests one falsifiable hypothesis. Codex presents evidence, not guesses.

security — read-only STRIDE + OWASP audit

$codex-autoresearch
Is this code secure?

ship — readiness verification and release gating

$codex-autoresearch
Ship it

exec — one-shot execution with no loop

$codex-autoresearch
Run the benchmark suite and summarize results

Inline Configuration (optional)

You can override defaults inline during the confirmation step — no file edits needed:

Example during confirmation:

You:   Go. Iterations: 30, Guard: npm test, Scope: src/api/

Cross-Run Learning

At the end of each iteration Codex writes a structured lesson to .agents/skills/codex-autoresearch/lessons.md:

Iteration 7 — KEPT
Hypothesis: replace explicit `any` with inferred generic in src/utils/mapper.ts
Change: added <T extends Record<string, unknown>> to mapKeys()
Result: any count 31 → 29
Lesson: Generic constraints on utility functions eliminate clusters of `any` downstream.

On session resume Codex reads this file first. Each new run benefits from prior runs.

To resume an interrupted run:

$codex-autoresearch
Resume

Codex re-reads the lessons file, checks git state, re-establishes the baseline, and continues.

Parallel Experiments

Request parallel mode during confirmation or at any time:

You:   Go, parallel 4

Codex runs four hypotheses concurrently, keeps the best result, discards the rest. Useful when hypothesis space is large.

Pivot Protocol

If the loop stalls, escalation happens automatically:

You are never asked for permission during escalation. The loop continues.

Real Code Examples

Example 1 — TypeScript any elimination (Python verify script)

If you want a custom verify script instead of a one-liner:

# scripts/count_any.py
import subprocess, sys

result = subprocess.run(
    ["grep", "-r", "--include=*.ts", r"\bany\b", "src/"],
    capture_output=True, text=True
)
count = len(result.stdout.strip().splitlines())
print(count)
sys.exit(0)  # always exit 0; the number is what matters

Tell Codex during confirmation:

Verify: python scripts/count_any.py
Guard:  npx tsc --noEmit

Example 2 — pytest coverage loop (Python)

# scripts/coverage_pct.py
import subprocess, re, sys

out = subprocess.check_output(
    ["pytest", "--cov=src", "--cov-report=term", "-q"],
    stderr=subprocess.STDOUT, text=True
)
match = re.search(r"TOTAL\s+\d+\s+\d+\s+(\d+)%", out)
if match:
    print(int(match.group(1)))
    sys.exit(0)
print(0)
sys.exit(0)

$codex-autoresearch