Alan Test Feature

Task: Test Feature & Generate Demo Report

Analyze the current branch's changes, scope the coverage level with the user, build an explicit testing strategy, start the app's dev server, test the feature in a real browser using agent-browser, capture screenshots and video (always — no exceptions), upload everything to S3, and generate a structured test report (always — never skip).

Phase 0: Environment Setup

1. Get artifact context IDs

Use the active task context when it is present in the prompt. Otherwise:

• The task ID is available from the ALAN_TASK_ID environment variable.
• The conversation ID is available from the ALAN_SESSION_ID environment variable.
• Resolve teamId with Alan MCP context/tools before creating the report.

When calling create_test_report, always pass taskId, conversationId, and teamId explicitly. Do not substitute one ID for another.

2. Verify agent-browser is available

agent-browser is a CLI tool by Vercel Labs (npm: agent-browser) — it is NOT a skill or MCP tool. It provides headless browser automation via bash commands (open, click, fill, screenshot, record). In sandboxes it is pre-installed by runtime-bootstrap.ts.

command -v agent-browser >/dev/null 2>&1 && echo "agent-browser: OK" || echo "agent-browser: MISSING"

If missing, install it:

npm install -g agent-browser && agent-browser install

If installation fails (e.g. no network, no npm), STOP and tell the user:

"agent-browser is not available and could not be installed. Install it manually: npm install -g agent-browser && agent-browser install"

Do NOT proceed to Phase 2 without a working agent-browser — all browser testing depends on it.

3. Create workspace

All captures (screenshots, videos) MUST be written to /tmp/alan-captures/.

mkdir -p /tmp/alan-captures

CRITICAL — file path rules: - ONLY write capture files to /tmp/alan-captures/. Never anywhere else. - NEVER write to .claude/, project directories, reports/, or any path inside the repo. - .claude/ is a sensitive system directory — writing to it will be blocked and will abort the test run. - If you are tempted to create a reports/ or screenshots/ folder anywhere other than /tmp/, stop and use /tmp/alan-captures/ instead.

Phase 0.5: Coverage Scoping & Context Gathering (MANDATORY — do before anything else)

Never skip this phase. You must understand what the user wants before writing a single test step.

1. Ask about coverage level

Use AskUserQuestion to ask the user:

What level of test coverage do you want for this feature?

- light       — Happy path only. Quick smoke test to verify the main flow works.
- standard    — Happy path + key edge cases + basic error states. (Default)
- comprehensive — Full coverage: happy path, all edge cases, error states, non-happy paths, boundary values, accessibility, responsiveness.

Also: are there any specific flows, known bugs, or risky areas you want me to focus on?

Wait for the response before continuing.

2. Ask targeted follow-up questions

Based on the feature (which you may not know yet — do a quick git diff --stat first to get a hint), use AskUserQuestion to ask 1–3 targeted follow-up questions. Examples:

• "Is there an authenticated state I should test? If yes, what test credentials should I use?"
• "Are there any known edge cases or previous bugs related to this feature?"
• "What's the definition of 'working correctly' for this feature — what should I see?"
• "Are there any specific non-happy paths you're concerned about (e.g. invalid input, network errors, empty states)?"
• "Any pages or user roles I should specifically include or exclude?"

Do not skip questions if context is unclear. Ask. A well-scoped test is 10× more valuable than a blind one.

3. Confirm the plan

After gathering answers, summarize back to the user (no tool call needed — just a short message):

• Coverage level chosen
• Specific areas / flows to focus on
• Any known risks or edge cases to target
• Rough count of test scenarios expected

Only proceed to Phase 1 after this confirmation.

Phase 1: Discover What to Test

1. Analyze the branch

Run these commands to understand what changed:

git log main..HEAD --oneline 2>/dev/null || git log HEAD~5..HEAD --oneline
git diff main...HEAD --stat 2>/dev/null || git diff HEAD~1 --stat

Read the actual diff to understand the feature or bug fix. Identify:

• What part of the app is affected (which pages, components, API routes)
• What the expected behavior change is
• What URL path to navigate to for testing

2. Discover and start the dev server

Look at the project to figure out how to run it:

1. Read package.json — check scripts.dev, scripts.start, scripts.serve
2. Check for docker-compose.yml / docker-compose.yaml / compose.yml
3. Check for Makefile (look for dev or serve targets)
4. Check for Procfile, .env, Pipfile, requirements.txt, Gemfile
5. Check for framework-specific files: next.config.*, vite.config.*, nuxt.config.*, angular.json, manage.py, config/routes.rb

Start the dev server in the background. Common patterns:

# Node.js
npm run dev &
# or: pnpm dev &, yarn dev &, npx next dev &, npx vite &

# Python
python manage.py runserver &
# or: flask run &, uvicorn main:app &

# Ruby
bundle exec rails server &

# Docker
docker compose up -d

3. Wait for the server

Poll until the server is responding:

# Replace PORT with the discovered port
for i in $(seq 1 30); do
  curl -sf http://localhost:PORT >/dev/null 2>&1 && break
  sleep 2
done

Check common ports if unclear: 3000, 5173, 8080, 4200, 8000, 4000, 3001, 8888.

4. Determine the app URL

• Parse dev server stdout/stderr for "Local:" or "ready on" messages with URLs
• Check .env or .env.local for PORT or VITE_PORT or similar
• Test common ports with curl -sf http://localhost:PORT >/dev/null
• If you cannot determine it, ask the user via AskUserQuestion

5. Summarize before proceeding

Tell the user:

• What you found in the diff (feature/fix summary)
• What URL you will test
• What test scenarios you plan to cover

Phase 1.5: Build Testing Strategy & Create Todos (MANDATORY)

Before opening the browser, you must have an explicit plan. Do not improvise test steps on the fly.

1. Draft the testing strategy

Based on:

• The coverage level the user chose in Phase 0.5
• The diff analysis from Phase 1
• The context gathered from the user

Write out the full test plan as a structured list. For each scenario, note:

• Scenario name — short label (e.g. "Happy path: create item")
• Type — happy path / non-happy path / edge case / error state / visual / navigation
• Steps — what to do
• Expected outcome — what "pass" looks like
• Screenshot needed — yes/no

Coverage requirements by level:

• light: happy path only (1–3 scenarios)
• standard: happy path + 2–4 non-happy paths + 1–2 error states
• comprehensive: happy path + all non-happy paths + all error states + boundary values + empty states + responsiveness + navigation

Non-happy path examples to consider:

• Invalid / missing required inputs
• Submitting with no data / empty state
• Duplicate entries (if applicable)
• Permission denied / unauthorized access
• Network error / API failure simulation (if testable)
• Rapid repeated actions (double-click, spam submit)
• Long input strings / special characters
• Back button / browser navigation mid-flow

2. Create todos for each scenario

Use TodoWrite to create a todo entry for each test scenario so progress is tracked. Each todo should be the scenario name.

3. Present the strategy

Output the full testing strategy as a numbered list before proceeding. The user should be able to see exactly what will be tested before Phase 2 starts.