Anycap Human Interaction

AnyCap Human Interaction

Read this entire file before starting. Skipping sections leads to incorrect workflows -- each media type has different capabilities and constraints.

Workflow guide for collecting structured visual feedback from humans using AnyCap's annotation tool. This skill teaches you when and how to involve humans in your workflow through visual annotation and screen recording.

For CLI command reference, read the anycap-cli skill. For media generation workflows, read the anycap-media-production skill.

Prerequisites

AnyCap CLI must be installed and authenticated. Read the anycap-cli skill if setup is needed.

Two Core Scenarios

AnyCap annotation excels at two distinct review workflows. Choose the one that fits your situation:

Both scenarios support all annotation tools (Rect, Arrow, Point, Freehand) and text labels. The difference is in what you get back and how the review is conducted.

Command Quick Reference

# Blocking -- opens browser, waits for Done click, outputs result
anycap annotate <target> [-o output.png]

# Non-blocking -- starts background server, returns session info
anycap annotate <target> --no-wait [-o output.png]

# Poll for result after human confirms done
anycap annotate poll --session <session_id>

# Stop background server
anycap annotate stop --session <session_id>

# List all active sessions (useful for recovery after context loss)
anycap annotate list

<target> is auto-detected by content: image file, URL (http:// or https://), video file, or audio file.

Key Flags

Tip: Use --port with a consistent value (e.g., --port 8888) across sessions. The browser stores each user's display name in localStorage, which is scoped by origin (host + port). A fixed port means returning collaborators are recognized automatically without re-entering their name.

Browser Auto-Open

Both blocking and non-blocking modes automatically attempt to open the annotation URL in the default browser. In headless environments (SSH, container), the CLI prints the URL to stderr instead. No error is raised.

Presenting to the Human -- Guidelines

When presenting an annotation session to the human, adapt your message based on context. Key points to communicate:

• Browser auto-open: On desktop, the page opens automatically -- acknowledge this ("the review page should already be open"). In headless/SSH, share the URL and tell them how to access it.
• Always mention Done: Tell the human to click Done when finished. This is how feedback gets saved.
• Recording for URL mode: Emphasize recording (Rec button) because URL mode cannot export an annotated screenshot. The recording is the primary artifact.
• Multi-user: If multiple reviewers will participate, mention real-time collaboration and that each person can save independently. Exception: URL/iframe mode is single-user (recording is the primary feedback artifact, and multiple users' cursors would make it confusing).
• Headless access: When using --bind 0.0.0.0, share the URL with the actual host IP. If behind SSH, suggest port forwarding.
• What to annotate: Briefly describe what tools are available (Rect, Arrow, Point, Freehand) and that each annotation can have a text label.

Do NOT use canned messages. Compose naturally based on the situation (what you just generated/modified, whether it is desktop or headless, single or multi-reviewer).

Headless / Remote Access

When running in a headless environment (SSH, container, cloud VM), the human cannot access 127.0.0.1 directly. Use --bind and --port to make the annotation server accessible:

# Bind to all interfaces on a fixed port
anycap annotate screenshot.png --no-wait --bind 0.0.0.0 --port 8888

The human can then access the annotation UI via:

• Direct access: http://<server-ip>:8888 (if the port is exposed)
• SSH port forward: ssh -L 8888:localhost:8888 user@host, then open http://localhost:8888
• Container port mapping: docker run -p 8888:8888 ..., then open http://localhost:8888
• Reverse proxy: expose through nginx, Caddy, or any reverse proxy with a path prefix

Always use --port with a fixed number in headless environments so the URL is predictable and forwardable.

Reverse Proxy Compatibility

The annotation UI works behind reverse proxies with arbitrary path prefixes. All asset, API, and WebSocket URLs are resolved relative to the page URL, so setups like the following work out of the box:

https://yourserver.com/tools/annotate/  ->  http://localhost:8888/

If your proxy passes query parameters (e.g., ?token=... for access control), they are preserved on all internal requests automatically. No additional configuration is needed on the annotation server side.

When to Use Annotation

Use annotation when:

• You generated an image/video and need the human to point at what to change
• You built or modified a web page and need the human to review it visually
• You need spatially-grounded feedback ("move this here", "this area is wrong")
• Text-only feedback would be ambiguous about location or visual details
• You want the human to record a narrated walkthrough of their feedback

Do NOT use annotation when:

• You only need a yes/no approval (just ask in chat)
• The feedback is purely textual (e.g., "change the title text to X")

Interaction Pattern

All annotation workflows follow this pattern:

sequenceDiagram
    participant Agent
    participant CLI as AnyCap CLI
    participant Human

    Agent->>CLI: anycap annotate <target> --no-wait
    CLI-->>Agent: {session, url, session_file, poll_command, stop_command}
    Agent->>Human: Present URL with guidance
    Human->>Human: Annotate, record, click Done
    Human->>Agent: Confirms done
    Agent->>CLI: poll_command
    CLI-->>Agent: Annotations + recording
    Agent->>Agent: Process feedback
    Agent->>CLI: stop_command

The "Done" Button

The human clicks Done in the annotation toolbar to save their feedback. The behavior differs by mode:

• Blocking mode (no --no-wait): Clicking Done ends the session. The CLI command returns immediately with the result. In collaborative modes (image, video, audio), other connected users see a "Feedback Submitted" overlay.
• Non-blocking mode (--no-wait): Clicking Done saves the feedback without ending the session. In collaborative modes, other users see a toast notification and can keep annotating. Each subsequent Done click overwrites the saved result. The agent polls to retrieve the latest saved state.

For agents: Always tell the human to click Done when they are finished. In non-blocking mode with multiple reviewers (image/video/audio only), each reviewer can save independently -- the poll result reflects the most recent save.

Session Recovery

Session state is persisted at .anycap/annotate/<session_id>.json in the working directory (returned as session_file in the start response). If you lose the session ID or commands after a context reset:

# List all sessions with their status and recovery commands
anycap annotate list