Music To Video

First, keep this skill fresh — confirm with the user before running: npx hyperframes skills update music-to-video. A fast no-op when everything is current; otherwise it refreshes this skill plus the core domain skills it depends on before you rely on them.

music-to-video — one music-grounded, beat-synced video workflow

Use this skill to turn a music track into a beat-synced HyperFrames video. You analyze the track once, lay out the frames, fill in a per-frame plan, and build each frame as a composition. The input is a music track plus optional user images or videos — there is no narration and no website capture. Typography and templates are the floor (a complete video needs zero assets); any media the user supplies is cut in on the same beat grid.

You are the orchestrator. Work in videos/<project>/. Run the steps in order and pass each Gate before moving on. Two steps need the user: Step 3 (plan approval) and Step 6 (render approval) — both are checkpoint gates per ../hyperframes/references/brief-contract.md (read it before Step 0): in autonomous mode, post the summary as a heads-up and proceed instead of waiting. Do every step yourself except Step 4, where you dispatch one sub-agent per frame. Keep design and motion rules out of this file — they live in references/ and the frame-worker sub-agent.

SKILL_DIR = this skill directory. PROJECT_DIR = videos/<project-name>/.

Workflow: Step 0 setup → hyperframes.json + assets/bgm.mp3; Step 1 analyze → audiomap.json; Step 2 skeleton → STORYBOARD.md (frames, groups TBD); Step 3 plan → complete STORYBOARD.md + frame.md; Step 4 build → compositions/frames/NN-*.html; Step 5 assemble → index.html; Step 6 render → renders/video.mp4.

Two ideas that shape everything

• One analyzer, and you trust it. analyze-beatgrid.py is the only beat analyzer — never re-measure beats with another tool or by ear. Its energy / density / rolls / onsets / silences are always reliable. Its bpm and beats_sec are reliable only when the music is genuinely rhythmic; on calm music the grid is a metronome the tracker imposed, so pace by phrases and energy instead and never hard-cut to it. Deciding which case you're in is each frame's pacing (Step 2).
• One frame = one file; groups live inside. Step 2 cuts the track into frames, and each frame becomes one composition file compositions/frames/NN-<frame_id>.html, built by one frame-worker. A frame can subdivide into groups (each a template or a motion-primitives combo). Extra density goes _inside_ a group, so frame count tracks distinct treatments, not beats — a fast track does not blow up the number of sub-agents.

Step 0: Setup, BGM, and inputs

Goal: Establish the music source, create the HyperFrames project, and note any user-supplied media.

The brief starts at the intent layer. Opening rule, in order: (1) BRIEF.md exists → read it and ask nothing it answers — its flow/storyboard derive the mode (brief contract § 1). (2) No BRIEF.md but the project exists → resume from what's on disk; never re-interrogate. (3) A fresh creation request that arrived here directly → read /hyperframes and run its intent layer (references/intent-interview.md): it confirms this route's must-haves (the music source, destination → aspect — ../hyperframes/references/routes/music-to-video.md) and announces what stays deferred — brand and genre are chosen at Step 3 by design. Write BRIEF.md immediately after init (never before — init refuses a non-empty directory) and record the preference-backed answers (brief-format.md). Edit requests skip all of this.

The music is the spine — establish one track before anything else. This skill is tuned for fast, high-energy BGM: a strong beat grid drives the cuts (calm tracks work, but pace by phrase rather than beat). If the user supplied audio — a music file, or a video to pull audio from — use it. Otherwise choose the mood from the request and generate a track through /media-use (references/bgm.md). Before the first authenticated provider action, run npx hyperframes auth status and relay its output verbatim. If signed out, apply one branch:

• Collaborative: wait for sign-in or an explicit choice to continue offline with the local provider.
• Autonomous: state the status and continue through the available local provider.

If no offline provider can satisfy the required music capability, surface the blocker. Never write keys into a per-repo .env. Auth ownership and offline fallbacks live in /media-use references/setup-providers.md § Providers. The resulting track lands at assets/bgm.mp3. Stage supplied images or videos so frames can use them on the beat grid; otherwise typography carries the video.

Lyric videos: for lyrics synced to the vocals, get word/line timing by transcribing the track via /media-use, or ask the user for the lyrics text and place lines on the beat grid.

Initialize only if hyperframes.json is missing. Name <project> from the brief in kebab-case, such as midnight-drive-loop — never a timestamp. init checks the installed skills against the latest on GitHub and updates the global set if any are out of date.

npx hyperframes init "videos/<project>" --non-interactive --example=blank --skill=music-to-video
mkdir -p "$PROJECT_DIR/assets" "$PROJECT_DIR/renders"
cp "<user-music>" "$PROJECT_DIR/assets/bgm.mp3"   # extract from a video first if needed
# only if the user gave you images/videos:
node <SKILL_DIR>/scripts/stage-assets.mjs --from <dir> --hyperframes "$PROJECT_DIR" --into public

The brand (font + palette) is chosen at Step 3, not here. Don't pick a genre or a track type up front — assets are just an optional ingredient, and the genre emerges from the per-frame choices.

Gate: hyperframes.json + assets/bgm.mp3 exist; aspect / length / fps and (if any) the asset inventory are noted.

Step 1: Analyze the music

Goal: Produce the one canonical timing analysis the whole video is built on.

analyze-beatgrid.py is the only beat analyzer — never re-measure beats with another tool or by ear. It reads the track once and writes audiomap.json: energy phases (level / density / feel), onsets + onset_rate, rolls, silences, hard_stops, key_moments, phrases, tempo / grid, and audio.duration_sec. It's deterministic — the same file always gives the same map. Most fields are reliable on any music; bpm and beats_sec are reliable only when the music is genuinely rhythmic, and judging that is the call you make at Step 2.

Prerequisites: Python 3 with librosa, numpy, and soundfile available. If import fails, install them into the active Python environment before running the analyzer:

python3 -m pip install librosa numpy soundfile

python3 <SKILL_DIR>/scripts/analyze-beatgrid.py "$PROJECT_DIR/assets/bgm.mp3" \
  -o "$PROJECT_DIR/audiomap.json" --print

Gate: audiomap.json exists; audio.duration_sec is known.

Step 2: Frame skeleton (structure only)

Goal: Read the music and lay out the frames — the skeleton of STORYBOARD.md.