4k Vfx

4k-vfx

Maps a video-to-video VFX move onto the Pika MCP's Seedance 4K reference-to-video. The user hands you a clip + the change they want; you read EVERY frame (locally extracted with ffmpeg and tiled into contact sheets) and understand the audio (speech + music + SFX + ambience), author a Seedance-faithful prompt that locks the original (face, gestures, camera move) and time-codes the change to the right beat, then re-render the same shot in 4K with the VFX baked in. One run produces one 4K clip for one requested change.

How it works — why reading EVERY frame (and the audio) is the mechanism (do not skip the reading pass)

Seedance treats the reference video as a motion/style anchor, NOT a pixel-locked base — it does not copy the input frame-for-frame, it re-generates the shot guided by your prompt. So fidelity comes from the PROMPT: the more exhaustively the prompt describes the original (subject identity, exact gestures, camera move, framing, lighting, palette, wardrobe) and the audio (dialogue + lip-sync, music/SFX beats), the more the 4K output reads as the same shot with the change layered on. Reading EVERY frame — not a sample — is how you build that exhaustive description: you locally extract all frames with ffmpeg, tile them 25-to-a-contact-sheet (5×5), and analyze every sheet so the prompt is built from the entire timeline rather than a handful of stills. Skip the read (or read only a sample) and the output drifts — a different face, a different camera move, a different room, a missed motion beat. The reading pass — all frames + the audio — IS the skill.

Prerequisites

pika MCP available, and local `ffmpeg` in the run environment (used to extract every frame and build the contact sheets). Two inputs from the user: (1) a video clip (a local file or a public URL) and (2) the VFX change they want made to it. Tools used: upload_asset, probe_media, analyze_media, transcribe_audio, estimate_cost, generate_reference_video, task_status (frame extraction + tiling is done locally with ffmpeg, not an MCP tool).

Stage 0 — Gather inputs (settle this first)

You need exactly two things before generating:

1. The video — the original clip to transform. If they gave a local file, you will upload it (Step 1); if they gave a public URL, keep it.
2. The change / VFX — what should be different in the output (e.g. "turn the plaza into an open desert on a finger snap", "make it snow", "set the room on fire behind me", "morph my jacket into glowing armor").

If either is missing, ask for it before doing anything else — do not invent a change, and do not proceed on a clip you cannot reach. Once both are in hand, say in one line what you're about to do ("Reading every frame of your clip to build the desert-on-snap 4K prompt…") and run Steps 1–5 without further check-ins — but stop at the Step 6 agreement gate before generating: the 4K render is the expensive, irreversible step, and it only fires after the user approves the prompt and cost.

Step 1 — Upload the clip (upload_asset)

Real video must be uploaded — inline base64 only works for tiny assets (<~3MB), so any actual clip goes through the presigned-upload path.

1. Call upload_asset with filename, mime_type (video/mp4 for mp4), and size_bytes (the file's byte size).
2. PUT the raw bytes to the returned presigned_url.
3. Keep the returned public_url → state.clip_url.

If the user already gave a public URL, skip this step and set state.clip_url to it.

Step 2 — Probe (probe_media)

Call probe_media on state.clip_url to read duration, fps, dimensions, and aspect ratio. Use these to: (a) sanity-check the all-frame extraction in Step 3 (duration × fps ≈ total frame count ≈ 25 × number of sheets), and (b) choose the output aspect_ratio and duration here, matched to the source — the Step-6 gate quotes this choice to the user and Step 7 fires with it.

Step 3 — Extract EVERY frame locally and tile into contact sheets (ffmpeg)

Read the whole video, not a sample. Use local ffmpeg to extract every frame at the source rate (no drops, no exceptions), then tile them into legible contact sheets at 25 frames per sheet (5×5 grid) so each analyze_media call covers 25 consecutive frames at once.

Work in a relative working directory (do not use an absolute system temp path):

mkdir -p frames sheets
ffmpeg -i INPUT -vsync 0 frames/f_%05d.png            # every frame, no drops

Then build the 25-per-sheet contact sheets (5×5), scaling each frame down so the sheet is legible and stamping the frame index onto each tile:

ffmpeg -i INPUT -vf "scale=480:-1,drawtext=text='%{n}':x=4:y=4:fontsize=20:fontcolor=yellow:box=1:boxcolor=black@0.5,tile=5x5" sheets/sheet_%03d.png

drawtext=text='%{n}' stamps the frame index on each tile so the analysis can reference exact frames; tile=5x5 groups 25 consecutive frames per sheet. If drawtext is unavailable in the local ffmpeg build, fall back to plain scale=480:-1,tile=5x5. INPUT is the local clip (or a local copy of the URL). Confirm the sheet count lines up with the probe in Step 2 — every frame must land on a sheet, no exceptions.

Step 4 — Read EVERY sheet + understand the audio (upload_asset, analyze_media, transcribe_audio)

This is the fidelity step, and it has two mandatory halves: read all the frames and understand the audio. Neither is optional.

4a — Read all the frames (every sheet)

For each contact sheet from Step 3 (cover ALL of them — every frame is on a sheet, no exceptions): upload it with upload_asset (the sheets are images, so upload to get a URL), then call analyze_media on that sheet URL with a query that reads the full progression across those 25 frames — pulling out everything the prompt must lock:

• Subject / identity — who/what is in frame; face, hair, build, distinguishing features.
• Exact motion & gestures — what the subject does, beat by beat, referencing the stamped frame indices (e.g. "raises right hand around frame 0048, snaps fingers around frame 0072").
• Camera move — static / pan / push-in / handheld / orbit, and its timing.
• Framing — shot size (close/medium/wide), subject position, headroom.
• Lighting — direction, hardness, color temperature, time of day.
• Palette — dominant colors, mood.
• Wardrobe / props / setting — clothing, objects, background.

Carry the read across all sheets so the entire timeline is understood start-to-end, not just a sampled middle.

4b — Understand the audio (mandatory, not just speech)

Always read the audio — it is required, not optional. Two passes:

• transcribe_audio for speech + timestamps (the words and when they're said), so the prompt can preserve dialogue and lip-sync beats.
• analyze_media on the audio (or the video URL) to understand the NON-speech audio too — music, sound effects, ambience, tone, and rhythm/beats.

Fold the audio understanding into the prompt where it matters — preserve dialogue + lip-sync if anyone speaks, and time the VFX change to an audio beat / sound cue when the change should land on the music or a sound (e.g. snap, hit, downbeat) rather than a bare timestamp.

Assemble the all-frame read and the audio read into a single faithful description of the original — that description is the raw material for Step 5.

Step 5 — Write the Seedance prompt (the authoring step, NEVER skip it)

Compose ONE prompt = a faithful, exhaustive description of the original (the locked elements) + the user's requested change, time-coded + a reference to the input clip via the `@Video1` token. Be thorough: every detail you read in Step 4 that you omit is a detail Seedance is free to change.

Template — fill the `[SLOTS]` from your Step-4 read: