Vfx

vfx

Maps a video-to-video VFX move onto the Pika MCP's Seedance reference-to-video, at 1080p by default (bump to 4K, or drop to 720p, only if the user asks). The user hands you a clip + the change they want; you read EVERY frame (locally extracted with ffmpeg and tiled into contact sheets) and understand the audio (speech + music + SFX + ambience), author a Seedance-faithful prompt that locks the original (face, gestures, camera move) and time-codes the change to the right beat, then re-render the same shot with the VFX baked in. One run produces one clip for one requested change.

For a 4K-only version of this skill with no resolution choice, see /pika:4k-vfx.

How it works — why reading EVERY frame (and the audio) is the mechanism (do not skip the reading pass)

Seedance treats the reference video as a motion/style anchor, NOT a pixel-locked base — it does not copy the input frame-for-frame, it re-generates the shot guided by your prompt. So fidelity comes from the PROMPT: the more exhaustively the prompt describes the original (subject identity, exact gestures, camera move, framing, lighting, palette, wardrobe) and the audio (dialogue + lip-sync, music/SFX beats), the more the output reads as the same shot with the change layered on, at any resolution. Reading EVERY frame — not a sample — is how you build that exhaustive description: you locally extract all frames with ffmpeg, tile them 25-to-a-contact-sheet (5×5), and analyze every sheet so the prompt is built from the entire timeline rather than a handful of stills. Skip the read (or read only a sample) and the output drifts — a different face, a different camera move, a different room, a missed motion beat. The reading pass — all frames + the audio — IS the skill.

Prerequisites

pika MCP available, and local `ffmpeg` in the run environment (used to extract every frame and build the contact sheets). Two inputs from the user: (1) a video clip (a local file or a public URL) and (2) the VFX change they want made to it. A third input is optional: (3) a resolution — defaults to 1080p if not specified. Tools used: upload_asset, probe_media, analyze_media, transcribe_audio, estimate_cost, generate_reference_video, task_status (frame extraction + tiling is done locally with ffmpeg, not an MCP tool).

Stage 0 — Gather inputs (settle this first)

You need exactly two things before generating, plus one implicit default:

1. The video — the original clip to transform. If they gave a local file, you will upload it (Step 1); if they gave a public URL, keep it.
2. The change / VFX — what should be different in the output (e.g. "turn the plaza into an open desert on a finger snap", "make it snow", "set the room on fire behind me", "morph my jacket into glowing armor").
3. The resolution (`state.resolution`) — default to `1080p` unless the user's request says otherwise. Look for explicit signals: "4K", "highest quality", "max resolution" → state.resolution = "4k"; "720p", "cheaper", "faster", "lower res is fine" → state.resolution = "720p". Absent any signal, stay at 1080p — do not ask "which resolution do you want?", just default silently and let the Step-6 gate surface the choice for confirmation. Set state.seedance_model from state.resolution right away, since Step 6 quotes it: `standard` for 1080p or 4k (fast/mini cap at 720p — REQUIRED), `fast` for 720p (cheaper, faster, and sufficient at that resolution).

If either video or change is missing, ask for it before doing anything else — do not invent a change, and do not proceed on a clip you cannot reach. Once both are in hand, say in one line what you're about to do ("Reading every frame of your clip to build the desert-on-snap 1080p prompt…") and run Steps 1–5 without further check-ins — but stop at the Step 6 agreement gate before generating: the render is the expensive, irreversible step, and it only fires after the user approves the prompt, resolution, and cost.

Step 1 — Upload the clip (upload_asset)

Real video must be uploaded — inline base64 only works for tiny assets (<~3MB), so any actual clip goes through the presigned-upload path.

1. Call upload_asset with filename, mime_type (video/mp4 for mp4), and size_bytes (the file's byte size).
2. PUT the raw bytes to the returned presigned_url.
3. Keep the returned public_url → state.clip_url.

If the user already gave a public URL, skip this step and set state.clip_url to it.

Step 2 — Probe (probe_media)

Call probe_media on state.clip_url to read duration, fps, dimensions, and aspect ratio. Use these to: (a) sanity-check the all-frame extraction in Step 3 (duration × fps ≈ total frame count ≈ 25 × number of sheets), and (b) choose the output aspect_ratio and duration here, matched to the source — the Step-6 gate quotes this choice (plus state.resolution from Stage 0) to the user and Step 7 fires with it.

Step 3 — Extract EVERY frame locally and tile into contact sheets (ffmpeg)

Read the whole video, not a sample. Use local ffmpeg to extract every frame at the source rate (no drops, no exceptions), then tile them into legible contact sheets at 25 frames per sheet (5×5 grid) so each analyze_media call covers 25 consecutive frames at once.

Work in a relative working directory (do not use an absolute system temp path):

mkdir -p frames sheets
ffmpeg -i INPUT -vsync 0 frames/f_%05d.png            # every frame, no drops

Then build the 25-per-sheet contact sheets (5×5), scaling each frame down so the sheet is legible and stamping the frame index onto each tile:

ffmpeg -i INPUT -vf "scale=480:-1,drawtext=text='%{n}':x=4:y=4:fontsize=20:fontcolor=yellow:box=1:boxcolor=black@0.5,tile=5x5" sheets/sheet_%03d.png

drawtext=text='%{n}' stamps the frame index on each tile so the analysis can reference exact frames; tile=5x5 groups 25 consecutive frames per sheet. If drawtext is unavailable in the local ffmpeg build, fall back to plain scale=480:-1,tile=5x5. INPUT is the local clip (or a local copy of the URL). Confirm the sheet count lines up with the probe in Step 2 — every frame must land on a sheet, no exceptions.

Step 4 — Read EVERY sheet + understand the audio (upload_asset, analyze_media, transcribe_audio)

This is the fidelity step, and it has two mandatory halves: read all the frames and understand the audio. Neither is optional.

4a — Read all the frames (every sheet)

For each contact sheet from Step 3 (cover ALL of them — every frame is on a sheet, no exceptions): upload it with upload_asset (the sheets are images, so upload to get a URL), then call analyze_media on that sheet URL with a query that reads the full progression across those 25 frames — pulling out everything the prompt must lock:

• Subject / identity — who/what is in frame; face, hair, build, distinguishing features.
• Exact motion & gestures — what the subject does, beat by beat, referencing the stamped frame indices (e.g. "raises right hand around frame 0048, snaps fingers around frame 0072").
• Camera move — static / pan / push-in / handheld / orbit, and its timing.
• Framing — shot size (close/medium/wide), subject position, headroom.
• Lighting — direction, hardness, color temperature, time of day.
• Palette — dominant colors, mood.
• Wardrobe / props / setting — clothing, objects, background.

Carry the read across all sheets so the entire timeline is understood start-to-end, not just a sampled middle.

4b — Understand the audio (mandatory, not just speech)

Always read the audio — it is required, not optional. Two passes:

• transcribe_audio for speech + timestamps (the words and when they're said), so the prompt can preserve dialogue and lip-sync beats.