Language Swap

/pika:language-swap

Tools below are Pika MCP tools, named bare — call each under whatever prefix your session exposes for the Pika MCP.

Translate and dub a video into another language while preserving the original speaker's voice. Pipeline: dub (one worker call) → lipsync (default ON) → burn target-language captions or bilingual captions.

The dubbing worker does the heavy lifting in a single call: it transcribes, translates, preserves each speaker's voice server-side (no separate clone step), and returns a fully A/V-synced video — so there is no manual transcribe/clone/TTS/replace chain to manage and no duration-drift handling to do by hand.

Segmented / multi-language dub (per-range languages)

Use this when the user wants different languages on different parts of one video (e.g. first half Spanish, second half Japanese), or wants to translate only some sections and keep the rest in the original voice. Both are the same thing: a timeline of segments, each tagged with a language; any uncovered range keeps the original audio.

dub_video takes a segments plan instead of target_language (pass exactly one — they are mutually exclusive):

dub_video(source_video_url=<video_url>, segments=[
  {start_s: 0,  end_s: 30, target_language: "es"},
  {start_s: 30, end_s: 60, target_language: "ja"}
])

How to build the plan: the user needs to know where the content is before they can pick ranges, so transcribe first — extract the audio with extract_audio_from_video, then transcribe_audio(audio=<audio_url>, timestamps=true), show the user the timestamped segments, and let them say which time range goes to which language. Then assemble segments[] (seconds, ordered, non-overlapping) and make ONE dub_video call. There is no separate "video understanding" tool — the timestamped transcript is the understanding step.

Behavior of the segmented path:

• Shared voice across all segments. The source speaker is cloned once and every segment — in every language — is spoken in that same cloned voice, then the clone is recycled, all inside the one dub_video call. You never clone or delete a voice yourself.
• Keep-original. Any time range NOT covered by a segment plays the original audio (voice + background) untouched. To translate only parts of a video, list only the parts you want translated.
• Length-locked. Output stays exactly the source length (each dubbed range is speed-fit to its window), so boundaries line up with the original timeline.
• Provider. Mixed-language-per-range always uses the voice-cloning route automatically; the single-call whole-video dubbing route can't mix languages per range, so don't force a single-call provider for a segmented plan. Every covered language must be supported on the voice-cloning route — if one isn't, surface the error and consult references/language-coverage.md.
• Result. Same dubbed-video result; target_language echoes the covered languages comma-joined (e.g. "spa,jpn"), and no single transcript_language is returned (the track is multi-language). Lipsync (Step 2, default ON, ≤5 min) still runs on the whole dubbed video. For captions (Step 3), use the returned multi-language subtitles[] in caption_mode="manual"; auto re-transcription can't pick a single language for a mixed track.

If dub_video rejects segments (older deployment without segmented support), fall back to dubbing each range single-language and concatenating — but prefer the one-call segmented path when available.

Behavior defaults

• Target language: required via --to <language>. Prefer language codes: es, fr, ja, de, pt-BR, zh-Hans. The dubbing worker accepts ISO/BCP-47-like tags and normalizes script/region subtags before calling ElevenLabs (for example zh-Hans → zh; zh-Hant-TW → zh).
• Lipsync: ON by default — re-matches the speaker's mouth to the translated audio (fal sync-lipsync; the full-video lip-matcher, distinct from the portrait-image animator). Pass --no-lipsync to skip it when the source has no on-camera face or to avoid the meaningful cost (~$4/min on the sync-2-pro tier). Applies only to videos ≤5 min — edit_lipsync hard-caps at 300 s upstream, so longer sources auto-skip lipsync (see Step 2); the dub itself has no length limit.
• BGM / background music: kept by default — the dub lays the translated voice over the original music / SFX bed. Pass --no-bgm for a translate-only output: the worker drops the original music and keeps only the translated speech (drop_background_audio=true).
• Captions: target-language captions are burned by default. When the user asks for bilingual / dual subtitles, burn the target-language (translated) row on top and the source-language (original) row below it — after dubbing, the translated speech is what's actually being said, so it's the primary row; the original is the secondary reference.
• Bilingual captions: enable when the user passes --bilingual-subtitles or asks for "bilingual subtitles", "dual subtitles", "two-language captions", "original + translated subtitles", "双语字幕", or "原文+译文字幕".
• Language coverage: if language support is questioned or a language-related upstream error occurs, consult references/language-coverage.md. Do not proactively surface provider-specific language-list details in normal user replies.

State variables produced and consumed

• video_url: input — from positional arg
• source_input_url: original positional URL — preserved for diagnostics if video_url is rehosted
• target_language: text — from --to <language>
• with_lipsync: boolean — defaults true; false only when --no-lipsync
• no_bgm: boolean — true when --no-bgm (maps to drop_background_audio=true)
• bilingual_subtitles: boolean — true when the user asks for bilingual / dual subtitles
• dubbed_video_url: dubbed, A/V-synced video — produced by Step 1
• dub_subtitles: optional target-language timed subtitles from the dub result — consumed by Step 3
• source_subtitles: optional source-language timed subtitles from the dub result — consumed by Step 3 for bilingual captions
• dub_transcript_srt: optional target-language SRT from the dub result — returned for review/debugging
• source_transcript_srt: optional source-language SRT from the dub result — returned for review/debugging
• source_transcript_language: optional source-language code from the dub result
• lipsynced_video_url: dubbed video with mouth re-matched — produced by Step 2 (when lipsync runs)
• caption_target_video_url: final visual video URL before captions are burned
• final_video_url: video with target-language captions burned in — produced by Step 3

Step 0 — Parse input

Required:

• Positional video_url — MUST be https://...
• --to <language> — target language (free-text or BCP-47 code)

Optional:

• --no-lipsync — skip the default mouth-matching step.
• --no-bgm — translate-only output; drop the original music/SFX bed.
• --bilingual-subtitles — burn source-language + target-language subtitle rows.

Infer bilingual_subtitles=true from user wording even if the explicit flag is absent.

If --to is missing, STOP and prompt the user — UNLESS the user wants different languages on different parts, or to translate only some sections: that is the per-range segmented path (see "Segmented / multi-language dub" above), which uses a segments plan instead of --to.

For the segmented path, first build the time-range plan: extract the audio with extract_audio_from_video, then transcribe it with timestamps via transcribe_audio(audio=<audio_url>, timestamps=true), show the user the timestamped segments, and capture which time range maps to which language into segments[].

Outputs: video_url, target_language, with_lipsync (default true), no_bgm (default false), bilingual_subtitles (default false).

Step 1 — Dub the video (state: dubbed_video_url)