Asr Transcribe To Text

ASR Transcribe to Text

Transcribe audio/video to speaker-labeled text. Local execution has two explicit routes. Long or unattended recordings use checkpointed whisper.cpp + Silero VAD blocks, then late-fuse pyannote speakers. Short/medium recordings may use the Qwen3-ASR + mlx-whisper alignment route. Neither route cuts ASR input at diarization turns; speaker attribution happens after continuous-context ASR.

Route before ASR: the transcript is the result, not the run

Before starting transcription, check the owning project's transcript catalog, external source index, and declared prior-work carriers for an existing canonical transcript using source ID, date, title, and entity terms. A verified human-reviewed/current transcript ends the task unless the user explicitly asked for a new independent comparison. Raw audio existing is not a reason to regenerate text that already exists.

When no canonical transcript exists:

1. For ordinary meeting/DJI recordings where cloud processing is allowed, use Feishu Minutes as the normal primary route (preprocess to a small M4A first).
2. Use local ASR when the user requires offline/privacy handling, Feishu is unavailable or failed, or the task explicitly needs an independent quality comparison.

2b. For a second, independent speaker timeline, use StepFun's async file endpoint via the stepfun-asr skill's asr_file.py. It is the only vendor diarization route here that does not share code or models with local pyannote, so agreement between the two is real corroboration rather than one tool agreeing with itself. Measured 2026-09-19 on a 10-minute film: both split the key turn identically, while Feishu Minutes mis-attributed it. The constraint is that it fetches a public URL — base64 is rejected — so it needs staging (see aliyun-multi-account.md); reach for it when an attribution is load-bearing, not for routine transcripts.

1. For non-meeting media or an explicit local/remote ASR request, choose the execution location by the audio-location rule below.

Do not run local ASR merely to make a two-route process look complete.

Choosing between them is usually not about speed — it's about where the audio already is. A remote GPU can be several times faster (a 4090 running vLLM measured ~61x realtime against ~15x for local MLX), but that gap is small change next to moving the files: transcription output is text, and text is ~10,000× smaller than the audio it came from (18.5 h of speech ≈ 330 K characters ≈ 1 MB, from ~2.6 GB of WAV). So:

Transcribe where the audio already lives, and move only the transcript.

Pulling a few hundred MB across a slow link to reach a faster GPU routinely costs more wall-clock than the entire transcription — measured once at 63 KB/s, which is over two hours for 500 MB, to save minutes of compute. If the recording is already on the remote box (it was recorded there, downloaded there, or lives in a share mounted there), run the ASR there and bring back the .txt.

Configuration persists in ${CLAUDE_PLUGIN_DATA}/config.json.

Speaker labels are the default. Every run produces [start-end] SPEAKER_xx: text + CSV. Plain-text-only output is the opt-out (--no-diarization) for monologues, podcasts, or when you just want a summary — see Step 3. It also drops every timestamp: the plain-text path returns the session text and nothing else. A monologue that needs line-level timestamps (subtitles, jump-to-moment archives) runs the full pipeline and ignores the SPEAKER_00 labels. One-time setup for diarization: pyannote is a gated HuggingFace model — it needs a token once (## Speaker Diarization & Identification below). First run without it FAILS with setup steps; after setup, full capability is permanent and auto-detected.

Step 0: Detect Platform and Load Config

cat "${CLAUDE_PLUGIN_DATA}/config.json" 2>/dev/null

If config exists, read values and proceed to Step 1.

If config does not exist, auto-detect platform first:

python3 -c "
import sys, platform
is_mac_arm = sys.platform == 'darwin' and platform.machine() in ('arm64', 'aarch64')
print(f'Platform: {sys.platform} {platform.machine()}')
print(f'Apple Silicon: {is_mac_arm}')
if is_mac_arm:
    print('RECOMMEND: local-mlx')
else:
    print('RECOMMEND: remote-api')
"

Then use AskUserQuestion with platform-aware defaults:

For macOS Apple Silicon (recommended: local):

ASR setup — your Mac has Apple Silicon, so local transcription is recommended.

Q1: Transcription mode?
  A) Local MLX — runs on your Mac's GPU, no API key needed, 15-27x realtime (Recommended)
  B) Remote API — send audio to a server (vLLM, Tailscale workstation, etc.)

Q2: Does your network have an HTTP proxy that might intercept traffic?
  A) Yes — bypass proxy for ASR traffic (Recommended if using Shadowrocket/Clash)
  B) No — direct connection

For other platforms (recommended: remote):

ASR setup — local MLX requires macOS Apple Silicon. Using remote API mode.

Q1: ASR Endpoint URL?
  A) https://asr.example.com/v1/audio/transcriptions (Self-hosted remote ASR)
  B) http://localhost:8002/v1/audio/transcriptions (Local ASR server)
  C) Custom URL

Q2: Proxy bypass needed?
  A) Yes (Recommended for Shadowrocket/Clash/corporate proxy)
  B) No

Save config:

mkdir -p "${CLAUDE_PLUGIN_DATA}"
python3 -c "
import json
config = {
    'mode': 'MODE',           # 'local-mlx' or 'remote-api'
    'model': 'MODEL_ID',      # local: 'mlx-community/Qwen3-ASR-1.7B-8bit', remote: 'Qwen/Qwen3-ASR-1.7B'
    'max_tokens': 8192,       # local only; PER 20-minute chunk, not per recording
    'endpoint': 'URL',        # remote only
    'noproxy': True,
    'max_timeout': 900        # remote only
    # 'diarization_declined': True  # set only after the user explicitly declines
    #   the pyannote setup in Step 3 — every run then warns + goes plain-text
    #   until an HF token appears (auto-detected)
}
with open('${CLAUDE_PLUGIN_DATA}/config.json', 'w') as f:
    json.dump(config, f, indent=2)
print('Config saved.')
"

Step 1: Resolve Input

Accept local files, direct media URLs, or web/podcast episode pages.

• Web or podcast page URL: inspect the page for an existing transcript first. Use an official/platform transcript only when it is directly accessible to the user's account. If the transcript endpoint requires a login token and none is available, say that clearly and fall back to ASR from the audio URL.
• Local file, direct media URL, or page URL fallback: run the bundled resolver. It extracts media from common page metadata (og:audio, media tags, JSON-LD, RSS-style enclosure links), downloads URLs with atomic temp-file replacement, verifies remote Content-Length when present, computes SHA-256, and validates the result with ffprobe.

uv run ${CLAUDE_SKILL_DIR}/scripts/resolve_media_input.py \
  INPUT_FILE_OR_URL [INPUT_FILE_OR_URL2 ...] \
  --output-dir OUTPUT_DIR \
  --manifest OUTPUT_DIR/media_manifest.json

For suspicious or high-value downloads, add --decode-check to make ffmpeg decode the whole file before transcription:

uv run ${CLAUDE_SKILL_DIR}/scripts/resolve_media_input.py \
  "https://www.xiaoyuzhoufm.com/episode/EPISODE_ID" \
  --output-dir OUTPUT_DIR \
  --manifest OUTPUT_DIR/media_manifest.json \
  --decode-check

Expected output:

Downloaded ... bytes in ...s -> OUTPUT_DIR/episode-title.m4a
OUTPUT_DIR/episode-title.m4a