Web Crawler

Preferred entry: call exports.py (don't hand-roll requests)

Ready-made helpers live in skills/web-crawler/exports.py. Prefer them over writing your own proxied_get/proxied_post calls — they already inject the proxy credentials, so there is no API key to find (don't read $SCRAPECREATORS_API_KEY / $FIRECRAWL_API_KEY, don't check .env, don't ask the user).

import sys; sys.path.insert(0, "/data/workspace/skills/web-crawler")
from exports import scrape_markdown, youtube_transcript, sc_get
scrape_markdown("https://example.com/article")          # Firecrawl fallback
youtube_transcript("https://youtube.com/watch?v=ID")     # ScrapeCreators
youtube_video("https://youtube.com/watch?v=ID")          # metadata: title/description/uploader (+ transcript)
sc_get("/v1/tiktok/profile", handle="charlidamelio")     # any SC endpoint

from exports import archive_fallback                      # paywall / Firecrawl-403
archive_fallback("https://www.nytimes.com/.../article.html")  # archive snapshot

Transcript route — ONE entry for any media link

from core.skill_tools import web_crawler
r = web_crawler.get_transcript(url, caller_id="chat:<thread>")
# {found, kind, source, text, url, title, tried, note}

Design rule: captions / subtitles / published transcripts are METADATA; the media file is PAYLOAD. Exhaust every metadata provider first; touch the media only when the user explicitly agrees. Same rule claude-video, spoken.md and yt-dlp's --skip-download workflow follow. Nothing here is site-specific:

found=False ⇒ report kind, title, tried, note (and candidate if any) and ask the user before any download. YouTube is recognised by URL: the captions API runs even when yt-dlp metadata fails, and a page scrape is never reported as a transcript for a media URL. yt-dlp --skip-download / -J / --list-subs are metadata calls and pass the bash gate; yt-dlp <url>, curl … .mp3, ffmpeg, whisper are held for confirmation.

Title only, no link → web_search("<show> <title> transcript") then get_transcript(<page url>).

Incident 2026-09-17: two 403s on transcript hosts led an agent to a 49 MB mp3 + ffmpeg + 4 whisper calls + an incomplete summary. Same link through get_transcript: RSS had no transcript → RSS declares youtube.com/@a16z → channel listing → one upload within 1.2 % duration, same day → captions, 55k chars, ~30 s, zero audio.

Quick trigger rules (read this first)

Use this skill immediately when any of these conditions is true:

• web_fetch returns HTTP 401/403/429/5xx
• Response is an anti-bot challenge page (for example Cloudflare "Attention Required", "Just a moment", or challenge/captcha pages)
• The page is JS-heavy and the first fetch misses required detail fields (for example publish time, author, listing code, updated time, price breakdown)
• Search results do not contain the requested field and the value must be extracted from the target page itself

Fallback rule:

• If ordinary fetch is blocked or incomplete, switch to Firecrawl fallback in this skill before asking the user for screenshots/manual text.

Error signatures -> action

Paywall / Firecrawl-blocked fallback chain (use archive_fallback)

When Firecrawl can't get the page either (it returns 403, or markdown comes back empty) the site is behind a hard paywall or aggressive WAF. Do NOT keep retrying Firecrawl. Recover the article from a web archive snapshot instead:

from exports import archive_fallback
res = archive_fallback("https://www.nytimes.com/.../article.html")
res["markdown"]      # full text, or "" if no snapshot exists anywhere
res["source"]        # "archive.today" | "wayback" | None
res["snapshot_url"]  # the snapshot that was scraped

How it works (and why this order):

• archive.today first (archive.ph / archive.is mirrors). User-triggered, real-browser captures; historically preserves full text behind paywalls. Best bet for NYT/WSJ/Economist. We scrape its /newest/ snapshot via Firecrawl (archive.today has its own Cloudflare, so scrape it through Firecrawl, never web_fetch it directly).
• Wayback Machine second (archive.org). Automated crawler that honors robots.txt and paywalls, so it often has NO full text for hard paywalls — but it's a good fallback for ordinary 403/Cloudflare pages that aren't paywalled.

Limitation: archives only return text someone already saved. If res["markdown"] == "", no snapshot exists — stop, tell the user, and try the outlet's official API/RSS or a different source. Do not fabricate the article.

China-app structured data (use apify_run)

When you need structured data from a China app — Douyin video search, Xiaohongshu notes, Weibo posts, Bilibili videos, JD/Taobao product prices, 1688 wholesale listings, Xianyu second-hand, Dewu sneakers, etc. — Firecrawl and ScrapeCreators don't cover these platforms. Use the Apify Store fallback instead. Apify is a serverless scraper marketplace with hundreds of community-maintained actors that run real browsers + proxy pools against Chinese platforms.

Auth: No user-supplied key needed. sc-proxy injects the platform Apify token automatically. The Authorization: Bearer header can be any fake value — the proxy replaces it with the real token. Do NOT read $APIFY_TOKEN from env, do NOT check .env, do NOT ask the user for an Apify key.

When to use Apify (vs Firecrawl/ScrapeCreators):

• ✅ China apps: 抖音, 小红书, 微博, B站, 京东, 淘宝, 1688, 闲鱼, 得物, 携程, 知乎, 豆瓣, 雪球, 快手, 爱奇艺, 优酷
• ✅ Southeast Asia e-commerce: Shopee, Lazada, Temu
• ❌ Western social media (TikTok/Instagram/YouTube/X/Reddit) → use ScrapeCreators first (cheaper)
• ❌ Generic web page scraping → use Firecrawl first (cheaper)