Feishu Doc Scraper

Feishu Doc Scraper

Extract a Feishu/Lark source into faithful local Markdown. Prefer the lark-cli API — it extracts the body programmatically (no model paraphrasing), follows a collection's reference graph, and reads permission boundaries from error codes instead of guessing. Treat the rendered browser page as a fallback, not the source of truth: in real collection-scraping work the API path consistently does the whole job while the browser path is never needed.

Scope (read this first)

This skill's contract is faithful per-source Markdown + a record of what was extracted. It does not decide how the resulting files are named, indexed, deduplicated against existing notes, or organized into a knowledge base — that belongs to the host PKM / the user's own conventions. Stopping at faithful extraction keeps this skill orthogonal and reusable. When the user wants the output filed into a vault, extract first, then hand the clean Markdown to their organizing workflow.

For every document read or archived, load [comments and feedback](references/comments-and-feedback.md). It owns the body-plus-discussion workflow, selected scope, body-only exception, snapshot commands, interpretation and coverage checks.

Extraction and durable storage are separate decisions. A downloaded MP4/XLSX/DOCX/image is a working copy, not evidence that the file belongs in Git or Git LFS. For a knowledge-base archive, default to:

• Git: faithful Markdown plus structured CSV/JSON/HTML, source locators, revision/permission state, byte counts, MIME types, and hashes.
• Platform original: Feishu remains the source of record for raw binary attachments when the stable document/file token is still retrievable. A local download is an optional cache and may be absent on a fresh clone.
• OSS/object storage: only when the archive needs an independent durable copy or the platform source is not a reliable long-term retrieval path. Uploading is a separate authorized operation, not a fallback the extractor chooses by itself.

Before filing an archive, declare this split in an artifact manifest and run the bundled storage validator. The complete schema and examples are in [references/archive-storage-contract.md](references/archive-storage-contract.md).

Choose the path

Is the source a Feishu/Lark URL (wiki / docx / sheets / minutes / base)?
├── YES → is lark-cli installed and authenticated to that tenant?
│        ├── YES → PATH A: lark-cli API extraction  (primary — start here)
│        │         └── hit code 131006 / 99991679 (permission denied)?
│        │              └── PATH B: owner-exported .docx → faithful Markdown
│        └── NO  → install/auth lark-cli first (it is worth it); only if
│                  truly impossible → PATH D: browser DOM fallback
├── the URL is a Minutes / 妙记 link, or a doc references one → PATH C: Minutes transcript
└── you were handed an exported .docx (not a URL) → PATH B

⚠️ `base` (Bitable) is not actually operationalized in Path A yet — the extractor only records the token (DISPATCH["url-base"]: "Bitable API (outside this skill) — record token") and Step 2 below has no row for it. Listed here for completeness of what a Feishu URL can be, not as a claim that Path A extracts Bitable content end-to-end.

A collection/hub is just a docx whose body references other docs — Path A handles it by recursively following the reference graph, not by visiting pages in a browser.

Path A — lark-cli API extraction (primary)

Full command catalog, recursion engine, cross-tenant and personal-space nuances: [references/lark-cli-api-extraction.md](references/lark-cli-api-extraction.md). The essentials for the common case:

1. Disable the proxy for Feishu domestic domains. Feishu's *.feishu.cn endpoints are direct-connect in mainland China; routing them through a local proxy leaks credentials through the proxy and gets DNS-hijacked. lark-cli itself warns about this. Always:

export LARK_CLI_NO_PROXY=1

This does not conflict with any "Claude/Anthropic domains must use the proxy" rule — Feishu is a different host and is direct.

2. Classify the URL, then resolve to a fetchable doc token.

• …/wiki/<node_token> — a wiki node token is not a doc token. Resolve it first: ``bash lark-cli wiki spaces get_node --params '{"token":"<node_token>"}' # → .data.node.obj_token and .data.node.obj_type (e.g. "docx") ``
• …/docx/<doc_token> — already a doc token, fetch directly.
• …/sheets/<token> — spreadsheet, use the sheets commands (see reference).
• …/minutes/<token> — Minutes, go to Path C.

3. Fetch the body programmatically — never via the model. The body field moved across lark-cli versions, so probe both rather than hard-coding one (this keeps working whichever version is installed):

lark-cli docs +fetch --doc <obj_token> --format json > /tmp/fetch.json 2> /tmp/fetch.err
# ≤1.0.32: clean Markdown in .data.markdown.
# 1.0.55: body moved to .data.document.content as HTML (.data.markdown is null).
if jq -e -r '.data.markdown // empty' /tmp/fetch.json > "<sanitized-title>.md" && [ -s "<sanitized-title>.md" ]; then
  : # got clean Markdown directly
else
  jq -r '.data.document.content' /tmp/fetch.json > "<sanitized-title>.html"
  pandoc -f html -t gfm "<sanitized-title>.html" > "<sanitized-title>.md"
fi

⚠️ `<sanitized-title>` must be a distinct name per document, never the literal string "source" reused across fetches. Step 4 below fetches multiple documents (the hub, then every child it references) into the same working directory — running this exact snippet twice with a hardcoded source.md/source.html would let the second fetch silently overwrite the first document's saved body before you ever got to check it. The rest of this section still says "source.html"/"source.md" as shorthand for whichever document's own saved files you're currently looking at — not one shared filename for the whole collection.

⚠️ On this pandoc branch, `pandoc -f html -t gfm` silently strips several of Feishu's custom embedded tags — verified against real documents, 2026-08-16/17, and the damage is worse the deeper you look:

• whiteboard (<whiteboard token="…">, an inline diagram block) vanishes with zero trace. Confirmed on a real document: 3 raw tags in .data.document.content → 0 trace in the pandoc-converted source.md (grep -c whiteboard found 3 hits in the raw HTML, 0 in the converted Markdown).
• mention-doc — whose real raw tag is <cite doc-id="…" file-type="wiki|docx" title="…" type="doc"></cite>, not the <mention-doc token="…" type="…">Title</mention-doc> shape this skill originally assumed — also vanishes with zero trace, and worse than just losing the token/type: the title lives in a title="…" attribute and the tag body is empty, so pandoc drops the whole element and not even the bare title text survives.
• sheet is unconfirmed either way (no real sheet-reference document was found to test) — treat it as capable of silently vanishing too until proven otherwise.
• image — whose real raw tag is a standard <img src="<drive-token>" alt="…" …>, not <image token="…"> — does not vanish: pandoc passes the raw <img> element through mostly intact (src/id/href/width/height/alt survive; name= is dropped; mime=/scale= are renamed data-mime=/data-scale=).
• lark-table turns out not to be a real tag for ordinary docx tables at all — they use plain <table> HTML, and pandoc generally converts them intact (clean GFM pipe-table syntax for single-paragraph cells, or a raw <table> HTML block for multi-paragraph cells) — not part of this silent-loss class.