Crawl4ai

Crawl4AI

Verified against `crawl4ai` `VERSION`. PEP 723 pins in scripts/*.py and tests/*.py floor at that version.

Overview

Crawl4AI wraps a headless browser (Playwright) plus a markdown-aware content pipeline. Use it when defuddle/curl can't reach the content — JavaScript-rendered pages, login-gated content, infinite scroll, multi-URL concurrency, repeatable schema-based extraction.

This skill exposes both interfaces of the underlying library:

• CLI (crwl) — quick, scriptable commands: CLI Guide
• Python SDK — full programmatic control: SDK Guide

Invoked with a URL argument

When the user runs /crawl4ai <url> with a single URL and no further qualifier, treat it as the JS-heavy fetch case and default to:

crwl <url> -c "wait_until=networkidle,page_timeout=60000" -o markdown

wait_until=networkidle waits for the network to be quiet for ~500ms post-load — the right default when the user hasn't named a specific element on a JS-rendered page. (Avoid wait_for=css:body: <body> exists at t=0 on every HTML response, so it's satisfied before JS renders content.) Then return the markdown to the agent context. Adjust to wait_for=css:<selector> if the user named a specific element. Skip the default and route to the relevant section below for any task that names extraction, batch / multi-URL, login / session, screenshot / PDF, or URL discovery — those each have their own pipeline. If the URL is clearly static (a docs page, a blog post), route the user to /fetch-web instead per the "When NOT to use" section below.

When NOT to use this skill

• Static HTML pages (most documentation sites, blog posts, news articles, tweets) — use /fetch-web or defuddle directly. Static extraction is ~0ms cold start; crawl4ai pays a ~2s browser startup tax.
• Local file conversion (.pdf, .docx, .pptx, .epub) — use /markdown-convert.
• One-URL agent-context reads (the agent just needs to read this page) — use /fetch-web and let it route to defuddle.
• Mutating UI flows (form fills, multi-step clicks, login + navigation) — /browse (gstack's persistent headless Chromium) is built for that.

When stuck

For unknown crwl/SDK flags, scrape failures, or extraction edge cases the references don't cover, see references/escalation.md for the lookup order (qmd solutions → upstream docs → GitHub issues → ask the user) and worked examples.

Quick Start

Installation

pip install crawl4ai
crawl4ai-setup

# Verify installation
crawl4ai-doctor

CLI (Recommended)

# Basic crawling - returns markdown
crwl https://example.com

# Get markdown output
crwl https://example.com -o markdown

# JSON output with cache bypass
crwl https://example.com -o json -v --bypass-cache

# See more examples
crwl --example

Python SDK

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com")
        print(result.markdown[:500])

asyncio.run(main())

For SDK configuration details: SDK Guide - Configuration.

Core Concepts

Configuration Layers

Both CLI and SDK use the same underlying configuration:

Key Parameters

Browser Configuration:

• headless: Run with/without GUI
• viewport_width/height: Browser dimensions
• user_agent: Custom user agent
• proxy_config: Proxy settings

Crawler Configuration:

• page_timeout: Max page load time (ms)
• wait_for: CSS selector or JS condition to wait for
• cache_mode: bypass, enabled, disabled
• js_code: JavaScript to execute
• css_selector: Focus on specific element

For complete parameters: CLI Config | SDK Config

Output Content

Every crawl returns:

• markdown - Clean, formatted markdown
• html - Raw HTML
• links - Internal and external links discovered
• media - Images, videos, audio found
• extracted_content - Structured data (if extraction configured)

Markdown Generation (Primary Use Case)

Crawl4AI excels at generating clean, well-formatted markdown.

CLI

crwl https://docs.example.com -o markdown                              # raw markdown
crwl https://docs.example.com -o markdown-fit                          # filtered (noise removed)
crwl https://docs.example.com -f templates/filter_bm25.yml -o markdown-fit   # BM25-relevance filter
crwl https://docs.example.com -f templates/filter_pruning.yml -o markdown-fit # quality-based filter

Filter templates: `templates/filter_bm25.yml` (relevance-scored against a query), `templates/filter_pruning.yml` (no query, prunes low-quality blocks).

Python SDK

from crawl4ai.content_filter_strategy import BM25ContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

bm25_filter = BM25ContentFilter(user_query="machine learning", bm25_threshold=1.0)
md_generator = DefaultMarkdownGenerator(content_filter=bm25_filter)

config = CrawlerRunConfig(markdown_generator=md_generator)
result = await crawler.arun(url, config=config)

print(result.markdown.fit_markdown)  # Filtered
print(result.markdown.raw_markdown)  # Original

For filter selection and config field reference, see Content Filters.

Data Extraction

1. Schema-Based CSS Extraction (Most Efficient)

No LLM required at extract time — fast, deterministic, cost-free. One-time LLM cost to derive the schema, then reuse indefinitely. The bundled scripts split the pipeline by responsibility:

./scripts/generate_schema.py https://shop.example.com "products with name, price, image" shop_schema.json
./scripts/extract_with_schema.py https://shop.example.com shop_schema.json products.json

Or via the CLI with the YAML strategy template + the saved schema:

crwl https://shop.example.com -e templates/extract_css.yml -s shop_schema.json -o json

Schema skeleton: `templates/css_schema.json`. Strategy YAML: `templates/extract_css.yml`.

2. LLM-Based Extraction

For one-off / irregular content where a CSS schema is too brittle:

./scripts/extract_with_llm.py https://news.example.com "Extract headlines, dates, summaries" news.json

Or via the CLI with the strategy template:

crwl https://news.example.com -e templates/extract_llm.yml -o json

Strategy YAML: `templates/extract_llm.yml`. Pays an LLM call per URL — for repeat extraction, prefer the schema pipeline above.

For extraction strategy reference: Extraction Strategies.

Advanced Patterns

Dynamic Content (JavaScript-Heavy Sites)

crwl https://example.com -c "wait_for=css:.ajax-content,scan_full_page=true,page_timeout=60000"
crwl https://example.com -C templates/crawler.yml                          # all options in a YAML file

Crawler config template: `templates/crawler.yml`.

Multi-URL Processing