Login
guia-matthieu/clawfu-skillsMIT3 files

Web Scraper

Extract structured data from websites. Use when: collecting competitor pricing; scraping product listings; extracting contact information; gathering research data; monitoring website changes

Use this skillView repositoryView on skills.sh
Specification
Skill ID
guia-matthieu/clawfu-skills/web-scraper
Publisher
guia-matthieu
Repository
clawfu-skills
Installs
0
Files
3
License
MIT
Synced
Sep 23, 2026
How to use it

Open any RiverX project, open the Skills panel in the chat, and search for this identifier. The files are fetched from the source repository at install time.

guia-matthieu/clawfu-skills/web-scraperInstalls these files
  • SKILL.md
  • scripts/main.py
  • scripts/requirements.txt

What this skill tells the agent

Web Scraper

Extract structured data from websites using BeautifulSoup and requests - turn any webpage into usable data.

When to Use This Skill

  • Competitor research - Scrape pricing, features, positioning
  • Lead generation - Extract contact info from directories
  • Content audit - Pull headings, links, meta data
  • Price monitoring - Track competitor pricing changes
  • Data collection - Gather research data from multiple sources

What Claude Does vs What You Decide

Claude DoesYou Decide
Structures analysis frameworksStrategic priorities
Synthesizes market dataCompetitive positioning
Identifies opportunitiesResource allocation
Creates strategic optionsFinal strategy selection
Suggests implementation approachesExecution decisions

Dependencies

pip install beautifulsoup4 requests pandas click lxml

Commands

Scrape Elements

python scripts/main.py scrape https://example.com --selector "h1,h2,p"
python scripts/main.py scrape https://example.com --selector ".product-price"

Extract Links

python scripts/main.py links https://example.com
python scripts/main.py links https://example.com --internal-only

Extract Emails

python scripts/main.py emails https://example.com
python scripts/main.py emails https://example.com --depth 2

Extract Structured Data

python scripts/main.py structured https://example.com/article --schema article
python scripts/main.py structured https://example.com/product --schema product

Examples

Example 1: Scrape Competitor Pricing

python scripts/main.py scrape https://competitor.com/pricing --selector ".price,.plan-name"

# Output:
# Extracted 6 elements
# 1. Starter - $29/mo
# 2. Pro - $99/mo
# 3. Enterprise - Contact us

Example 2: Extract Article Content

python scripts/main.py structured https://blog.example.com/post --schema article

# Output: article_data.json
# {
#   "title": "How to Scale Your Startup",
#   "author": "Jane Doe",
#   "date": "2024-01-15",
#   "content": "...",
#   "word_count": 1523
# }

CSS Selector Reference

SelectorDescriptionExample
tagElement typeh1, p, div
.classClass name.price, .title
#idElement ID#main-content
tag.classTag with classdiv.product
tag[attr]Has attributea[href]
parent > childDirect childul > li
tag1, tag2Multipleh1, h2, h3

Ethical Scraping Guidelines

  1. Check robots.txt - Respect site's scraping policy
  2. Rate limit - Don't overload servers (1-2 req/sec)
  3. Identify yourself - Use descriptive User-Agent
  4. Cache requests - Don't re-scrape unchanged pages
  5. Terms of Service - Check if scraping is allowed

Skill Boundaries

What This Skill Does Well

  • Structuring strategic analysis
  • Identifying market opportunities
  • Creating strategic frameworks
  • Synthesizing competitive data

What This Skill Cannot Do

  • Replace market research
  • Guarantee strategic success
  • Know proprietary competitor info
  • Make executive decisions

Related Skills

  • competitor-monitor - Monitor competitor changes
  • pdf-extractor - Extract from PDFs

Skill Metadata

  • Mode: centaur
category: automation
subcategory: data-extraction
dependencies: [beautifulsoup4, requests, pandas]
difficulty: intermediate
time_saved: 5+ hours/week

More from guia-matthieu

guia-matthieu/clawfu-skills

Conversion Copywriting

Write copy that gets a \"yes\" using Joanna Wiebe's research-first, Voice of Customer methodology Use when: **Writing landing pages, emails, or sales pages** that need measurable conversion results; **Starting a new copy project** and need a systematic process to follow; **Struggling with what to write** and staring at a blank page; **Wanting to prove ROI** to clients with data-backed decisions; **Improving existing copy** through validation and testing

guia-matthieu/clawfu-skills

Design Trends 2026

Apply 2026's top graphic design trends to any creative brief. Based on Kittl × Savee's 2026 Design Trends Report (10 trends + 2 honorable mentions), backed by Adobe, Figma, and Pinterest data. Use when: **Designing a brand identity** — pick the right aesthetic for your audience; **Creating social media assets** — use trending visual languages that perform; **Briefing a designer or AI image tool** — give precise style direction with vocabulary and references; **Refreshing a visual identity** — know what's rising vs saturating; **Building mood boards** — combine trends intentionally with data-backed rationale.

guia-matthieu/clawfu-skills

Headline Formulas

25+ proven headline formulas that stop the scroll, capture attention, and drive clicks. Templates and examples for every situation. Use when: Writing headlines for landing pages, ads, or articles; Creating email subject lines that get opens; Crafting social media hooks; A/B testing headline variations; Overcoming headline writer's block

Use this skill in a project

Open any RiverX project, search the Skills panel for guia-matthieu/clawfu-skills/web-scraper, and install it in one click.

Use this skill