webclaw

// Overview

webclaw is a web content extraction tool written in Rust. Its README frames the problem it solves: most scraping tools hand an AI agent either a blocked page, login wall or empty app shell, or raw HTML full of navigation, scripts, styling, ads and duplicated boilerplate. webclaw turns a URL into clean content instead — markdown, plain text, cleaned HTML, structured JSON, or a compact "llm" format meant for agent context and RAG ingestion.

The repository holds the open-source CLI, an MCP server, the extraction engine and a self-hostable server, split into Rust crates: webclaw-core (HTML to markdown, text, JSON and LLM-ready output, with no network I/O so it can be used on its own), webclaw-fetch (fetching, crawling, batching and URL mapping), webclaw-llm (local and hosted LLM providers), webclaw-pdf (PDF text extraction), webclaw-mcp and webclaw-cli. Through MCP, clients such as Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode and Codex CLI get scrape, crawl, map, batch, extract, summarize, diff and brand as native tools.

The CLI and MCP server work locally without an account for the core extraction path. The same project also runs a hosted API at webclaw.io, with a free plan and paid tiers, for protected-site access, JavaScript rendering, async crawl and research jobs, web search and usage tracking; TypeScript, Python and Go SDKs call that API. LLM-powered features such as structured extraction and summaries can run against a local Ollama instance or an OpenAI-compatible, Anthropic-compatible or OrcaRouter endpoint configured through environment variables.

// What it does

• Five output formats — markdown, llm, text, json and html — plus --only-main-content and CSS --include / --exclude selectors to strip navigation, footers and ads
• Same-origin crawling with depth and page limits, URL mapping without full extraction, and parallel batch scraping of many URLs
• An MCP server (zero-install via npx @webclaw/mcp, or auto-configured by npx create-webclaw) that exposes scrape, crawl, map, batch, extract, summarize, diff and brand to agent clients
• LLM-backed structured extraction and page summaries using a local Ollama model or an OpenAI-, Anthropic- or OrcaRouter-compatible provider
• Page snapshot diffing with --diff-with and brand extraction of colors, fonts, logos and metadata with --brand
• Optional hosted API (--cloud, or the TypeScript / Python / Go SDKs) for JavaScript rendering, protected sites, web search and research jobs, and proxy support via WEBCLAW_PROXY or a proxy pool file

// Getting started

webclaw ships as a single binary. The README lists Homebrew, prebuilt macOS / Linux / Windows binaries on GitHub Releases, a Docker image and a Cargo install from Git; for agent use, an npx installer configures the MCP server for you.

brew tap 0xMassi/webclaw
brew install webclaw

# or, with Docker
docker run --rm ghcr.io/0xmassi/webclaw https://example.com

webclaw https://stripe.com --format markdown
webclaw https://docs.anthropic.com --format llm
webclaw https://example.com/blog/post --only-main-content

webclaw https://example.com \
  --include "article, main, .content" \
  --exclude "nav, footer, .sidebar, .ad"

webclaw https://docs.rust-lang.org --crawl --depth 2 --max-pages 50

npx create-webclaw

# or add this to your MCP client config:
{
  "mcpServers": {
    "webclaw": {
      "command": "npx",
      "args": ["-y", "@webclaw/mcp"]
    }
  }
}

webclaw https://example.com/pricing --format json > pricing-old.json
webclaw https://example.com/pricing --diff-with pricing-old.json

export WEBCLAW_API_KEY=wc_your_key

webclaw https://example.com --cloud

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it when your coding agent's built-in fetch returns raw HTML or a blocked page and you want it to get clean page content through MCP instead
• Reach for it when you need to crawl docs, help centers or blogs into markdown or compact LLM text for a RAG index
• Reach for it when you want to monitor competitor pricing pages, changelogs or docs by diffing snapshots from the command line
• Reach for it when you need messy pages turned into structured JSON for an automation, using a local Ollama model or your own LLM provider

// How webclaw compares

webclaw alongside other open-source web scraping & crawling tools AI/TLDR tracks, ranked by GitHub stars.