What is TRAWL?
TRAWL is a self-hosted web scraping engine for applications and AI agents. It sits in the Web Scraping & Crawling category of the Data & Labeling tools and is open-source, written in TypeScript and licensed AGPL-3.0. To get started: start the container.
Overview
TRAWL is a self-hosted web scraping engine for applications and AI agents. You give it a known URL and it returns the page — as HTML through its REST API, or as Markdown, extracted JSON records, a screenshot or browser diagnostics through its Model Context Protocol server. It does not search the web; the caller supplies the URL.
Its core job is getting past the walls that stop plain HTTP clients. A request starts as a cheap HTTP fetch and escalates through browser tiers only when needed, opening Camoufox (a hardened Firefox) to clear JavaScript and proof-of-work challenges such as Cloudflare, Anubis, Akamai, Imperva and DDoS-Guard, plus supported CAPTCHA widgets including Turnstile, reCAPTCHA v2, hCaptcha and GeeTest Slide. Once a site accepts a clearance, TRAWL caches it in Redis or memory so repeat requests fetch fresh content over HTTP without repeating the full solve. Solving is local by default; a 2Captcha fallback is optional.
The same engine is exposed several ways: a native /scrape API, a FlareSolverr-compatible /v1 endpoint that works as a drop-in replacement for clients such as Prowlarr and Jackett, a challenge-aware HTTP/HTTPS forward proxy, isolated browser sessions that keep cookies and storage between requests, and a local dashboard with request history and failure causes. It is written in TypeScript on Bun and Elysia, ships as Docker images for AMD64 and ARM64, and is AGPL-3.0 licensed.
What it does
- Tiered execution: plain HTTP first, escalating to Camoufox browser tiers only when a challenge is detected
- Challenge handling for Cloudflare, Anubis, Akamai, Imperva, DDoS-Guard and supported AWS WAF and DataDome flows, plus Turnstile, reCAPTCHA v2, hCaptcha, GeeTest Slide, ALTCHA, Friendly Captcha and CAP widgets
- Clearance reuse through a Redis or in-memory cache, so repeat requests skip the full solve when the site accepts the saved session
- MCP server over Streamable HTTP with read-only read, scrape, extract, screenshot and inspect tools for AI clients
- Native /scrape API, FlareSolverr-compatible /v1 endpoint and an optional challenge-aware HTTP/HTTPS forward proxy
- Isolated named browser sessions plus a local metrics dashboard with tier outcomes and failure causes
Getting started
TRAWL runs as a Docker container. The quick start from the README starts one browser with the MCP endpoint, an in-memory clearance cache and a persistent dashboard; keep the API and MCP endpoint on a trusted network, since MCP has no authentication.
Start the container
docker run -d --name trawl \
-p 127.0.0.1:8191:8191 \
--shm-size=1gb \
-e SESSION_CACHE_DRIVER=memory \
-e BROWSER_POOL_SIZE=1 \
-e MCP_ENABLED=true \
-e METRICS_DASHBOARD_ENABLED=true \
-v trawl_metrics:/data/metrics \
ghcr.io/germondai/trawl:1.9.3
curl http://localhost:8191/healthScrape a page through the native API
The response carries the HTML, cookies, the tier that won and timings. Setting solveEmbeddedCaptchas to false returns page content without attempting optional form widgets; access challenges are still cleared.
curl -X POST http://localhost:8191/scrape \
-H 'Content-Type: application/json' \
-d '{"url":"https://nopecha.com/demo/cloudflare","solveEmbeddedCaptchas":false}'Connect an AI client over MCP
With MCP_ENABLED=true, add a remote MCP server using Streamable HTTP in your client. The exact configuration syntax belongs to the client; the tools are read, scrape, extract, screenshot and inspect.
Name: trawl
Transport: Streamable HTTP
URL: http://localhost:8191/mcp
Open the local dashboard
Visit http://localhost:8191/dashboard for request history, tier outcomes and failure causes. The mounted volume keeps the history across container recreation.
Commands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Give an agent a self-hosted MCP tool that reads pages as Markdown even when they sit behind a browser challenge
- Replace FlareSolverr or Byparr behind Prowlarr or Jackett with a compatible /v1 endpoint
- Feed a scraping or RAG pipeline from protected pages without paying per request to a hosted unblocking API
- Keep logged-in browser state for a multi-step scrape using isolated named sessions
How TRAWL compares
TRAWL alongside other open-source web scraping & crawling tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| Firecrawl | ★ 198k | A crawling service and API that converts whole websites into clean Markdown or structured JSON ready for LLMs. |
| Agent Reach | ★ 108k | A capability layer that installs, health-checks and routes readers so a CLI agent can read and search web pages, YouTube, RSS, GitHub, Reddit, Twitter/X and more without API fees. |
| Scrapling | ★ 92.6k | A Python web scraping framework whose parser relocates your elements when pages change, with stealthy fetchers and a Scrapy-like spider engine for full crawls. |
| Crawl4AI | ★ 86.8k | A local-first Python web crawler that turns pages into clean Markdown for use in RAG and LLM pipelines. |
| Scrapy | ★ 65.2k | A mature Python framework for writing fast spiders that crawl websites and extract structured data at scale. |
| changedetection.io | ★ 35.5k | A self-hosted web page change monitor with XPath/CSS/jq filters, browser steps and restock alerts, which can route every diff through an LLM to suppress noise. |
| ScrapeGraphAI | ★ 32.4k | A Python library that uses LLMs and a graph pipeline to extract data from pages based on natural-language prompts. |
| TRAWL | ★ 1.1k | Self-hosted scraping engine with challenge handling, an HTTP API and an MCP server for agents |