# TRAWL

> Self-hosted scraping engine with challenge handling, an HTTP API and an MCP server for agents

- Category: Data & Labeling / Web Scraping & Crawling
- Language: TypeScript
- Licence: AGPL-3.0
- Repository: https://github.com/germondai/trawl
- Website: https://trawl.germondai.com
- Canonical page: https://ai-tldr.dev/tools/trawl/

## In short

TRAWL is a self-hosted web scraping engine for applications and AI agents. It sits in the Web Scraping & Crawling category of the Data & Labeling tools and is open-source, written in TypeScript and licensed AGPL-3.0. To get started: start the container.

## Overview

TRAWL is a self-hosted web scraping engine for applications and AI agents. You give it a known URL and it returns the page — as HTML through its REST API, or as Markdown, extracted JSON records, a screenshot or browser diagnostics through its Model Context Protocol server. It does not search the web; the caller supplies the URL.

Its core job is getting past the walls that stop plain HTTP clients. A request starts as a cheap HTTP fetch and escalates through browser tiers only when needed, opening Camoufox (a hardened Firefox) to clear JavaScript and proof-of-work challenges such as Cloudflare, Anubis, Akamai, Imperva and DDoS-Guard, plus supported CAPTCHA widgets including Turnstile, reCAPTCHA v2, hCaptcha and GeeTest Slide. Once a site accepts a clearance, TRAWL caches it in Redis or memory so repeat requests fetch fresh content over HTTP without repeating the full solve. Solving is local by default; a 2Captcha fallback is optional.

The same engine is exposed several ways: a native /scrape API, a FlareSolverr-compatible /v1 endpoint that works as a drop-in replacement for clients such as Prowlarr and Jackett, a challenge-aware HTTP/HTTPS forward proxy, isolated browser sessions that keep cookies and storage between requests, and a local dashboard with request history and failure causes. It is written in TypeScript on Bun and Elysia, ships as Docker images for AMD64 and ARM64, and is AGPL-3.0 licensed.

## What it does

- Tiered execution: plain HTTP first, escalating to Camoufox browser tiers only when a challenge is detected
- Challenge handling for Cloudflare, Anubis, Akamai, Imperva, DDoS-Guard and supported AWS WAF and DataDome flows, plus Turnstile, reCAPTCHA v2, hCaptcha, GeeTest Slide, ALTCHA, Friendly Captcha and CAP widgets
- Clearance reuse through a Redis or in-memory cache, so repeat requests skip the full solve when the site accepts the saved session
- MCP server over Streamable HTTP with read-only read, scrape, extract, screenshot and inspect tools for AI clients
- Native /scrape API, FlareSolverr-compatible /v1 endpoint and an optional challenge-aware HTTP/HTTPS forward proxy
- Isolated named browser sessions plus a local metrics dashboard with tier outcomes and failure causes

## Getting started

TRAWL runs as a Docker container. The quick start from the README starts one browser with the MCP endpoint, an in-memory clearance cache and a persistent dashboard; keep the API and MCP endpoint on a trusted network, since MCP has no authentication.

1. Start the container

```bash
docker run -d --name trawl \
  -p 127.0.0.1:8191:8191 \
  --shm-size=1gb \
  -e SESSION_CACHE_DRIVER=memory \
  -e BROWSER_POOL_SIZE=1 \
  -e MCP_ENABLED=true \
  -e METRICS_DASHBOARD_ENABLED=true \
  -v trawl_metrics:/data/metrics \
  ghcr.io/germondai/trawl:1.9.3

curl http://localhost:8191/health
```

2. Scrape a page through the native API

   The response carries the HTML, cookies, the tier that won and timings. Setting solveEmbeddedCaptchas to false returns page content without attempting optional form widgets; access challenges are still cleared.

```bash
curl -X POST http://localhost:8191/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://nopecha.com/demo/cloudflare","solveEmbeddedCaptchas":false}'
```

3. Connect an AI client over MCP

   With MCP_ENABLED=true, add a remote MCP server using Streamable HTTP in your client. The exact configuration syntax belongs to the client; the tools are read, scrape, extract, screenshot and inspect.

```text
Name: trawl
Transport: Streamable HTTP
URL: http://localhost:8191/mcp
```

4. Open the local dashboard

   Visit http://localhost:8191/dashboard for request history, tier outcomes and failure causes. The mounted volume keeps the history across container recreation.

## When to use it

- Give an agent a self-hosted MCP tool that reads pages as Markdown even when they sit behind a browser challenge
- Replace FlareSolverr or Byparr behind Prowlarr or Jackett with a compatible /v1 endpoint
- Feed a scraping or RAG pipeline from protected pages without paying per request to a hosted unblocking API
- Keep logged-in browser state for a multi-step scrape using isolated named sessions