AnakinScraper

// Overview

AnakinScraper OSS is a self-hosted web scraping API built for feeding LLMs. You POST a URL and get back clean markdown with the boilerplate stripped, ready to drop into a RAG pipeline or an agent's context without preprocessing; with a Gemini key it can also return structured JSON extracted from the page.

Its distinguishing idea is a handler chain that degrades gracefully. A request first tries a plain local HTTP fetch, then an anti-detect browser (Camoufox, a hardened Firefox rather than headless Chrome), then any third-party scraping API you have plugged in. Most pages resolve on the free local handler, so paid APIs are only called for the small fraction that genuinely need them. Per-domain configs choose which handlers to use, set timeouts and headers, and define failure patterns — a CAPTCHA page, a missing required string — that automatically retry the job on the next handler.

Deployment is deliberately light: a single Go binary with in-memory job storage runs with no database at all, or Docker Compose brings up the server, the browser service and PostgreSQL for persistence. Proxy choice is learned per domain by Thompson sampling from real success and failure. A built-in React dashboard covers scraping, job history, domain configs and proxy scores. The project is AGPL-3.0 licensed.

// What it does

• Handler chain with automatic fallback: local HTTP fetch → anti-detect Camoufox browser → external API, so paid services are a last resort
• LLM-ready markdown output with boilerplate removal, plus optional Gemini-powered structured JSON extraction
• Per-domain configs for handler choice, timeouts, retries, custom headers, blocked domains and content validation
• Failure detection: define failure or required patterns per domain and the job retries with the next handler automatically
• Proxy auto-selection by Thompson sampling, learning the best proxy per domain from live success and failure
• Sync, async and batch endpoints, plus a built-in React dashboard for scraping, job tracking and proxy monitoring
• Zero-config mode — one Go binary, no database, no Redis, no message queue

// Getting started

The fastest path needs only Go 1.25+; jobs are held in memory and lost on restart. Set DATABASE_URL for persistence, or use Docker for the full stack including the browser service.

cd server && go run cmd/server/main.go

curl -s -X POST http://localhost:8080/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}' | jq .markdown

git clone https://github.com/Anakin-Inc/anakin.git
cd anakin
make up

curl -s -X POST http://localhost:8080/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "generateJson": true}' | jq .generatedJson

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Feed a RAG pipeline clean markdown from arbitrary public pages without writing per-site parsers
• Give an agent a scraping endpoint it can call, with the anti-detect browser reserved for pages that block plain fetches
• Cut third-party scraping-API spend by routing only the pages that fail locally to a paid handler
• Self-host crawling inside your own network when pages or credentials cannot go through a hosted service

// How AnakinScraper compares

AnakinScraper alongside other open-source web scraping & crawling tools AI/TLDR tracks, ranked by GitHub stars.