Scrapling

// Overview

Scrapling is an open-source Python framework for web scraping that handles everything from a single request to a full-scale crawl. Its parser learns from website changes and automatically relocates your elements when a page's structure updates, so selectors that would normally break keep working.

On top of the parser, Scrapling ships fetchers that send requests under the radar and can bypass anti-bot systems like Cloudflare's Turnstile out of the box. A Scrapy-like spider framework lets you scale up to concurrent, multi-session crawls with pause and resume, streaming results, and built-in proxy rotation, all in a few lines of Python.

// What it does

• Adaptive parser: mark elements with auto_save, then pass adaptive=True later to re-find them after a site's design changes.
• Stealthy fetchers (Fetcher, StealthyFetcher, DynamicFetcher) that spoof browser fingerprints and bypass Cloudflare Turnstile and similar anti-bot checks.
• Scrapy-like Spider API with start_urls, async parse callbacks, and Request/Response objects for full crawls.
• Concurrent crawling with configurable concurrency limits, per-domain throttling, download delays, and built-in proxy rotation.
• Checkpoint-based pause and resume: stop a crawl with Ctrl+C and restart to continue from where it left off.
• Streaming mode, optional robots.txt compliance, a development cache mode, and built-in JSON/JSONL export.

// Getting started

Scrapling requires Python 3.10 or higher. The base install includes only the parser engine; add the fetchers extra and run the install command to get browsers and stealth dependencies for the fetchers and spiders.

pip install scrapling

pip install "scrapling[fetchers]"
scrapling install

from scrapling.fetchers import StealthyFetcher

StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)
products = p.css('.product', auto_save=True)
products = p.css('.product', adaptive=True)

from scrapling.spiders import Spider, Response

class MySpider(Spider):
    name = "demo"
    start_urls = ["https://example.com/"]

    async def parse(self, response: Response):
        for item in response.css('.product'):
            yield {"title": item.css('h2::text').get()}

MySpider().start()

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Scraping data from sites that frequently redesign their pages, where a self-healing parser keeps your selectors working instead of breaking.
• Collecting data from sites protected by anti-bot systems such as Cloudflare Turnstile, using the stealthy fetchers.
• Running large concurrent crawls across many pages with proxy rotation, pause/resume, and streaming results for pipelines or dashboards.

// How Scrapling compares

Scrapling alongside other open-source web scraping & crawling tools AI/TLDR tracks, ranked by GitHub stars.