Miasma

// What it does

• Serves poisoned training data from a configurable `poison-source` (the poison fountain by default), with response caching that can be switched off
• Each response includes a configurable number of self-directing links (`link-count`) under your chosen `link-prefix`, so scrapers loop back through your proxy
• `max-depth` cuts scrapers loose after a set depth, letting you cap how much poison — and egress — each one costs you
• `max-in-flight` bounds concurrent requests and answers overflow with an immediate 429; `force-gzip` compresses every response to cut egress
• Optional per-User-Agent request metrics stored in a local SQLite file and served at a basic-auth-protected endpoint
• Configured by CLI flags or a YAML, TOML or JSON config file; HTML from the poison source is escaped by default

// Getting started

The README walks through trapping scrapers on a path called `/naughty-bots` behind an Nginx reverse proxy; the same result works with other setups.

cargo install miasma

# start with the default configuration
miasma

# list every configuration option
miasma --help

docker run --rm -p 9999:9999 austinweeks/miasma:latest \
    --link-prefix '/naughty-bots' \
    --max-in-flight 30

<a
  href="/naughty-bots/"
  style="display: none;"
  aria-hidden="true"
  tabindex="-1"
>
  Amazing high quality data here!
</a>

http {
  # Reserve 8MB memory for tracking user agents
  limit_req_zone $http_user_agent zone=miasma:8m rate=1r/s;

  server {
    location = /naughty-bots {
      port_in_redirect off;
      return 301 /naughty-bots/;
    }
    location /naughty-bots/ {
      # Rate limit via the 'miasma' zone with no queueing
      limit_req_status 429;
      limit_req zone=miasma burst=5 nodelay;

      # Proxy requests to Miasma
      proxy_pass http://localhost:9855/;
    }
  }
}

miasma --link-prefix '/naughty-bots' -p 9855 -c 50 --force-gzip --link-count 5 --max-depth 8

User-agent: *
Disallow: /naughty-bots

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Make AI training crawlers that ignore robots.txt pay for scraping a personal site or blog by feeding them poisoned pages instead of your content
• Identify which bots hit a site hardest using Miasma's per-User-Agent request metrics
• Add a scraper tarpit to an existing Rust web server by using Miasma as a library
• Run a low-memory trap alongside an Nginx (or other reverse-proxy) deployment with bounded egress via max-depth and forced gzip

// How Miasma compares

Miasma alongside other open-source web scraping & crawling tools AI/TLDR tracks, ranked by GitHub stars.