AI/TLDR

Agent Reach

A pre-wired web-access layer that gives a CLI agent readers for the sites it can't reach

Web Scraping & CrawlingOpen source
Language
Python
License
MIT

Overview

Agent Reach is an MIT-licensed capability layer that gives a CLI AI agent access to the open internet. Rather than being one more scraper, it is the selection-and-plumbing layer above them: for each platform it picks a working method, installs it, health-checks it, and routes requests to a primary backend with fallbacks behind it — so when a site changes or an endpoint is deprecated, the agent keeps working.

Coverage splits into two tiers. Web pages, YouTube, RSS, GitHub, Bilibili and semantic web search work with zero configuration and no API keys. Twitter/X, Reddit, Facebook, Instagram and LinkedIn need you to be logged in, and Xiaohongshu, V2EX and Xueqiu sit in between. Credentials are kept locally in `~/.agent-reach/config.yaml` and never leave the machine, and the underlying tools it installs are themselves open source, so there are no per-call API fees.

Installation is deliberately agent-driven: you hand your agent a link to the project's install document and it does the rest, which is why the project describes itself as agent-agnostic — it works with Claude Code, OpenClaw, Cursor, Windsurf or any CLI-capable agent. A built-in `agent-reach doctor` command reports which channels currently work and why the others don't, and Exa semantic search is wired in through mcporter as a free, key-less MCP backend.

What it does

  • One capability layer that selects, installs and health-checks a working reader for each platform
  • Zero-config access to web pages, YouTube, RSS, GitHub, Bilibili and semantic web search
  • Login-based access to Twitter/X, Reddit, Facebook, Instagram, LinkedIn, Xiaohongshu, V2EX and Xueqiu
  • Multi-backend routing — every platform has a primary plus fallbacks, so a site change doesn't break the agent
  • `agent-reach doctor` self-diagnostic that reports which channels work and why the rest don't
  • Credentials stay local in `~/.agent-reach/config.yaml`; the underlying tools are open source, so there are no API fees
  • Agent-agnostic — works with Claude Code, OpenClaw, Cursor, Windsurf or any CLI-capable agent

Getting started

Agent Reach is installed by the agent itself: you paste one instruction pointing at the project's install document, and the agent performs the setup.

Ask your agent to install it

Paste this into Claude Code, OpenClaw, Cursor or any CLI-capable agent. It fetches the install document and works through the setup itself.

texttext
Install Agent Reach for me: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md

Check which channels are live

The doctor command reports every channel's status and, when one is down, why.

bashbash
agent-reach doctor

Use the zero-config channels straight away

Web pages, YouTube, RSS, GitHub, Bilibili and semantic search need no keys and no login — just ask the agent for the content.

Add a login-gated platform

For Twitter/X, Reddit, Facebook, Instagram, LinkedIn or Xiaohongshu, ask the agent to configure that platform; it walks you through the login and stores the result locally in ~/.agent-reach/config.yaml.

Optionally install system-wide

By default the installer only checks the environment without modifying it. The --system flag performs a system-wide setup instead.

bashbash
agent-reach install --system

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

When to use it

  • Give a coding or research agent read access to social platforms it otherwise cannot open
  • Pull YouTube transcripts, RSS feeds and GitHub content into an agent workflow without writing a scraper
  • Run web research from an agent without paying per-call search-API fees
  • Keep an agent's web access working when a platform changes, via the fallback backends and doctor check

How Agent Reach compares

Agent Reach alongside other open-source web scraping & crawling tools AI/TLDR tracks, ranked by GitHub stars.

ToolStarsWhat it does
Firecrawl★ 183kA crawling service and API that converts whole websites into clean Markdown or structured JSON ready for LLMs.
Agent Reach★ 84.3kA pre-wired web-access layer that gives a CLI agent readers for the sites it can't reach
Crawl4AI★ 84kA local-first Python web crawler that turns pages into clean Markdown for use in RAG and LLM pipelines.
Scrapling★ 82.8kA Python web scraping framework whose parser relocates your elements when pages change, with stealthy fetchers and a Scrapy-like spider engine for full crawls.
Scrapy★ 64.4kA mature Python framework for writing fast spiders that crawl websites and extract structured data at scale.
changedetection.io★ 34.4kA self-hosted web page change monitor with XPath/CSS/jq filters, browser steps and restock alerts, which can route every diff through an LLM to suppress noise.
ScrapeGraphAI★ 31.2kA Python library that uses LLMs and a graph pipeline to extract data from pages based on natural-language prompts.
Crawlee★ 25.9kA Node.js/TypeScript scraping library with proxy rotation and browser fingerprinting for building reliable crawlers.