Overview
Agent Reach is an MIT-licensed capability layer that gives a CLI AI agent access to the open internet. Rather than being one more scraper, it is the selection-and-plumbing layer above them: for each platform it picks a working method, installs it, health-checks it, and routes requests to a primary backend with fallbacks behind it — so when a site changes or an endpoint is deprecated, the agent keeps working.
Coverage splits into two tiers. Web pages, YouTube, RSS, GitHub, Bilibili and semantic web search work with zero configuration and no API keys. Twitter/X, Reddit, Facebook, Instagram and LinkedIn need you to be logged in, and Xiaohongshu, V2EX and Xueqiu sit in between. Credentials are kept locally in `~/.agent-reach/config.yaml` and never leave the machine, and the underlying tools it installs are themselves open source, so there are no per-call API fees.
Installation is deliberately agent-driven: you hand your agent a link to the project's install document and it does the rest, which is why the project describes itself as agent-agnostic — it works with Claude Code, OpenClaw, Cursor, Windsurf or any CLI-capable agent. A built-in `agent-reach doctor` command reports which channels currently work and why the others don't, and Exa semantic search is wired in through mcporter as a free, key-less MCP backend.
What it does
- One capability layer that selects, installs and health-checks a working reader for each platform
- Zero-config access to web pages, YouTube, RSS, GitHub, Bilibili and semantic web search
- Login-based access to Twitter/X, Reddit, Facebook, Instagram, LinkedIn, Xiaohongshu, V2EX and Xueqiu
- Multi-backend routing — every platform has a primary plus fallbacks, so a site change doesn't break the agent
- `agent-reach doctor` self-diagnostic that reports which channels work and why the rest don't
- Credentials stay local in `~/.agent-reach/config.yaml`; the underlying tools are open source, so there are no API fees
- Agent-agnostic — works with Claude Code, OpenClaw, Cursor, Windsurf or any CLI-capable agent
Getting started
Agent Reach is installed by the agent itself: you paste one instruction pointing at the project's install document, and the agent performs the setup.
Ask your agent to install it
Paste this into Claude Code, OpenClaw, Cursor or any CLI-capable agent. It fetches the install document and works through the setup itself.
Install Agent Reach for me: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.mdCheck which channels are live
The doctor command reports every channel's status and, when one is down, why.
agent-reach doctorUse the zero-config channels straight away
Web pages, YouTube, RSS, GitHub, Bilibili and semantic search need no keys and no login — just ask the agent for the content.
Add a login-gated platform
For Twitter/X, Reddit, Facebook, Instagram, LinkedIn or Xiaohongshu, ask the agent to configure that platform; it walks you through the login and stores the result locally in ~/.agent-reach/config.yaml.
Optionally install system-wide
By default the installer only checks the environment without modifying it. The --system flag performs a system-wide setup instead.
agent-reach install --systemCommands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Give a coding or research agent read access to social platforms it otherwise cannot open
- Pull YouTube transcripts, RSS feeds and GitHub content into an agent workflow without writing a scraper
- Run web research from an agent without paying per-call search-API fees
- Keep an agent's web access working when a platform changes, via the fallback backends and doctor check
How Agent Reach compares
Agent Reach alongside other open-source web scraping & crawling tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| Firecrawl | ★ 183k | A crawling service and API that converts whole websites into clean Markdown or structured JSON ready for LLMs. |
| Agent Reach | ★ 84.3k | A pre-wired web-access layer that gives a CLI agent readers for the sites it can't reach |
| Crawl4AI | ★ 84k | A local-first Python web crawler that turns pages into clean Markdown for use in RAG and LLM pipelines. |
| Scrapling | ★ 82.8k | A Python web scraping framework whose parser relocates your elements when pages change, with stealthy fetchers and a Scrapy-like spider engine for full crawls. |
| Scrapy | ★ 64.4k | A mature Python framework for writing fast spiders that crawl websites and extract structured data at scale. |
| changedetection.io | ★ 34.4k | A self-hosted web page change monitor with XPath/CSS/jq filters, browser steps and restock alerts, which can route every diff through an LLM to suppress noise. |
| ScrapeGraphAI | ★ 31.2k | A Python library that uses LLMs and a graph pipeline to extract data from pages based on natural-language prompts. |
| Crawlee | ★ 25.9k | A Node.js/TypeScript scraping library with proxy rotation and browser fingerprinting for building reliable crawlers. |