How to Give Agents Web Search and Browsing

// In plain English

A large language model's knowledge is frozen at its training cutoff. Ask it about something that happened last week, and it either confesses ignorance or confidently makes something up. Web access is the standard fix: you give the agent a tool that can fetch live information from the internet, so it reads before it answers rather than guessing from stale memory.

The analogy that works best is a researcher versus a librarian. A librarian (the base model) knows a lot from years of reading but can only recall what was in the books when she last left the building. A researcher (an agent with web tools) steps outside, checks today's news, fetches the right source, and comes back with a verified answer. Web access turns the librarian into the researcher.

There are three distinct approaches, each involving a different trade-off between speed, cost, and capability:

• Search APIs — send a query, get back ranked snippets and URLs. Fast, cheap, no JavaScript required.
• Page-fetch tools — give the agent a URL, get back the page's clean text. One level deeper than search.
• Full browser automation — launch a real browser, click buttons, fill forms, handle login walls, render JavaScript-heavy pages.

// Why it matters

Without web access, an agent's factual claims are bounded by its training data — which may be months or years old, and which never contained private or niche information in the first place. The consequences are real:

• Outdated pricing or product info. A model trained before a major API pricing change will quote the wrong numbers with full confidence.
• Hallucinated citations. Asked to cite sources it can't access, a model invents plausible-sounding URLs that return 404.
• Gaps on niche topics. Documentation for an obscure library, a regional news story, a competitor's recent product launch — the training corpus simply may not include it.
• Inability to complete real tasks. Booking a flight, reading a live dashboard, checking order status — these require the current state of a live system, not a cached copy.

Beyond fixing staleness, web access changes what kinds of tasks are possible. An agent that can search and fetch can do original research, monitor for changes, and synthesize information across multiple live sources — tasks that are fundamentally impossible with static knowledge alone.

For builders, the practical question is not whether to add web access but which tier to add. The right answer depends on latency tolerance, whether pages use JavaScript, whether logins are involved, and how much of the page the agent actually needs.

// How the three approaches work

The three approaches sit on a complexity spectrum. You start with the simplest that meets the task's requirements and escalate only when you need to.

Tier 1: Search APIs

A search API accepts a natural-language or keyword query and returns a list of results: each result has a title, a snippet (usually 100–300 words), and a URL. The agent reads the snippets and decides whether it needs to fetch any of the full pages. This is the cheapest and fastest tier — most search API calls resolve in under 200ms and cost fractions of a cent.

The leading options for agent use as of 2026 differ in what kind of relevance they optimize for. Tavily was built specifically for AI agents: it returns pre-cleaned, snippet-optimized results and has a 180ms p50 latency with a free tier and ~$0.008 per query on paid plans. Exa uses neural (embedding-based) search over its own index — it finds conceptually related pages rather than keyword matches, which is valuable for open-ended research queries. Exa Instant delivers results in under 150ms. Brave Search operates its own independent web index (not a Google or Bing reskin), is privacy-preserving, and benchmarks showed it with the lowest average latency in independent tests at around 669ms. Its Data for AI tier starts at $5/month for 2,000 queries. Serper wraps Google results and is often the cheapest option at scale for SERP-style queries.

Tier 2: Page-fetch tools

Search snippets are trimmed and summarized — they may not contain the specific sentence the agent needs. A page-fetch tool takes a URL and returns the full page content, cleaned of navigation menus, ads, and boilerplate, and converted to plain text or Markdown that the LLM can read efficiently.

Jina Reader (r.jina.ai) is the simplest implementation: prepend https://r.jina.ai/ to any URL and the API returns clean Markdown. It is free for low-volume use with no API key; authenticated requests get a higher free-tier allowance of 1 million tokens, and paid plans start around $20/month. Firecrawl is a more full-featured scraping API that handles JavaScript-rendered pages, supports crawling entire sites, and converts output to AI-ready Markdown or structured JSON. Its free tier provides 1,000 credits/month, with paid plans starting at $16/month for 3,000 credits. Jina also exposes s.jina.ai as a combined search-and-fetch endpoint that retrieves the top 5 results and fetches each page in one call.

Tier 3: Full browser automation

Some tasks cannot be solved with a URL lookup: logging into a service, clicking through a multi-step form, or reading data from a JavaScript-heavy single-page app that renders nothing in static HTML. These require a real browser that executes JavaScript, manages cookies, and can interact with page elements.

Playwright is the de-facto standard for AI agent browser control. Its accessibility tree feature is the key reason: instead of dumping raw HTML (which can be tens of thousands of tokens), Playwright exposes a structured, semantic representation of interactive elements with their roles, names, and states. This is far cheaper for an LLM to read and reason about than parsing DOM markup. Playwright also supports three browser engines (Chromium, Firefox, WebKit), provides built-in auto-waiting so the agent doesn't race with page load, and has an official MCP server for direct agent integration. Puppeteer is the older alternative — Chrome-only, lower-level, and less suited to the parallel-session patterns agents often need.

// Choosing the right approach

The decision tree is driven by four questions: Does the task need current information? Does it require reading full page content? Is the page JavaScript-rendered? Does it require user interaction (logins, forms)?

The combined pattern: search, then fetch

The most common production pattern for research agents combines tiers one and two. The agent calls a search API to get 5–10 candidate URLs, reads the snippets to filter to the most promising 2–3, then fetches those full pages. This gives the agent the content depth of a page-fetch tool while paying the cost and latency of a search API for the filtering step.

from langchain_tavily import TavilySearch
from langchain_community.document_loaders import WebBaseLoader

# Step 1: search for candidate URLs
search = TavilySearch(max_results=5)
results = search.invoke("Claude 4 Sonnet pricing per million tokens")

# Step 2: fetch full content of the top result
top_url = results[0]["url"]
loader = WebBaseLoader(top_url)
doc = loader.load()
# doc[0].page_content now contains the full page text

Jina also bundles this pattern into a single endpoint: https://s.jina.ai/<query> searches the web and returns the full text of the top 5 pages in one call, eliminating the need to chain two separate tool calls.

// Context and cost pitfalls

Adding web access to an agent is straightforward technically, but there are several ways it goes wrong in production.

Raw HTML floods the context window

Fetching a page and passing raw HTML to the model is the most common mistake. A typical news article's HTML is 50,000–150,000 characters — most of it navigation menus, ad slots, script tags, and tracking pixels. Even with a 200k-token context window, raw HTML from three pages can consume the entire budget and leave no room for reasoning. Always convert to clean Markdown or plain text before passing to the model. Jina Reader and Firecrawl both do this automatically.

Search results compound across turns

In a multi-turn research agent, the model's context accumulates every prior search result and page fetch. By turn 10, the model may be re-reading 80,000 tokens of prior web content on every single step. This is expensive (you pay for input tokens on every request) and degrades reasoning quality as the model struggles to separate fresh observations from earlier context. Strategies to mitigate this include summarizing fetched pages before storing them, evicting old tool results from context after they have been used, and capping the number of fetch results stored at any one time.

Snippet hallucination and misattribution

A search API returns snippets — short extracts chosen by the search engine's own algorithm, not by you. The model may misread a truncated snippet, confuse which snippet came from which URL, or stitch two snippets together into a claim that neither source actually made. The fix is to always fetch and read the full page before making a specific factual claim, and to include the source URL in the agent's output so it can be verified.

Rate limits and retry costs

Agentic loops that call search APIs on every step can hit rate limits quickly on free or low-tier plans. Budget your search calls explicitly: tell the agent in the system prompt how many searches it is allowed per task, or track calls programmatically and refuse further calls once the budget is exhausted. A research agent allowed to search without limits can rack up hundreds of API calls for a single query.

// Going deeper

Neural vs. keyword search is a real architectural choice. Exa's embedding-based index finds pages that are conceptually related to the query even when the exact words don't appear. This is better for open-ended research ("recent breakthroughs in protein folding") but sometimes over-retrieves tangentially related content. Tavily and Brave use more traditional relevance signals and tend to be sharper on factual lookups ("current price of model X"). Many production agents run both and take the union of results.

Exa Deep is an agentic search endpoint that internally does query expansion, parallel sub-searches, and LLM-assisted re-ranking before returning structured, cited results. It is slower (it is designed for complex research tasks, not quick lookups) but can replace an entire multi-step search-and-summarize loop with a single API call. If your agent spends multiple turns doing research, Exa Deep is worth benchmarking as a single-call replacement.

The Playwright MCP server is the standard way to give a Claude or other MCP-compatible agent full browser control without writing custom integration code. The server exposes browser_navigate, browser_click, browser_fill_form, browser_snapshot, and related tools. The snapshot tool is the critical one: it returns the accessibility tree of the current page, not a screenshot or raw DOM, which keeps the token cost manageable. Combining browser_navigate + browser_snapshot + browser_click covers the majority of real-world browser automation tasks.

Caching search results is an underused optimization. If your agent is likely to search for the same or similar queries across multiple tasks (common in customer support or research assistants), caching results with a TTL of a few hours can dramatically reduce both cost and latency. Most search APIs are deterministic enough that the same query returns the same top results within a short window.

Verification is a first-class concern. Web search gives agents the ability to cite sources — but also the ability to cite sources incorrectly. Best practice is to have the agent quote the exact sentence from the fetched page that supports its claim, and to include the source URL in its response. This makes it easy for a human reviewer (or a second agent acting as a verifier) to spot misattribution quickly. The zero-hallucination standard that applies to training-time knowledge applies equally to web-fetched content: if the page doesn't say it, the agent shouldn't claim it does.

MCP as the universal connector means that once you package your search or fetch capability as an MCP server with well-described tools, it becomes available to any MCP-compatible agent or IDE without custom integration work. Tavily, Firecrawl, Brave, and Exa all publish official MCP servers. This is now the preferred way to expose web tools to agents — you get standardized tool discovery, consistent error handling, and compatibility with the growing ecosystem of MCP-aware clients.