Web Fetch

Feature Definition

Web fetch is the direct URL retrieval primitive.

It takes a known URL and returns content the model can reason over.

In agent workflows, it commonly follows search:

1. search finds candidate pages
2. fetch retrieves exact target pages
3. read/summarize pipelines consume retrieved content

Fetch vs Search

Search answers: “What URLs are relevant?”

Fetch answers: “What is at this URL right now?”

They should be separate tools.

Combining them creates ambiguous behavior and harder permission policy.

Why It Is Hard

Real-world fetch handling is messy:

• websites block automated user agents
• content types vary (HTML, markdown, JSON, image, PDF)
• huge responses can destroy context budgets
• HTML must often be converted before model use
• URL handling must be hardened against unsafe targets

So a production fetch tool needs:

• strict URL validation
• timeout and response-size bounds
• content-type aware conversion
• explicit failure modes
• predictable output shape

---

See Also

• Web Search for query/ranking behavior before fetch.
• Approval Flow and Sandbox Modes for external-network permission policy.
• Token Budgeting for response-size limits and context-pressure tradeoffs.

---

Aider Implementation

Aider at commit b9050e1d5faf8096eae7a46a9ecc05a86231384b supports URL fetch through /web, not through a model-callable JSON tool.

Entry Point: /web

aider/commands.py::cmd_web:

1. validates URL argument presence
2. lazily initializes Scraper
3. calls scraper.scrape(url)
4. prefixes output with Here is the content of <url>:
5. injects content into chat history

URL Auto-Ingestion from User Input

BaseCoder.check_for_urls() in aider/coders/base_coder.py detects URLs in user messages and asks whether to add them.

If approved, it calls cmd_web(url, return_content=True) and appends fetched content to the same user turn.

Transport Selection (Playwright vs HTTPX)

aider/scrape.py::Scraper.scrape chooses transport:

• Playwright path when available
• HTTPX fallback otherwise

Playwright path (scrape_with_playwright):

• launches Chromium
• sets custom UA
• page.goto(..., wait_until="networkidle", timeout=5000)
• reads rendered HTML via page.content()

HTTPX path (scrape_with_httpx):

• follow_redirects=True
• basic UA header
• client.get(url)
• returns text + MIME type

HTML Detection and Conversion Pipeline

If MIME is HTML or body heuristically looks like HTML, Aider converts content:

1. try_pandoc() ensures pandoc availability (downloads if missing)
2. html_to_markdown() parses with BeautifulSoup
3. slimdown_html() removes heavy/noisy elements (svg, img, data URLs, most attrs)
4. pypandoc.convert_text(..., "markdown", format="html")

If conversion fails, Aider returns reduced HTML.

Output Injection

Fetched result is inserted as a user message payload, so subsequent model turns treat it as normal chat context.

Aider does not enforce a strict byte cap inside this fetch path.

---

Codex Implementation

Codex at commit 4ab44e2c5cc54ed47e47a6729dfd8aa5a3dc2476 does not ship a dedicated web_fetch tool in its core tool registry.

No Dedicated web_fetch Tool

codex-rs/core/src/tools/spec.rs registers built-in web capability as web_search, controlled by web_search_mode.

There is no first-party function tool that takes { url, format } and returns converted page content.

What Codex Does Provide

Codex consumes Responses API web_search_call actions:

• search
• open_page
• find_in_page

in codex-rs/protocol/src/models.rs.

These are provider-native web-search actions, not a separate fetch tool contract managed by Codex runtime.

Practical Implication

For explicit URL fetch behavior, Codex deployments usually rely on:

• provider-native web search actions, or
• external MCP tools that implement fetch/read

This keeps core Codex runtime simpler, but moves URL-fetch semantics to provider/MCP layers.

Codex also explicitly disables web search during review tasks (core/src/codex.rs review flow), so review sessions should not assume any web retrieval.

---

OpenCode Implementation

OpenCode at commit 7ed449974864361bad2c1f1405769fd2c2fcdf42 implements a dedicated webfetch tool in packages/opencode/src/tool/webfetch.ts.

Tool Contract (webfetch)

Parameters:

• url: string
• format: "text" | "markdown" | "html" (default markdown)
• timeout (seconds, optional, max 120)

Tool description lives in webfetch.txt.

Permission Gate

Before network call, tool requests permission via:

• permission: "webfetch"
• patterns: [url]
• metadata containing url/format/timeout

HTTP Request Construction

Execution validates URL prefix:

• must start with http:// or https://

It builds format-specific Accept headers with quality weights (q= values), plus browser-like UA and language headers.

If the first response looks like Cloudflare bot challenge (403 + cf-mitigated: challenge), OpenCode retries with User-Agent: opencode.

Timeout and Size Limits

abortAfterAny() enforces timeout:

• default 30s
• max 120s

Size bounds:

• MAX_RESPONSE_SIZE = 5 MB
• checks both content-length header and actual downloaded byte length

Oversized responses are rejected.

Content-Type Dispatch

After download, tool branches on MIME:

• image/* (except SVG variants) -> base64 attachment
• HTML -> convert depending on requested format
• non-HTML text -> return as-is

For images, tool returns attachment metadata and data URL.

HTML to Markdown/Text Conversion

Markdown path:

• uses turndown
• strips script, style, meta, link

Text path:

• uses HTMLRewriter
• skips content inside script/style/noscript/iframe/object/embed

HTML path:

• returns raw HTML string

This gives explicit caller control over conversion fidelity vs verbosity.

---

Pitfalls & Hard Lessons

Charset and Encoding Drift

Many pages declare wrong or missing charsets.

Naive UTF-8 decoding can produce corrupted text, especially for legacy docs.

Bot Mitigation and Anti-Automation

Cloudflare and similar systems can block scripted fetches.

Retrying with different UA helps sometimes, but not reliably.

HTML-to-Markdown Is Lossy

Conversion drops layout, scripts, and interactive state.

Great for LLM readability.

Bad for exact visual semantics.

Unbounded Fetches Destroy Context Budgets

Without hard byte and token limits, a single fetch can consume most of a turn budget.

Hard caps must exist at tool boundary, not just later in prompt assembly.

URL Safety Requires Explicit Policy

Fetch tools are network egress.

Without host/IP policy checks, they can become SSRF primitives against internal infrastructure.

---

OpenOxide Blueprint

Tool Contract

Start with explicit schema:

• url (required)
• format (markdown default)
• timeout_ms (bounded)
• optional max_bytes (bounded by global cap)

Secure Fetch Pipeline

Implement a staged pipeline:

1. parse and normalize URL
2. enforce scheme allowlist (http, https)
3. resolve host and deny private/link-local/loopback ranges
4. enforce domain policy allow/deny lists
5. perform HTTP GET with redirect and timeout rules
6. stream body with byte cap enforcement

Conversion Strategy

Dispatch by MIME:

• text/html -> readability extraction + markdown conversion
• text/markdown / text/plain -> pass-through
• application/json -> preserve pretty text
• media/PDF -> attachment path

Keep conversion modules separate from transport so policies can reuse transport without conversion.

Size and Token Budget Controls

Enforce at least three limits:

• raw response byte cap
• converted text character cap
• model-injected token cap

If truncated, return deterministic metadata:

• truncated: true
• original_bytes
• returned_bytes
• optional artifact path for full output

Binary/Media Handling

For images/PDF, return structured attachments instead of forcing text conversion.

For unsupported binary types, return MIME and size metadata with a clear error.

Crates

• reqwest for HTTP transport
• url and ipnet for URL/IP validation
• encoding_rs for robust text decoding
• scraper or kuchiki for HTML parsing
• html2md or htmd for markdown conversion
• serde / serde_json for structured metadata