Web Search

Feature Definition

Web search is the external discovery primitive for an agent.

When the repository itself cannot answer a question, search is how the model reaches current documentation, upstream issue threads, release notes, and standards updates.

In coding systems, search is usually invoked for:

version-specific migration guidance
API behavior validation against canonical docs
latest security advisories and CVEs
ecosystem comparisons before dependency choice
operational troubleshooting snippets

Search is distinct from URL fetch.

Search asks:

“Which sources should I read?”

Fetch asks:

“What does this URL contain?”

Merging these into one tool creates blurry semantics, worse policy controls, and weaker observability.

Why It Is Hard

The naive implementation is trivial:

query -> provider -> return text.

The production implementation is not.

Hard problems:

query construction under underspecified user intent
freshness requirements for “latest/current/today”
ranking quality variance across engines
citation fidelity after model summarization
payload normalization across heterogeneous APIs
permission and compliance constraints
cost and latency control under retries/fallbacks

Search Lifecycle in Agent Systems

A robust search lifecycle usually has six phases.

intent extraction
query planning
provider execution
result normalization
context packaging
citation emission

Most failures happen in phase 2 or phase 4.

Phase 2 failures:

query too broad
missing version/year qualifiers
no domain scoping when needed

Phase 4 failures:

dropping key metadata
collapsing source diversity into one blob
losing URL/title boundaries

Query Construction Is a First-Class System

Good search quality is mostly query quality.

A coding agent should not rely on the model “figuring out a good query” on every turn.

It needs deterministic query policy:

if user asks for “latest”, force year and recency hints
if task is framework API usage, boost official docs domains
if task is operational bug, include exact error string variant

Without explicit query policy, search output quality swings wildly by prompt wording.

Result Contracts Matter

Search output must be machine-usable, not just human-readable.

At minimum, each result should preserve:

title
URL
snippet/abstract
rank
source provider
retrieval timestamp

Dropping these fields makes post-hoc debugging and citation tracing expensive.

Interaction with Token Budgeting

Search can silently become one of the largest token contributors in a turn.

If the tool returns large unbounded blobs, it competes directly with:

repository context
tool outputs
final answer budget

So search tools need independent output limits, not just global prompt truncation later.

Aider Implementation

Aider at commit b9050e1d5faf8096eae7a46a9ecc05a86231384b does not provide a model-callable web search tool.

No Model-Callable Web Search Tool

Aider has no JSON/function-call contract for web search.

There is no tool where the model sends a query and receives structured ranked results.

This is an intentional architecture difference.

Aider centers on:

local repo context
user-directed file inclusion
edit-format pipelines

not autonomous external retrieval orchestration.

What Exists: `/web` URL Scraping

Aider includes /web command support in aider/commands.py (cmd_web).

That command:

accepts a concrete URL
scrapes content via aider/scrape.py
injects fetched content into chat

This is fetch/scrape, not indexed search.

No provider ranking, no query planning, no multi-result normalization.

URL Auto-Detection in User Input

BaseCoder.check_for_urls() in aider/coders/base_coder.py scans user input with regex, asks whether each URL should be added, and if approved, appends content from cmd_web(url, return_content=True).

This is useful ergonomically, but still URL-driven, not query-driven.

Scrape Transport Path (Context for Search Gap)

aider/scrape.py shows Aider’s web capability focus.

Transport selection:

Playwright path when available
HTTPX fallback otherwise

Playwright path:

launches Chromium
applies custom UA
navigates page
captures rendered HTML

HTTPX path:

performs direct GET
follows redirects
returns textual body + MIME hint

For HTML, Aider runs conversion pipeline:

HTML detection heuristic (looks_like_html)
optional pandoc setup (try_pandoc)
soup cleanup (slimdown_html)
markdown conversion (pypandoc.convert_text)

This pipeline optimizes “get URL content into prompt” not “search the web by intent”.

Practical Consequences

Because Aider has no search tool contract:

the model cannot autonomously gather candidate sources
search delegation depends on user workflow
no structured result schema exists for ranking/citations
no search-specific permission category exists

For OpenOxide docs, this should be documented as a deliberate gap, not framed as a defect.

Aider is optimized for different tradeoffs.

Lessons from Aider for OpenOxide

Even without search, Aider contributes useful patterns:

URL auto-detection with user approval gates
scrape fallback between browser and raw HTTP
HTML slimming before conversion

These patterns are reusable inside a fetch layer, while keeping search orchestration separate.

Codex Implementation

Codex at commit 4ab44e2c5cc54ed47e47a6729dfd8aa5a3dc2476 exposes web search as provider-native web_search through Responses API tool specs.

Tool Surface (`web_search`)

In codex-rs/core/src/client_common.rs, ToolSpec includes WebSearch variant:

WebSearch { external_web_access: Option<bool> }.

This single boolean-like knob encodes cached vs live behavior.

Codex does not define engine-specific built-ins like bing_search or brave_search.

Engine selection is provider-side, outside core tool schema.

Registration in Tool Builder

codex-rs/core/src/tools/spec.rs adds ToolSpec::WebSearch inside build_specs() based on resolved web_search_mode.

Mapping:

Cached -> external_web_access: Some(false)
Live -> external_web_access: Some(true)
Disabled -> omitted

This is explicit, small surface area, and easy to test.

Config Resolution Path

Mode resolution lives in codex-rs/core/src/config/mod.rs.

resolve_web_search_mode() precedence:

profile/global explicit web_search
legacy feature flag WebSearchCached
legacy feature flag WebSearchRequest
no mode (None)

Later, config load path sets fallback default to Cached when unresolved.

Legacy Flag Compatibility

codex-rs/core/src/features.rs keeps backward compatibility for:

web_search_request
web_search_cached

and emits migration guidance toward top-level web_search ("live" | "cached" | "disabled").

This matters for docs stability:

older config keys still exist, but are migration paths, not preferred API.

Per-Turn Resolution with Sandbox Policy

resolve_web_search_mode_for_turn() in config/mod.rs applies runtime policy.

Important behavior:

under DangerFullAccess, mode prefers Live when allowed
under read-only/restricted modes, it keeps preferred mode if allowed, else falls back among allowed modes

So search mode is not purely static config.

It is runtime-constrained by safety policy.

Requirements Constraints and App-Server Mapping

app-server/src/config_api.rs maps requirements to API shape and normalizes allowed_web_search_modes to always include Disabled.

Operational implication:

policy can always force-off web search, even if other modes are constrained.

This is a practical safety invariant.

Search Action Model

codex-rs/protocol/src/models.rs defines WebSearchAction variants:

Search { query, queries }
OpenPage { url }
FindInPage { url, pattern }
Other

These represent action-level semantics rather than provider-specific payloads.

This is useful for UI, logging, and downstream analysis.

Parsing Response Items into Turn Items

core/src/event_mapping.rs parses ResponseItem::WebSearchCall into TurnItem::WebSearch.

When action exists, Codex computes human-readable detail using web_search_action_detail() from core/src/web_search.rs.

When action is missing/partial, it uses WebSearchAction::Other with empty query detail.

This avoids parse failures from partial streaming payloads.

Human Output Path

exec/src/event_processor_with_human_output.rs handles:

WebSearchBegin -> “Searching the web…”
WebSearchEnd -> “Searched: ” when detail exists

Detail is built by web_search_detail() using action + fallback query.

So CLI output stays concise without losing essential action context.

JSONL Output Path

exec/src/event_processor_with_jsonl_output.rs tracks active search calls by call_id.

Behavior:

on begin: create synthetic item id, emit ItemStarted
on end: reuse mapped item id if present, emit ItemCompleted

This preserves event continuity for clients consuming structured streams.

Test Coverage

Codex includes dedicated tests for search mode behavior, not only parser snapshots.

Examples in core/tests/suite/web_search.rs and core/src/tools/spec.rs tests:

cached mode sets external_web_access=false
live mode sets external_web_access=true
mode precedence over legacy flags
per-turn mode change under different sandbox policy

This is strong evidence that mode semantics are intentional and guarded.

Review Mode Interaction

In core/src/codex.rs, review task setup disables web search by disabling search features and forcing turn mode to Disabled.

This is high-value design:

review reproducibility and safety are prioritized over web augmentation.

What Codex Does Not Provide Here

Codex does not implement in-runtime:

custom search engine adapters
query rewriting heuristics
result reranking logic

Those concerns are primarily delegated to provider-native web search behavior.

This keeps core runtime lean, but reduces backend portability without provider changes.

OpenCode Implementation

OpenCode at commit 7ed449974864361bad2c1f1405769fd2c2fcdf42 implements a first-party websearch tool and also supports provider-defined OpenAI web search wrappers.

Tool Registration and Visibility

packages/opencode/src/tool/registry.ts includes WebSearchTool in candidate list, but applies runtime filter:

for websearch and codesearch, allow only when:

model.providerID === "opencode", or
OPENCODE_ENABLE_EXA flag is enabled

So the tool surface is provider-aware and can be feature-flagged.

Tool Contract (`websearch.ts`)

Parameters:

query (required)
numResults (optional)
livecrawl: fallback | preferred
type: auto | fast | deep
contextMaxCharacters (optional)

Defaults from code:

DEFAULT_NUM_RESULTS = 8
type = auto
livecrawl = fallback

This contract is tighter than freeform search text, and gives explicit control knobs for depth/latency tradeoffs.

Permission Gate

Before calling network, tool invokes ctx.ask() with:

permission: "websearch"
patterns: [params.query]
metadata containing all query options

Permission categories are wired in config/config.ts where websearch is part of permission schema.

This gives policy-level control independent of model behavior.

Query Construction Guidance in Prompt Text

packages/opencode/src/tool/websearch.txt includes explicit instruction:

inject current year into recency-sensitive queries.

Example concept:

if year is 2026 and user asks latest AI news, query should include 2026.

This is prompt-level query planner guidance, not runtime algorithmic rewriting, but still materially improves freshness targeting.

Exa MCP Request Construction

Tool constructs JSON-RPC request payload:

jsonrpc: "2.0"
id: 1
method: "tools/call"
params.name: "web_search_exa"
arguments mapped from tool params

Target endpoint:

POST https://mcp.exa.ai/mcp

Headers include:

accept: application/json, text/event-stream
content-type: application/json

So OpenCode treats Exa as MCP-like remote tool backend.

Timeout and Abort Wiring

Tool uses abortAfterAny(25000, ctx.abort) from util/abort.ts.

That composes:

local timeout signal (25s)
external cancellation signal from agent context

Important implementation detail:

abortAfter uses .bind(controller) in timeout callback to avoid closure capture keeping large objects alive.

This is a subtle memory-hygiene optimization.

SSE-Like Response Parsing

Implementation reads full response text, then line-scans for data: prefixes.

For first parseable data payload with non-empty result.content, it returns result.content[0].text.

If none found, returns fallback no-results message.

If HTTP status is non-OK, throws with status and response text.

If abort triggers, throws timeout-specific error.

Output Contract and Metadata

Tool returns:

output: text content string
title: Web search: <query>
metadata: empty object

No structured multi-result schema is returned from this layer.

Practical implication:

downstream model sees flattened text, not an explicit array of result objects.

Interaction with Tool Truncation Layer

OpenCode tool wrapper (tool/tool.ts) automatically truncates output via Truncate.output() unless tool metadata already marks truncation state.

So even if websearch returns large content, global tool truncation policies still apply.

Provider-Defined OpenAI Web Search Wrappers

Separately, OpenCode includes provider-defined tool factories:

openai.web_search
openai.web_search_preview

in provider/sdk/copilot/responses/tool/.

Argument support includes:

allowed domains filters
search context size
user location hints

openai-responses-prepare-tools.ts converts these arguments into Responses API tool fields.

openai-responses-language-model.ts auto-adds include key:

web_search_call.action.sources

when web search provider-defined tools are present, so source metadata is returned automatically.

This is stronger citation support than flattened first-party text path.

Relationship with CodeSearch Tool

OpenCode also ships codesearch tool (backed by Exa context API).

websearch and codesearch share provider/flag gating patterns, but target different retrieval intents:

websearch for broad web discovery
codesearch for API/library code context retrieval

This separation is useful and should be preserved.

Limitations in Current OpenCode Path

Current first-party websearch limitations:

parser returns first content block only
no explicit per-result ranking metadata contract
no built-in multi-provider fallback
no deterministic citation object normalization

These are valid targets for OpenOxide improvements.

Pitfalls & Hard Lessons

Query Quality Dominates Result Quality

Weak queries produce weak results, independent of engine quality.

If user asks:

“fix this timeout issue”

and model searches exactly that, results are noise.

Query planner needs to add:

stack/runtime terms
exact error signatures
version constraints

Search and Fetch Boundary Confusion

Teams often overload one tool for both roles.

That causes:

unclear permission controls
inconsistent output schemas
weak caching strategy

Keep search and fetch separate, with explicit handoff.

Flattened Output Loses Provenance

If tool returns one combined blob, you lose source boundaries, rank, and provider metadata.

That hurts:

citation quality
debugging
trust and reviewability

Cached vs Live Is a Semantic Contract

Codex’s external_web_access changes freshness semantics.

If agent claims “latest” while using cached mode, answer confidence may be overstated.

Mode should be visible in telemetry and optionally echoed in answer rationale.

Permission Fatigue Is Real

Per-query ask prompts can degrade usability quickly.

Pattern-based always-allow is needed, but over-broad patterns become dangerous.

Designing this UX is not optional.

Ranking Drift Across Providers

Same query on Bing/Brave/Exa can produce materially different top results.

If output path assumes stable ranking, behavior becomes non-deterministic across environments.

Prompt Injection in Retrieved Content

Search results can include adversarial text (“ignore previous instructions”, etc.).

If the tool injects raw page text without safety framing, model behavior can degrade.

Retrieved content must be treated as untrusted input.

Domain Reputation and Trust

Not all indexed pages are equal.

Agent should prefer:

official docs
standards bodies
authoritative project repositories

and down-rank low-trust content farms.

Token Budget Blowups

Search tools often return too much text because “it might be useful”.

This silently evicts:

local code context
prior reasoning state
tool outputs required for edits

Result: worse coding performance.

API Schema Drift

Provider schemas evolve.

Without strict adapter boundaries and integration tests, search path can fail at runtime on minor payload shape changes.

Compliance and Data Residency

Some environments require:

specific search providers
regional restrictions
deny-listed domains

Search subsystem must expose policy points instead of hardcoding provider behavior.

OpenOxide Blueprint

Architecture Overview

Use a layered design:

planner
provider adapters
normalizer
policy gate
context packer
telemetry recorder

Do not couple these into one function.

Each layer needs independent tests.

Core Types

Suggested baseline model:

pub enum SearchBackend {
    Bing,
    Brave,
    Exa,
    DuckDuckGoMcp,
}

pub struct SearchRequest {
    pub user_query: String,
    pub planned_queries: Vec<String>,
    pub max_results: usize,
    pub recency_days: Option<u32>,
    pub allowed_domains: Vec<String>,
    pub denied_domains: Vec<String>,
    pub backend_hint: Option<SearchBackend>,
}

pub struct SearchResult {
    pub title: String,
    pub url: String,
    pub snippet: String,
    pub rank: usize,
    pub backend: SearchBackend,
    pub fetched_at_unix: i64,
}

This ensures every backend returns comparable fields.

Provider Trait and Adapter Boundary

#[async_trait::async_trait]
pub trait WebSearchProvider: Send + Sync {
    fn backend(&self) -> SearchBackend;

    async fn search(
        &self,
        req: &ProviderSearchRequest,
    ) -> anyhow::Result<Vec<ProviderSearchResult>>;
}

Use backend-specific DTOs internally, then normalize at boundary.

Never leak provider-native schema beyond adapter module.

Multi-Backend Support Plan

Implement adapters incrementally:

bing_adapter.rs
brave_adapter.rs
exa_adapter.rs
duckduckgo_mcp_adapter.rs

DuckDuckGo MCP adapter should call remote MCP tool endpoint through existing MCP client stack, not custom HTTP shortcuts, so permissions and auth stay consistent.

Query Planner Pipeline

Planner stages:

classify intent (docs, news, api, bug, howto)
enrich with context entities (library name, framework version, error codes)
apply recency policy when prompt implies temporal freshness
apply domain policy hints
generate primary + alternate query candidates

Store planner output in telemetry.

Do not hide rewriting decisions.

Query Construction Heuristics

Concrete heuristics to ship early:

if user includes exact error text, keep exact quoted variant in at least one candidate
if user asks “latest”, append current year and recency window
if repository already reveals dependency version, include version-constrained candidate
if user asks standards/legal/security, prioritize canonical domains list

Execution Strategy

Start with deterministic fallback mode:

call primary backend
if failure or empty, call fallback backend

Later add hedged parallel mode with strict global timeout budget.

Suggested controls:

per-backend timeout
overall turn search budget
max retries per backend
circuit-breaker on repeated backend failures

Result Normalization

Normalization steps:

URL canonicalization (strip tracking params, normalize scheme/host casing)
snippet cleanup (whitespace collapse, unsafe control character removal)
dedupe by canonical URL
rank reconciliation across multi-backend responses

Never drop backend source attribution.

Result Ranking and Fusion

In multi-backend mode, use simple fusion first:

reciprocal rank fusion (RRF)
backend confidence weighting

Expose tunables in config, but ship conservative defaults.

Context Packaging for the Model

Do not inject raw provider payloads.

Build stable context envelope:

[SearchResult #1]
Title: ...
URL: ...
Snippet: ...
Backend: ...

[SearchResult #2]
...

Keep envelope deterministic so prompt caching is effective.

Citation Object Model

When model uses search evidence, store explicit citation records:

source URL
title
backend
retrieval timestamp
optional quote offsets

This enables:

answer traceability
reproducible audits
UI citation rendering

Policy and Permissions

Search should have dedicated permission type.

Policy dimensions:

backend allowlist
domain allow/deny lists
recency access policy
cached/live availability
user approval mode

Integrate with existing sandbox/approval model, not ad-hoc per-tool logic.

Safety Controls

Treat all retrieved text as untrusted.

Add safeguards:

content framing in prompt (external content may be malicious)
optional content sanitization pass
provider/domain trust scoring
max snippet size caps per result

Observability

Emit structured events for each stage:

planner output
backend request start/end
timeout/retry outcomes
normalization stats
dedupe drops
final injected result count

Store enough detail for replay tests without storing raw sensitive payloads by default.

Caching Strategy

Add two cache tiers:

query cache (short TTL)
normalized result cache (longer TTL with backend key)

Cache key should include:

canonical query
backend
policy mode
domain filters
recency window

This avoids cross-policy cache contamination.

Testing Strategy

Ship with layered test matrix.

Unit tests:

query planner rewrites
URL canonicalization
dedupe logic
ranking fusion

Adapter tests:

schema decode with recorded fixtures
error handling and retries

Integration tests:

mode gating (cached/live/disabled)
fallback backend selection
policy enforcement with domain filters

Golden tests:

stable context envelope formatting
citation object serialization

Rollout Plan

Phase 1:

single backend (Exa or Brave)
deterministic planner
normalized results schema

Phase 2:

add Bing adapter
add DuckDuckGo MCP adapter
fallback orchestration

Phase 3:

multi-backend fusion
citation UI enhancements
adaptive planner tuning from telemetry

Non-Goals (Initial)

fully autonomous browsing sessions
full-page crawling in search tool itself
probabilistic ML rankers in v1

Keep v1 deterministic and debuggable.

Crates

Core runtime:

reqwest
serde
serde_json
url
chrono
anyhow
thiserror

Optional resilience/infra:

tower (retry, timeout, circuit breaker)
tracing + tracing-subscriber
dashmap or moka for caches

MCP interop path:

existing MCP client crate stack in OpenOxide core

No duplicate HTTP stack for DuckDuckGo MCP path.