Documentation on BetterWebSearch MCP

Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

BetterWebSearch works out of the box with DuckDuckGo — no API keys, no .env, no configuration.

Option 1: npx (recommended)#

npx -y better-web-search-mcp --help # keyless — DuckDuckGo works immediately
npx -y better-web-search-mcp # start MCP stdio server

Or install globally:

npm i -g better-web-search-mcp
better-web-search-mcp --help
better-web-search-mcp --version

Option 2: From source (30s)#

git clone https://github.com/PhantomPixelDev/BetterWebSearch-MCP.git
cd BetterWebSearch-MCP
npm install
npm run build
npx better-web-search-mcp --help
BRAVE_API_KEY= npx better-web-search-mcp # still works, DuckDuckGo fallback

No .env required for the first test. The server boots, prints a provider banner to stderr, and every tool works.

Installation

Mon, 01 Jan 0001 00:00:00 +0000

Requirements#

Node.js 20+ and npm
No API keys required — DuckDuckGo works out of the box

Install via npx (no install)#

npx -y better-web-search-mcp --help # keyless — DuckDuckGo works immediately
npx -y better-web-search-mcp # start MCP stdio server

Install globally#

npm i -g better-web-search-mcp
better-web-search-mcp --help
better-web-search-mcp --version

Install from source#

git clone https://github.com/PhantomPixelDev/BetterWebSearch-MCP.git
cd BetterWebSearch-MCP
npm ci
npm run build
npm test # 189+ tests, keyless suite
npx tsc --noEmit # lint

Native binding note#

better-sqlite3 needs a compiled native addon. If it fails to load (older Node, missing build tools, CI sandbox), the cache automatically falls back to an in-memory Map — no action needed. To restore the SQLite cache, run npm rebuild better-sqlite3 (Node 20+ recommended).

Tools Reference

Mon, 01 Jan 0001 00:00:00 +0000

BetterWebSearch MCP exposes 6 tools (5 unique + 1 alias). All are keyless-first — DuckDuckGo provides search without any API keys.

Overview#

Tool	What it does	Keyless?
`web_search`	Fast search (aggregated, deduped, reranked)	✓
`web_research` / `deep_search`	Deep research: rewrites question → parallel searches → top-10 extraction → citations	✓
`web_extract`	Clean extraction `HTTP → hydration → browser` with `confidence` & `api_endpoints`	✓
`web_find`	`site:` scoped search	✓
`web_news`	Recent news + timeline + diversity	✓

`web_search`#

Fast multi-provider search. Aggregates results from all enabled providers, deduplicates, re-ranks by relevance, and returns the top matches.

Configuration

Mon, 01 Jan 0001 00:00:00 +0000

All environment variables are optional. The server starts and works via DuckDuckGo when no keys are set. Create a .env only if you want additional providers.

cp .env.example .env # then edit

Environment Variables#

Variable	Required	Description
`BRAVE_API_KEY` or `BETTER_WEB_SEARCH_BRAVE_API_KEY`	no	Brave Search API key (primary). Get one
`TAVILY_API_KEY` or `BETTER_WEB_SEARCH_TAVILY_API_KEY`	no	Tavily key. tavily.com
`BETTER_WEB_SEARCH_DISABLE_CACHE`	no	`true` → in-memory fallback (no SQLite)
`BETTER_WEB_SEARCH_CACHE_PATH`	no	Custom SQLite path (default `data/cache.db`)
`BETTER_WEB_SEARCH_DISABLE_BROWSER`	no	`true` → skip Playwright fallback

Key aliases#

The config loader supports both the bare name and the BETTER_WEB_SEARCH_* prefix:

Providers

Mon, 01 Jan 0001 00:00:00 +0000

BetterWebSearch MCP uses a provider abstraction layer. Each provider implements the same SearchProvider interface, so the aggregation layer treats them uniformly. Providers are enabled or disabled based on environment keys.

Provider overview#

Provider	Key needed?	Env var	Status
DuckDuckGo	No	—	Always on (keyless fallback)
Brave	Yes	`BRAVE_API_KEY`	Primary — better ranking & recency
Tavily	Yes	`TAVILY_API_KEY`	Optional — when present
SerpApi	Yes	`SERPAPI_KEY`	Stub — not yet implemented

Keyless-first design#

DuckDuckGo is always enabled and requires no API key. This means:

Extraction

Mon, 01 Jan 0001 00:00:00 +0000

BetterWebSearch MCP extracts page content through a three-tier escalation pipeline managed by the AccessRouter. Each tier is progressively more powerful (and slower); the router only escalates when the current tier doesn’t produce enough content.

Overview#

URL → Level 1: HTTP fetch (< 1s)
 → Level 2: Hydration / structured data (1–3s)
 → Level 3: Playwright browser + API interception (3–10s)

Only escalates when needed. A page-cache hit short-circuits the entire pipeline.

Tier 1: HTTP Fetch#

Speed: < 1 second
Method: Standard fetch with user-agent spoofing, 10s timeout, 2MB cap
Confidence: 0.85

Fetches the raw HTML and runs Mozilla Readability to extract clean markdown. Wins when the page has enough visible text (> 500 characters after stripping tags).

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

System overview#

graph TD
 A[Query / Question] --> B[expandQueries<br/>4–6 variants]
 B --> C[Providers<br/>Brave / Tavily / DuckDuckGo]
 C -->|Promise.allSettled| D[Deduplicate<br/>normalizeUrl]
 D --> E[Rerank<br/>relevance + domainScore + recency]
 E --> F[Top 10 Results]
 F --> G[AccessRouter]
 G --> H[Level 1: HTTP Fetch<br/>&lt; 1s]
 H -->|not enough content| I[Level 2: Hydration Data<br/>JSON-LD, __NEXT_DATA__]
 I -->|not enough content| J[Level 3: Playwright Browser<br/>3–10s]
 H --> K[Content Fusion]
 I --> K
 J --> K
 K --> L{Confidence scores<br/>api 0.99 &gt; jsonld 0.95<br/>&gt; rendered 0.90 &gt; readability 0.85}
 L --> M[Structured Response]
 M --> N[domainProfile + apiPatterns<br/>SQLite Cache]

Components#

1. MCP Server (`src/index.ts`)#

The CLI entry point. Parses --help / --version flags (exiting before MCP handshake), then starts the stdio MCP server with all 6 tools registered.

Token Efficiency

Mon, 01 Jan 0001 00:00:00 +0000

The point of doing web work inside the server is that the model never has to read the pages. This page reports what that is actually worth, measured rather than asserted. The harness ships in the repository under benchmarks/.

Method#

Two workflows, the same questions, the same pages:

Path	What lands in the agent’s context
baseline	The agent drives the tools: one `web_search`, then `web_extract` on each of the top 5 results. Every extracted page enters context in full.
research	One `web_research` call. The response is the cited passages plus their metadata.

Both paths open the same URLs, so this isolates the compression the pipeline performs from any difference in what was retrieved.

Changelog

Mon, 01 Jan 0001 00:00:00 +0000

All notable changes to better-web-search-mcp are documented here. Format follows Keep a Changelog and Semantic Versioning.

0.4.4 - 2026-09-02#

Added#

Second benchmark run with the browser tier enabled, recorded in benchmarks/results/browser.json. It is a negative result and is published as one: enabling Tier 3 leaves the compression unchanged within run-to-run noise (86.2% versus 86.5% overall, 83.9% versus 83.8% median) while research spends 24% longer. Tier 3 earns its place on JavaScript-rendered pages that would otherwise extract nothing, not as a way to return less text

0.4.3 - 2026-09-02#

Fixed#

The intermittent CI abort is finally addressed at the right layer. Earlier attempts blamed the wrong thing: switching to forked processes, then serializing them, then closing handles at process exit each lowered the rate without removing it. The mechanism is that vitest tears down the better-sqlite3 addon’s N-API environment between test files, so every file boundary after the addon has loaded is a chance to hit Assertion failed: (env) != nullptr. npm test now runs the one suite that opens a real database in its own vitest invocation, where that environment is destroyed at process exit instead of at a file boundary

0.4.2 - 2026-09-02#

Fixed#

SQLite handles are closed on process exit. A Database left open at exit is finalized during teardown, and when that happened after the N-API environment was gone the addon aborted the process with Assertion failed: (env) != nullptr. It showed up as intermittent CI failures that neither forked nor serialized test runs fully removed, but the same race can strand a WAL file in a real deployment. Open databases are now tracked and closed synchronously on exit, with the hook installed once rather than per database

0.4.1 - 2026-09-02#

Fixed#

A cited passage could be an entire page. splitPassages emitted an over-long block whole, and plenty of extracted pages contain no blank lines at all, so the whole document became a single passage. web_research then returned more text than reading the pages directly would have — the first benchmark run measured a 267% increase against the baseline it was supposed to beat. Blocks longer than 1,200 characters are now windowed on sentence boundaries, without cutting words and with offsets still addressing the source

Added#

Token-efficiency benchmark under benchmarks/, run with npm run bench. It compares an agent driving the tools itself against a single web_research call over the same pages, and is what caught the bug above. Measured over 12 questions: 820,229 characters down to 110,973, an 86.5% reduction overall and 83.8% median. Payload is counted in characters, which is exact and tokenizer-independent; the harness makes no claim about answer quality, which would need a judge model

0.4.0 - 2026-09-02#

Added#

Source independence detection. URL deduplication only ever caught the same page twice; syndication went straight through. Five outlets running one wire story, a vendor press release quoted verbatim, or an article and its own AMP variant all arrived as distinct URLs and were reported as five sources. Pages are now clustered by content shingles (Jaccard over 5-word shingles, union-find so clustering is transitive), and two pages from the same host are treated as one account
evidence block on web_research responses, reporting only things the server actually measured: sources_opened, independent_sources, derivative_sources, query_term_coverage, and cited_spans. There is no language model in this server, so there is no confidence score — an invented one would be worse than none

Changed#

Citations spread across independent accounts rather than distinct URLs, and derivative sources no longer contribute quotes at all. Previously the pass that topped up remaining citation slots could refill them with the same story under a second byline, presenting one claim as corroborated

0.3.0 - 2026-09-02#

Security#

SSRF protection. web_extract took URLs straight from the calling agent and fetchPage followed them with no validation, so http://localhost:8080/, http://192.168.1.1/, and the cloud metadata endpoint at http://169.254.169.254/ were all fetchable and came back as ordinary page content. Every URL is now checked before the request: non-HTTP schemes are refused, hostnames are resolved, and any private, loopback, link-local, carrier-grade-NAT, multicast or reserved address is rejected. IPv4-mapped IPv6 forms such as ::ffff:127.0.0.1 are judged by their IPv4 rules
Redirects are validated per hop. Fetching now uses manual redirects and re-runs the guard on each Location, so a public host cannot bounce the fetcher onto a private one. Chains are capped at 5 hops
Prompt-injection screening. Page text is attacker-controlled, so every web_extract result now carries a security block marking content untrusted and reporting text that tries to address an agent, with the matched pattern and its offset. Suspicious content gets a warning banner; the page text itself is never rewritten, so extraction stays faithful to the source

Changed#

web_research selects passages that answer the question. It previously took the first 400 characters of each of the top 5 pages and never looked at the question, so answers were largely cookie notices and page intros. Passages are now scored against the question with BM25, and the best ones are returned best-first across distinct sources. This also cuts tokens, since only matching paragraphs are included

Added#

web_research responses include citations: each cited span carries its source, the verbatim quote, character offsets into that page’s extracted content, and a relevance score, so a claim can be attributed to a span rather than to a whole page
fetchPage accepts allowPrivateHosts for local fixture servers and ssrf for injecting DNS resolution in tests

0.2.3 - 2026-09-02#

Fixed#

BETTER_WEB_SEARCH_DISABLE_CACHE and BETTER_WEB_SEARCH_CACHE_PATH are finally honored. Both were parsed by the config loader, printed in the startup banner, and documented in --help, but nothing passed them to the cache — the server opened data/cache.db regardless of what either said

Changed#

Test files now run serially in a single forked process. Forks alone were not enough: parallel files still raced the better-sqlite3 addon’s own teardown, tripping Assertion failed: (env) != nullptr inside the native module
Security policy now tracks the 0.2.x line and describes the cache mitigations accurately
The v0.1.0 launch checklist is marked historical rather than reading as a current process

0.2.2 - 2026-09-02#

Fixed#

A locked, corrupt, or unwritable data/cache.db no longer takes the server down at boot. Opening SQLite is guarded and falls back to the in-memory cache with a warning on stderr; directory creation is inside the guard too, so a read-only or permission-denied cache path degrades instead of throwing
The startup banner no longer advertises SerpApi, which has not been part of the provider set since 0.2.0

Added#

mcp.json now tracks the package version automatically via the version lifecycle hook, with a test asserting the two stay in step — the manifest had silently read 0.1.0 while npm was publishing 0.2.1

Changed#

release:* scripts no longer run npm publish locally. Publishing is the tag-triggered workflow’s job; doing both meant the local publish won and CI then failed with E403 cannot publish over the previously published versions, which is how 0.2.1 shipped without provenance and without a GitHub release

0.2.1 - 2026-09-02#

Changed#

Test runner uses forked processes, reducing intermittent Worker exited unexpectedly CI failures caused by loading the better-sqlite3 native addon inside a worker thread (see 0.2.3 for the follow-up that closed the remaining window)
Compiled tests are excluded from the published package: 252 files down to 148, 126.4 kB down to 85.1 kB

0.2.0 - 2026-09-01#

Fixed#

The Level 3 browser tier never ran in production. No call site passed a browserPool to getPage, so the escalation guard skipped it entirely and BrowserPool was constructed only in tests. The router now creates a shared pool lazily on first escalation, so JS-rendered pages actually get rendered
web_extract’s flat 8s per-URL budget was shorter than a browser render’s 23s worst case (15s navigation + 4s intelligent wait + 4s DOM stability), so every Level 3 extraction was killed before it could return. The budget is now sized to the tiers that can run: 8s without the browser, 35s with it
BETTER_WEB_SEARCH_DISABLE_BROWSER was documented but never read; it now actually disables the browser tier
Aborted requests are no longer retried. Providers share one AbortController across attempts, so a retry after the timeout fired could only fail again instantly, burning the retry budget
Brave and Tavily now set retryNetworkErrors, matching DuckDuckGo

Added#

Domain-profile shortcut: the per-domain profile written after every extraction is finally read back, so a domain already served without JavaScript skips the render. mode: "browser" still overrides it
Per-provider 10s timeout inside aggregateSearch — one hung provider no longer gates the other providers’ results
Clean shutdown on SIGINT/SIGTERM so a lazily launched chromium is not orphaned
16 tests (229 total), including a browser-tier suite covering pool reuse, the disable flag, and the profile shortcut

Removed#

The unimplemented SerpApi stub is no longer added to enabledProviders(); it was a guaranteed no-op call on every search. The class remains as a placeholder

Notes#

web_extract on JavaScript-heavy pages is slower than 0.1.2 (up to 35s) but now returns real content instead of a confidence-0 fallback

0.1.2 - 2026-08-31#

Added#

Project banner in the README and on the site homepage
Hugo docs site on the hugo-book theme with native sidebar navigation

Changed#

Node baseline bumped to 22 LTS
Root directory cleanup and a rewritten end-user README

Fixed#

Release workflow npm authentication
Sidebar navigation, duplicate H1 headings, and mobile menu on the docs site

0.1.1 - 2026-08-29#

Fixed#

Packaging and release-workflow fixes on top of the initial release

0.1.0 - 2026-08-29#

Added#

Initial public release — keyless-first MCP server for web research
Multi-provider search abstraction: Brave (primary), Tavily (optional), DuckDuckGo (keyless fallback), SerpApi stub — aggregated via Promise.allSettled, deduplicated and reranked
Query expansion: 4–6 parallel variants incl. German translations (e.g. unlimited mobile internet Germany → unbegrenztes Datenvolumen)
Ranking pipeline: normalizeUrl (utm/gclid/fbclid strip, trailing slash, hash), domainScore curated tiers, rerank (0.5 overlap +0.2 domain +0.2 recency +0.1 score)
3-tier extraction: Level 1 fetchPage (10s, 2MB cap, UA) → Level 2 JSON-LD / __NEXT_DATA__ / self.__next_f / __NUXT__ / __APOLLO_STATE__ → Level 3 Playwright pool (3, image/font/media block, intelligent Race networkidle/article/innerText>1000, page.on('response') JSON capture)
Content Fusion with confidence scores (api 0.99 > jsonld 0.95 > rendered 0.90 > readability 0.85)
Alternative source discovery: AMP variants ?output=1 /amp + quoted-title site: searches, snippet evidence search_snippet 0.6
SQLite cache (better-sqlite3 WAL, data/cache.db) + in-memory fallback — search_cache 15min, page_cache 1h, api_patterns, domain_profiles — domainProfile self-learning per domain
6 MCP tools: web_search, web_research/deep_search (parallel queries, bounded concurrency 3, 8s per page, extractive citations), web_extract (auto/fast/browser), web_find, web_news (recency + timeline + diversity)
Central config loader: BRAVE_API_KEY + BETTER_WEB_SEARCH_BRAVE_API_KEY aliases, cache/browser toggles, keyless hint banner to stderr
CLI: --help / --version (exit before MCP handshake), startup banner listing providers
OpenCode / Claude / Cursor / VS Code configs: mcp.json, .vscode/mcp.json, smithery.yaml
195 tests (vitest), tsc strict, npm run build emits dist/

Notes#

No API keys required for first test — npx better-web-search-mcp works via DuckDuckGo
Add BRAVE_API_KEY for richer ranking & recency filtering
Publish-ready: npm pack --dry-run validated, prepare/prepublishOnly hooks, files whitelist

Contributing

Mon, 01 Jan 0001 00:00:00 +0000

Thank you for considering a contribution! Please see the full Contributing Guide on GitHub for detailed instructions.

Quick start#

git clone https://github.com/PhantomPixelDev/BetterWebSearch-MCP.git
cd BetterWebSearch-MCP
npm ci
npm run build
npm test # 189+ tests — fully keyless
npm run lint # tsc --noEmit — must be zero errors

Key points#

Keyless by default — no API keys needed to develop or test
Node 20+ required
TypeScript strict — no any, no @ts-ignore
Vitest for tests — deterministic mocks, no external keys
Conventional Commits — feat(extraction): ..., fix(providers): ..., docs: ...

PR checklist#

npm run lint — zero errors
npm test — all 189+ tests pass
npm run build — dist/ compiles cleanly
New functionality has tests
Types are explicit — no any or @ts-ignore
CHANGELOG.md updated
npm pack --dry-run — tarball looks correct
No secrets, API keys, or .env values committed

Security#

If you discover a security vulnerability, do not open a public issue. Follow the process in SECURITY.md.

Documentation on BetterWebSearch MCP

Quickstart

Option 1: npx (recommended)#

Option 2: From source (30s)#

Installation

Requirements#

Install via npx (no install)#

Install globally#

Install from source#

Native binding note#

Tools Reference

Overview#

web_search#

Configuration

Environment Variables#

Key aliases#

Providers

Provider overview#

Keyless-first design#

Extraction

Overview#

Tier 1: HTTP Fetch#

Architecture

System overview#

Components#

1. MCP Server (src/index.ts)#

Token Efficiency

Method#

Changelog

0.4.4 - 2026-09-02#

Added#

0.4.3 - 2026-09-02#

Fixed#

0.4.2 - 2026-09-02#

Fixed#

0.4.1 - 2026-09-02#

Fixed#

Added#

0.4.0 - 2026-09-02#

Added#

Changed#

0.3.0 - 2026-09-02#

Security#

Changed#

Added#

0.2.3 - 2026-09-02#

Fixed#

Changed#

0.2.2 - 2026-09-02#

Fixed#

Added#

Changed#

0.2.1 - 2026-09-02#

Changed#

0.2.0 - 2026-09-01#

Fixed#

Added#

Removed#

Notes#

0.1.2 - 2026-08-31#

Added#

Changed#

Fixed#

0.1.1 - 2026-08-29#

Fixed#

0.1.0 - 2026-08-29#

Added#

Notes#

Contributing

Quick start#

Key points#

PR checklist#

Security#

`web_search`#

1. MCP Server (`src/index.ts`)#