Token Efficiency

The point of doing web work inside the server is that the model never has to read the pages. This page reports what that is actually worth, measured rather than asserted. The harness ships in the repository under benchmarks/.

Method#

Two workflows, the same questions, the same pages:

Path	What lands in the agent’s context
baseline	The agent drives the tools: one `web_search`, then `web_extract` on each of the top 5 results. Every extracted page enters context in full.
research	One `web_research` call. The response is the cited passages plus their metadata.

Both paths open the same URLs, so this isolates the compression the pipeline performs from any difference in what was retrieved.

Payload is counted in characters — exact and tokenizer-independent. Token figures are an estimate at 4 characters per token, for orientation only. The reduction ratio is the result that matters, and it barely moves between tokenizers.

Results#

12 questions spanning definition, technical, comparison, troubleshooting and research categories. Run 2026-09-02 against the live web with BETTER_WEB_SEARCH_DISABLE_BROWSER=true.

	Baseline	`web_research`
Payload	820,229 chars	110,973 chars
Tokens (est.)	~205,000	~28,000
Wall clock	137.3s	49.6s

86.5% less text overall, 83.8% median.

Does the browser tier change this?#

The run above keeps to the HTTP and structured tiers. Repeating it with the browser tier enabled answers whether Tier 3 pages — the heaviest ones — shift the result:

	HTTP + structured	Browser enabled
Reduction (overall)	86.5%	86.2%
Reduction (median)	83.8%	83.9%
`web_research` wall clock	49.6s	61.3s
Baseline payload	820,229 chars	788,743 chars

It does not. The compression is the same within run-to-run noise — the two baselines differ by 3.8% on their own, since live pages change between runs — while research spends 24% longer. Tier 3 simply does not fire often for these questions: most pages carry enough content over plain HTTP.

That is a useful negative result. The browser tier earns its place on JavaScript-rendered pages that would otherwise extract nothing, not as a way to return less text.

Per question:

Question	Baseline	Research	Reduction	Independent sources
What is the Model Context Protocol?	101,958	12,765	87.5%	4/5
What does Mozilla Readability do?	48,385	8,186	83.1%	4/5
How does Playwright wait for network idle?	19,946	8,610	56.8%	4/5
SQLite WAL mode tradeoffs	54,591	8,738	84.0%	5/5
BM25 term frequency saturation	160,596	10,808	93.3%	5/5
PostgreSQL vs MySQL for JSON	43,894	7,371	83.2%	4/4
Vitest compared to Jest	45,925	7,543	83.6%	5/5
Preventing SSRF on user URLs	46,921	8,464	82.0%	5/5
npm provenance from GitHub Actions	57,736	8,117	85.9%	5/5
Prompt-injection defenses for browsing agents	74,365	10,869	85.4%	4/5
RAG chunking strategies	54,930	12,232	77.7%	7/10
Technical SEO for static docs sites	110,982	7,270	93.4%	5/5

The 4/5 and 7/10 columns are the syndication check working on live pages: those runs opened five or ten URLs but found fewer genuinely separate accounts.

Does compression keep the answer?#

Payload reduction is only a win if the retained text still contains the answer. Measuring that usually needs a judge model. npm run bench:quality avoids one by restricting itself to questions with an unambiguous marker — a port number, a status code, a license name — and asking a narrower question a regex can settle: given the answer was in the raw pages, did compression keep it?

Retention is conditioned on the baseline on purpose. A question whose marker never appeared in the fetched pages is a retrieval miss, not a compression loss, and would otherwise blame the passage selector for weak search results.

34 questions, run 2026-09-04:


Answer present in baseline	34 / 34
Answer retained after compression	33 / 34
Retention	97.1%
Payload reduction on this set	92.4%

By category:

Category	Retained	Retention
Numbers (ports, limits)	8 / 8	100%
HTTP status codes	6 / 6	100%
Acronym expansions	12 / 12	100%
Facts (licenses, defaults)	7 / 8	87.5%

Acronym questions: 58.3% to 100%#

Earlier runs retained only 58.3% of acronym questions. The cause was specific: “What does TLS stand for?” is answered by a sentence that states the expansion once, while the pages around it repeat the acronym constantly, so BM25 ranked the answer below the noise. Passage selection now detects an acronym-definition question and surfaces the passage that spells the acronym out.

That claim is measured rather than asserted. Replaying a single frozen page corpus, so the only variable is the selection code:

	Before	After
Acronym	6 / 12 (50.0%)	12 / 12 (100%)
Overall	27 / 34 (79.4%)	33 / 34 (97.1%)

A live run afterwards reported the same 97.1% and 12/12, which is the check that the offline harness is not flattering itself.

The one remaining loss is not an acronym question. “Which HTTP request header is used to make a conditional request with an ETag?” is answered by If-None-Match inside reference tables, which the paragraph-based splitter does not keep intact.

This still does not measure whether the sources are correct, only whether the answer survived the pipeline.

What this does not measure#

Answer quality. Judging whether the retained passages still support a correct answer requires a judge model, which this harness does not have. A smaller payload is only a win if the right text survived. Per-question query_term_coverage is recorded as a weak proxy and is deliberately not folded into the headline figure.
End-to-end agent cost. A real agent may issue follow-up searches; this is one question, one pass.
Other tools. No comparison against Exa, Tavily’s answer endpoint or hosted web search, because they cannot be run under identical conditions against the same pages.

Reduction also tracks how much page text existed in the first place. The 56.8% case had a roughly 20k-character baseline, where the fixed cost of the response’s source list dominates.

Reproducing#

npm run build
npm run bench                    # all questions
npm run bench -- --questions 3   # a quick subset

Results are written to benchmarks/results/latest.json after every question, so an interrupted run still leaves usable data. Runs hit the live web and are not bit-reproducible: pages change, and DuckDuckGo rate-limits scrapers. Set BRAVE_API_KEY or TAVILY_API_KEY for steadier results.