API: extraction/hydration

better-web-search-mcp / extraction/hydration

extraction/hydration#

Interfaces#

HydrationSummary#

Defined in: src/extraction/hydration.ts:108

What a hydration source contributed, without the payload itself.

Properties#

Property	Type	Description	Defined in
`present`	`string`[]	Which hydration containers were present on the page.	src/extraction/hydration.ts:110
`readable_chars`	`number`	Characters of readable text recovered from them into the page content.	src/extraction/hydration.ts:112

PublishableStructured#

Defined in: src/extraction/hydration.ts:116

The structured data a response carries, with payloads summarized away.

Properties#

Property	Type	Description	Defined in
`hydration`	`HydrationSummary`	What hydration containers held, rather than what they contained.	src/extraction/hydration.ts:120
`jsonLd`	`unknown`[]	Schema.org entities found on the page, capped in size.	src/extraction/hydration.ts:118

Variables#

MAX_HYDRATION_CHARS#

const MAX_HYDRATION_CHARS: 20000 = 20_000

Defined in: src/extraction/hydration.ts:32

Cap on extracted text, so one payload cannot dominate a page.

MAX_JSONLD_CHARS#

const MAX_JSONLD_CHARS: 8000 = 8_000

Defined in: src/extraction/hydration.ts:20

Cap on the JSON-LD kept in a response, in serialized characters.

Functions#

hydrationText()#

hydrationText(data): string

Defined in: src/extraction/hydration.ts:72

Pull the readable text out of a hydration payload.

Duplicate strings are dropped, since payloads routinely carry the same summary under several keys, and the total is capped so a single page cannot flood a research response.

Parameters#

Parameter	Type	Description
`data`	`unknown`	The parsed hydration payload.

Returns#

string

Prose found in the payload, joined by blank lines.

isReadableString()#

isReadableString(value): boolean

Defined in: src/extraction/hydration.ts:54

Whether a string from a payload reads as content.

Parameters#

Parameter	Type
`value`	`string`

Returns#

boolean

publishableStructured()#

publishableStructured(data, readableChars?): PublishableStructured | undefined

Defined in: src/extraction/hydration.ts:149

Reduce extracted structured data to something worth sending to an agent.

Fixing the page content was only half the problem. The response also carried the raw payload under structured_data, so web_extract on a Next.js page still shipped about 80KB of {"props":{"pageProps"… — the content field was clean while the response was not. A live run through an MCP client is what surfaced it; module-level tests had only ever checked content.

JSON-LD is kept, since schema.org entities are small and directly useful. Hydration containers are replaced by a note of which were present: their readable text has already been lifted into the page content by hydrationText, so the payload itself has no remaining consumer.

Parameters#

Parameter	Type	Default value
`data`	`StructuredLike` \| `undefined`	`undefined`
`readableChars`	`number`	`0`

Returns#

PublishableStructured | undefined

stripTags()#

stripTags(value): string

Defined in: src/extraction/hydration.ts:38

Remove tags and decode the few entities that matter, then collapse space.

Parameters#

Parameter	Type
`value`	`string`

Returns#

string