API: ranking/independence

better-web-search-mcp / ranking/independence

ranking/independence#

Interfaces#

IndependenceInput#

Defined in: src/ranking/independence.ts:30

A document participating in independence analysis.

Properties#

Property	Type	Description	Defined in
`content`	`string`	The extracted page content.	src/ranking/independence.ts:34
`url`	`string`	The source URL, used to derive the registrable-ish host.	src/ranking/independence.ts:32

IndependenceResult#

Defined in: src/ranking/independence.ts:38

Where a document sits among its near-duplicates.

Properties#

Property	Type	Description	Defined in
`cluster`	`number`	Index of the cluster this document belongs to.	src/ranking/independence.ts:40
`duplicates`	`number`	How many other documents share this cluster.	src/ranking/independence.ts:47
`primary`	`boolean`	Whether this is the representative of its cluster. Exactly one document per cluster is primary; the rest are derivative copies of the same text.	src/ranking/independence.ts:45

Variables#

DUPLICATE_THRESHOLD#

const DUPLICATE_THRESHOLD: 0.4 = 0.4

Defined in: src/ranking/independence.ts:24

Jaccard similarity at or above which two documents are near-duplicates.

SHINGLE_SIZE#

const SHINGLE_SIZE: 5 = 5

Defined in: src/ranking/independence.ts:21

Words per shingle. Long enough to be distinctive, short enough to survive edits.

Functions#

analyzeIndependence()#

analyzeIndependence(docs): IndependenceResult[]

Defined in: src/ranking/independence.ts:117

Group documents into clusters of near-duplicates.

Two documents join the same cluster when their shingle sets reach DUPLICATE_THRESHOLD, or when they share a host — the same site publishing two pages on a topic is not two independent confirmations. Clustering is transitive via union-find, so A~~B and B~~C puts all three together even when A and C alone fall below the threshold.

The first document in each cluster is its primary, which keeps the incoming relevance order meaningful: callers rank before calling this.

Parameters#

Parameter	Type	Description
`docs`	readonly `IndependenceInput`[]	Documents in the caller’s preferred (usually ranked) order.

Returns#

IndependenceResult[]

One result per input document, in the same order.

countIndependent()#

countIndependent(results): number

Defined in: src/ranking/independence.ts:196

How many distinct clusters a set of independence results spans.

Parameters#

Parameter	Type
`results`	readonly `IndependenceResult`[]

Returns#

number

hostOf()#

hostOf(url): string

Defined in: src/ranking/independence.ts:94

The host of a URL, lowercased and stripped of a leading www..

Parameters#

Parameter	Type
`url`	`string`

Returns#

string

jaccard()#

jaccard(a, b): number

Defined in: src/ranking/independence.ts:78

Jaccard similarity of two sets: |intersection| / |union|.

Parameters#

Parameter	Type
`a`	`ReadonlySet`<`string`>
`b`	`ReadonlySet`<`string`>

Returns#

number

shingles()#

shingles(text, size?): Set<string>

Defined in: src/ranking/independence.ts:60

Build the set of word shingles for a document.

Exported for testing; callers normally use analyzeIndependence.

Parameters#

Parameter	Type	Default value
`text`	`string`	`undefined`
`size`	`number`	`SHINGLE_SIZE`

Returns#

Set<string>