@amigo-labs/bm25

BM25 full-text search for Node.js — stateful native index, pure-Rust tokenizer + inverted postings. Build once, query many.

Install

pnpm add @amigo-labs/bm25

Usage

import { Bm25Index } from '@amigo-labs/bm25'

const idx = new Bm25Index({ k1: 1.5, b: 0.75 })
idx.addAll([
  { id: 'a', text: 'rust programming language' },
  { id: 'b', text: 'javascript development guide' },
  { id: 'c', text: 'rust web assembly' },
])

idx.search('rust')
// [
//   { id: 'a', score: 0.87 },
//   { id: 'c', score: 0.82 },
// ]

// Add a single doc incrementally:
idx.addDoc('d', 'rust cli tooling')

// Limit results:
idx.search('rust', { limit: 2 })

// Strip English stopwords at both index- and query-time:
new Bm25Index({ removeStopwords: true })

Options

interface IndexOptions {
  k1?: number                 // default 1.5
  b?: number                  // default 0.75
  removeStopwords?: boolean   // default false, English list
}

interface SearchOptions {
  limit?: number              // default 10
}

Shape

• Stateful — an index is an opaque native object. Build it once in the constructor, query many times.
• Thread-safe inside a single index (Mutex-protected).
• Single-FFI-crossing corpus ingest via addAll(docs).
• Shared Rust core with @amigo-labs/minisearch (both wrap an internal _search-core crate).

Install for the browser

The same import works in Angular, React, Vite, esbuild, and webpack ≥ 5 — the bundler picks the WASM build via the browser conditional export:

import { Bm25Index } from '@amigo-labs/bm25'

The _search-core tokenizer + BM25 scorer is the same code on both sides, so ranking is identical between Node and the browser.

Scope cuts

• No multi-field weighted index (v0.2). Concatenate boosted text if needed.
• No per-doc boost (fldWeights-style).
• No save / load — rebuild on startup.
• No stemming. For corpora requiring Porter stemmer, stay on wink-bm25-text-search.

See __conformance__/divergences.md.

License

MIT

Perf-Review: @amigo-labs/bm25

Status: 🟢 Green (measured) · Reviewed: 2026-07-02 · Version: 0.1.1

Verdict

13.14× on query throughput vs. the JS baseline (bench 2026-05-28). BM25 retrieval has one of the best Green shapes in the RAG category: the index is built once (substantial compute — tokenize + IDF + posting lists), lives long-term as a NAPI class, and answers each query with 10–500 µs of real work. The candidate review predicted Green for index build and Yellow-leaning-Green for queries; the measured query multiplier lands clearly in Green territory. The stateful-class shape means the FFI boundary carries only the query string in and a compact hit list out — no per-call index marshalling.

Evidence

Measured speedup (docs/benchmarks/bm25.json, 2026-05-28, commit 45d29b4)

Scenario	@amigo-labs/bm25	JS baseline	Speedup
query, 1000-doc index	18 012.67 Hz	1 370.76 Hz (okapibm25, rebuild per query)	13.14×
index build, 1000 docs (addAll)	287.54 Hz	— (not benched)	—

• docs/packages.json speedup: "13× faster".
• Install size: 25 KB vs wink-bm25-text-search’s 5.1 MB node_modules footprint.

Benchmark gaps

• Index build has no JS baseline in the shard — wink-bm25-text-search’s build path was not benched. The candidate review predicted Green here; unverified.
• Large-corpus buckets (50k+ chunks) from the candidate’s RAG scenario are not measured.

What shipped vs. the candidate prediction

The candidate review sketched a stateful Bm25Index NAPI class with batch add and top-k query — that is what shipped. Scope cuts in v0.1:

• RSJ+1 IDF variant (non-negative), not wink’s exact IDF formula.
• Simpler tokenizer: lowercase + alphanumeric split, no stemming pipeline.
• No multi-field documents, no field boosts, no index persistence yet.

Divergences

Scores are not numerically comparable to wink-bm25-text-search (IDF variant + tokenizer differences); ranking order matches on the conformance corpus except where stemming would merge terms. See crates/bm25/__conformance__/divergences.md.

Pre-port assessment: wink-bm25-text-search.md

References

• Crate: crates/bm25
• Bench shard: docs/benchmarks/bm25.json
• docs/packages.json speedup: "13× faster"