@amigo-labs/text-splitters

RAG text splitters — RecursiveCharacterTextSplitter and MarkdownTextSplitter equivalents, with tiktoken-aware length as a built-in length metric (no JS callback round-trips).

Backed by text-splitter for the segmentation engine and tiktoken-rs for BPE encoding.

Install

pnpm add @amigo-labs/text-splitters

Usage

import {
  splitText,
  splitTextBatch,
  splitMarkdown,
  countTokens,
} from '@amigo-labs/text-splitters'

splitText('your long document…', { chunkSize: 1000, chunkOverlap: 200 })

// Token-aware chunking (for LLM context budgeting):
splitText(doc, {
  chunkSize: 512,
  chunkOverlap: 64,
  lengthMetric: 'tiktoken:cl100k_base',
})

// Markdown-aware (keeps headings/code-blocks intact when they fit):
splitMarkdown(doc, { chunkSize: 1000 })

// Batch one FFI crossing for N documents:
splitTextBatch([doc1, doc2, doc3], { chunkSize: 1000 })

countTokens('hello world', 'tiktoken:cl100k_base')

Install for the browser

The same import works in Angular, React, Vite, esbuild, and webpack ≥ 5 — the bundler picks the WASM build via the browser conditional export:

import { splitText, splitMarkdown, countChars } from '@amigo-labs/text-splitters'

Token metrics are Node-only. countTokens and lengthMetric: "tiktoken:*" throw in the WASM build — the tiktoken-rs BPE tables (~1.5 MB) don’t ship on wasm32. Use countChars / character-based metrics in the browser.

Options

interface SplitterOptions {
  chunkSize?: number       // default 1000
  chunkOverlap?: number    // default 0
  lengthMetric?:
    | 'chars'                    // default
    | 'tiktoken:cl100k_base'     // GPT-3.5 / GPT-4
    | 'tiktoken:o200k_base'      // GPT-4o
}

Scope

• RecursiveCharacterTextSplitter via splitText.
• MarkdownTextSplitter via splitMarkdown (preserves headings, code-blocks, list-items when possible).
• TokenTextSplitter replaced by lengthMetric: 'tiktoken:*'.
• Batch helpers.

Not exposed: lengthFunction callback, custom separators, HTMLTextSplitter / LatexTextSplitter as distinct classes, and createDocuments / splitDocuments (do those in JS after the split).

See __conformance__/divergences.md for the why.

License

MIT

Perf-Review: @amigo-labs/text-splitters

Status: 🟡 Yellow — long-document bucket 🔴 red-flagged (measured) · Reviewed: 2026-07-02 · Version: 0.1.1

Verdict

The measurement inverts the candidate prediction. The candidate review predicted Green at RAG scale and Yellow on tweet-sized inputs; measured reality is the opposite — 2.56× on ~240 B, 1.21× on ~14 KB, and 4.74× slower (0.21×) on the ~140 KB long-document bucket. That long bucket is precisely the RAG-ingest scenario the port exists for, and it fails the candidate’s Green gate (≥3× RAG-small, ≥5× RAG-large) outright. Prime suspect: the text-splitter crate’s semantic-hierarchy scanning cost on large inputs. This is the portfolio’s clearest Phase-C investigation item.

Evidence

Measured speedup (docs/benchmarks/text-splitters.json, 2026-06-10, commit 8c743bf)

Bucket	@amigo-labs splitText	@langchain/textsplitters	Ratio
short (~240 B)	230 315 Hz	90 119 Hz	2.56×
medium (~14 KB)	1 364 Hz	1 129 Hz	1.21×
long (~140 KB)	17.66 Hz	83.72 Hz	0.21× (4.74× slower)

• docs/packages.json speedup: "up to 2.6× faster / 4.7× slower" — deliberately not sugar-coated.

What shipped vs. the candidate prediction

• splitText / splitTextBatch / splitMarkdown / countTokens.
• lengthMetric enum replaced the lengthFunction callback, as the candidate review designed (a JS callback per chunk would have destroyed the FFI budget).
• No custom separators, no createDocuments.
• Tiktoken-based length metrics are Node-only — the WASM build excludes the ~1.5 MB BPE tables.

Phase-C action plan

1. Profile the long bucket — isolate whether the regression is the text-splitter crate’s semantic scan or chunk-string marshalling (an offsets API would fix only the latter).
2. Re-bench after the fix; the crate needs the long bucket at ≥1× minimum to hold Yellow, ≥5× for the originally predicted Green.

Divergences

Chunk boundaries are close to but not byte-identical with @langchain/textsplitters on mixed-separator documents. See crates/text-splitters/__conformance__/divergences.md.

Pre-port assessment: langchain__textsplitters.md

References

• Crate: crates/text-splitters
• Bench shard: docs/benchmarks/text-splitters.json
• docs/packages.json speedup: "up to 2.6× faster / 4.7× slower"