Skip to content

chunking

diapason.tools.storage.chunking

Document chunking with configurable size and overlap.

Splits text into fixed-size chunks (measured in whitespace-split tokens) with a configurable overlap. Paragraph boundaries are respected when they fall within the chunk window.

Chunk dataclass

A single chunk produced by the chunking pipeline.

ChunkConfig dataclass

Parameters controlling the chunking strategy.

chunk_text(text, *, source='', config=None)

Split text into chunks respecting paragraph boundaries.

Parameters

text: The full document text. source: Originating filename or identifier. config: Chunking parameters (uses defaults if None).

Returns

List of :class:Chunk objects, in order.