chunking
diapason.tools.storage.chunking
¶
Document chunking with configurable size and overlap.
Splits text into fixed-size chunks (measured in whitespace-split tokens) with a configurable overlap. Paragraph boundaries are respected when they fall within the chunk window.