Skip to content

pipeline

diapason.connectors.pipeline

IngestionPipeline — deduplicate, chunk, and store Documents.

Takes Document objects from connectors, deduplicates by doc_id, splits content using SemanticChunker, and persists chunks to a KnowledgeStore.

Typical usage::

store = KnowledgeStore(db_path=":memory:")
pipeline = IngestionPipeline(store)
n_chunks = pipeline.ingest(connector.sync())

IngestionPipeline

Deduplicate, chunk, and index documents into a KnowledgeStore.

Parameters

store: The KnowledgeStore instance to write chunks into. max_tokens: Soft upper-limit on chunk size passed to SemanticChunker. attachment_store: Optional AttachmentStore for persisting attachment blobs and extracting text from supported MIME types (PDF, plain text, etc.). When None (default) attachments are silently ignored. embedder: Optional embedding client (e.g. OllamaEmbedder). When provided, every chunk is embedded at ingest time and the resulting float32 vector is written to the embedding BLOB column alongside embedding_model_version. None (default) skips embedding so in-memory tests and offline runs don't depend on a sidecar daemon.

ingest(documents)

Ingest an iterable of documents into the knowledge store.

Duplicate doc_id values are silently skipped (both across calls and within a single batch).

Parameters

documents: An iterable of Document objects (e.g. from a connector's sync() method).

Returns

int The total number of chunks written to the store in this call.