pipeline
diapason.connectors.pipeline
¶
IngestionPipeline — deduplicate, chunk, and store Documents.
Takes Document objects from connectors, deduplicates by doc_id,
splits content using SemanticChunker, and persists chunks to a
KnowledgeStore.
Typical usage::
store = KnowledgeStore(db_path=":memory:")
pipeline = IngestionPipeline(store)
n_chunks = pipeline.ingest(connector.sync())
IngestionPipeline
¶
Deduplicate, chunk, and index documents into a KnowledgeStore.
Parameters¶
store:
The KnowledgeStore instance to write chunks into.
max_tokens:
Soft upper-limit on chunk size passed to SemanticChunker.
attachment_store:
Optional AttachmentStore for persisting attachment blobs and
extracting text from supported MIME types (PDF, plain text, etc.).
When None (default) attachments are silently ignored.
embedder:
Optional embedding client (e.g. OllamaEmbedder). When provided,
every chunk is embedded at ingest time and the resulting float32
vector is written to the embedding BLOB column alongside
embedding_model_version. None (default) skips embedding so
in-memory tests and offline runs don't depend on a sidecar daemon.
ingest(documents)
¶
Ingest an iterable of documents into the knowledge store.
Duplicate doc_id values are silently skipped (both across
calls and within a single batch).
Parameters¶
documents:
An iterable of Document objects (e.g. from a connector's
sync() method).
Returns¶
int The total number of chunks written to the store in this call.