Skip to content

embedding_store

diapason.connectors.embedding_store

EmbeddingStore --- disk-persistent ColBERT token-level embeddings.

Stores per-chunk ColBERTv2 token embeddings as individual .pt files on disk with a SQLite index for O(1) chunk_id lookups. Designed to be used by ColBERTReranker so that document embeddings computed at ingest time can be reused across queries instead of re-encoding on every search.

All torch usage is lazily imported and guarded behind try/except so that the store degrades gracefully when PyTorch is not installed --- every public method returns None or False instead of raising.

EmbeddingStore

Stores ColBERT token-level embeddings on disk.

Each chunk_id maps to a tensor of shape (num_tokens, 128) saved as a standard PyTorch .pt file. A small SQLite database provides an O(1) index from chunk_id to the tensor filename.

Parameters

store_dir: Root directory for the embedding store. Defaults to ~/.diapason/embeddings/. The directory is created if it does not already exist.

close()

Close the underlying SQLite connection.

count()

Return the number of stored embeddings.

delete(chunk_id)

Delete the embedding for a chunk.

Returns True if the embedding existed and was removed.

get(chunk_id)

Load a chunk's embedding from disk.

Returns None if the chunk has no stored embedding or if torch is not installed.

has(chunk_id)

Check if embeddings exist for a chunk.

store(chunk_id, embedding)

Save a chunk's embedding tensor to disk.

Parameters

chunk_id: Unique identifier for the chunk (matches the KnowledgeStore PK). embedding: Tensor of shape (num_tokens, 128) from ColBERT encoding.