embedding_store
diapason.connectors.embedding_store
¶
EmbeddingStore --- disk-persistent ColBERT token-level embeddings.
Stores per-chunk ColBERTv2 token embeddings as individual .pt files on
disk with a SQLite index for O(1) chunk_id lookups. Designed to be used by
ColBERTReranker so that document embeddings computed at ingest time can
be reused across queries instead of re-encoding on every search.
All torch usage is lazily imported and guarded behind try/except so
that the store degrades gracefully when PyTorch is not installed --- every
public method returns None or False instead of raising.
EmbeddingStore
¶
Stores ColBERT token-level embeddings on disk.
Each chunk_id maps to a tensor of shape (num_tokens, 128) saved
as a standard PyTorch .pt file. A small SQLite database provides an
O(1) index from chunk_id to the tensor filename.
Parameters¶
store_dir:
Root directory for the embedding store. Defaults to
~/.diapason/embeddings/. The directory is created if it does
not already exist.
close()
¶
Close the underlying SQLite connection.
count()
¶
Return the number of stored embeddings.
delete(chunk_id)
¶
Delete the embedding for a chunk.
Returns True if the embedding existed and was removed.
get(chunk_id)
¶
Load a chunk's embedding from disk.
Returns None if the chunk has no stored embedding or if
torch is not installed.
has(chunk_id)
¶
Check if embeddings exist for a chunk.