dense
diapason.tools.storage.dense
¶
In-memory dense retrieval backend.
Uses any :class:Embedder (default: :class:OllamaEmbedder with
nomic-embed-text) to embed stored text, then ranks queries by
cosine similarity via a single matrix multiply against an
L2-normalized matrix of document embeddings.
Design notes¶
- No persistence. The index lives in memory and is rebuilt at
startup via :mod:
scripts.index_docs. The docs corpus is small (~700 chunks) so this is fine and keeps the implementation simple. - Normalization happens at embed time, not at query time. Storing
unit vectors means retrieval is one
docs @ querydot product. - Store growth is amortized: we keep a list of per-call embedding
matrices and concatenate lazily in :meth:
retrieve. This avoids an O(n²)np.concatenatepattern while still giving callers one contiguous matrix when they actually need to search.
DedupeReport
dataclass
¶
Audit trail for a deduplication pass.
DenseMemory
¶
Bases: MemoryBackend
In-memory dense retrieval via cosine similarity.
The embedder is lazy: it is created on first :meth:store or
:meth:retrieve call, so instantiating :class:DenseMemory does
not require Ollama to be running.
Parameters¶
embedder:
An :class:Embedder instance. If None, defaults to
:class:OllamaEmbedder with nomic-embed-text.
clear()
¶
Drop all stored documents.
count()
¶
Number of stored documents.
delete(doc_id)
¶
Remove a document by id. Returns True if it existed.
retrieve(query, *, top_k=5, **kwargs)
¶
Return top-k documents by cosine similarity.
Scores are in [-1, 1] for normalized vectors; for
nomic-embed-text in practice scores on reasonable queries
fall in [0.3, 0.8]. Empty index or query → empty list.
store(content, *, source='', metadata=None)
¶
Embed and store one document. Returns its id.
store_many(contents, *, sources=None, metadatas=None)
¶
Embed a batch of documents in one go. Much faster than per-doc.
Accepts parallel lists; missing sources/metadatas default to empty.
DuplicateGroup
dataclass
¶
A cluster of chunks judged to be near-duplicates of each other.
MdChunk
dataclass
¶
A markdown chunk annotated with its header breadcrumb.
chunk_markdown(text, *, source='', max_section_tokens=500, paragraph_overlap_tokens=50, max_section_chars=4000)
¶
Split markdown into chunks using ##/### as primary boundaries.
Strategy
- Detect section boundaries at h2 (
##) and h3 (###). h1 (#) is treated as the document title (captured in the breadcrumb but not used to split). - Skip headers that appear inside fenced code blocks — those are
usually shell comments (
# Install X) not real headers. - If a section body exceeds
max_section_tokens(whitespace tokens) ORmax_section_chars(raw chars), slide a window over its paragraphs withparagraph_overlap_tokensoverlap. - Prefix each chunk with a
breadcrumbof its parent headers so the embedding captures hierarchical context.
The char cap is the critical safety net: embedding models count BPE
tokens, and technical content (file paths, code, URLs) has ~8 BPE
tokens per whitespace token — so a whitespace-token-only limit
silently lets 2–3× overflows through and some embedders (e.g.
nomic-embed-text) reject them at runtime. 4000 chars is a
conservative ceiling for nomic-embed-text's 8192-token window.
Empty input → empty list.
dedupe_chunks(chunks, *, ngram_n=5, similarity_threshold=0.7, min_files_for_dup=3)
¶
Drop near-duplicate chunks that recur across many source files.
A chunk cluster is considered a duplicate (and collapsed to one canonical entry) when:
- Pairwise word-level n-gram Jaccard >=
similarity_thresholdon the body text (breadcrumb stripped). N-grams are computed after lowercasing and stripping non-alphanumeric punctuation, so superficial differences (capitalization, typography) don't hide duplication. - The cluster spans >=
min_files_for_dupdistinct source files. Two-file repeats are kept on the assumption they may be legitimately doc-specific; >=3 is the bar for boilerplate.
For each qualifying cluster, the chunk from the most-specific source path wins (deepest dir, longest basename, lexicographic tiebreak); the rest are dropped.
Returns (surviving_chunks, report). The chunker, embedder and
retrieval logic are NOT modified — this function is a pure pre-
processing pass over the chunk list before embedding.
log_dedupe_report(report, *, level=logging.INFO)
¶
Emit a human-readable summary of a dedupe pass at level.