Skip to content

dense

diapason.tools.storage.dense

In-memory dense retrieval backend.

Uses any :class:Embedder (default: :class:OllamaEmbedder with nomic-embed-text) to embed stored text, then ranks queries by cosine similarity via a single matrix multiply against an L2-normalized matrix of document embeddings.

Design notes

  • No persistence. The index lives in memory and is rebuilt at startup via :mod:scripts.index_docs. The docs corpus is small (~700 chunks) so this is fine and keeps the implementation simple.
  • Normalization happens at embed time, not at query time. Storing unit vectors means retrieval is one docs @ query dot product.
  • Store growth is amortized: we keep a list of per-call embedding matrices and concatenate lazily in :meth:retrieve. This avoids an O(n²) np.concatenate pattern while still giving callers one contiguous matrix when they actually need to search.

DedupeReport dataclass

Audit trail for a deduplication pass.

DenseMemory

Bases: MemoryBackend

In-memory dense retrieval via cosine similarity.

The embedder is lazy: it is created on first :meth:store or :meth:retrieve call, so instantiating :class:DenseMemory does not require Ollama to be running.

Parameters

embedder: An :class:Embedder instance. If None, defaults to :class:OllamaEmbedder with nomic-embed-text.

clear()

Drop all stored documents.

count()

Number of stored documents.

delete(doc_id)

Remove a document by id. Returns True if it existed.

retrieve(query, *, top_k=5, **kwargs)

Return top-k documents by cosine similarity.

Scores are in [-1, 1] for normalized vectors; for nomic-embed-text in practice scores on reasonable queries fall in [0.3, 0.8]. Empty index or query → empty list.

store(content, *, source='', metadata=None)

Embed and store one document. Returns its id.

store_many(contents, *, sources=None, metadatas=None)

Embed a batch of documents in one go. Much faster than per-doc.

Accepts parallel lists; missing sources/metadatas default to empty.

DuplicateGroup dataclass

A cluster of chunks judged to be near-duplicates of each other.

MdChunk dataclass

A markdown chunk annotated with its header breadcrumb.

chunk_markdown(text, *, source='', max_section_tokens=500, paragraph_overlap_tokens=50, max_section_chars=4000)

Split markdown into chunks using ##/### as primary boundaries.

Strategy
  1. Detect section boundaries at h2 (##) and h3 (###). h1 (#) is treated as the document title (captured in the breadcrumb but not used to split).
  2. Skip headers that appear inside fenced code blocks — those are usually shell comments (# Install X) not real headers.
  3. If a section body exceeds max_section_tokens (whitespace tokens) OR max_section_chars (raw chars), slide a window over its paragraphs with paragraph_overlap_tokens overlap.
  4. Prefix each chunk with a breadcrumb of its parent headers so the embedding captures hierarchical context.

The char cap is the critical safety net: embedding models count BPE tokens, and technical content (file paths, code, URLs) has ~8 BPE tokens per whitespace token — so a whitespace-token-only limit silently lets 2–3× overflows through and some embedders (e.g. nomic-embed-text) reject them at runtime. 4000 chars is a conservative ceiling for nomic-embed-text's 8192-token window.

Empty input → empty list.

dedupe_chunks(chunks, *, ngram_n=5, similarity_threshold=0.7, min_files_for_dup=3)

Drop near-duplicate chunks that recur across many source files.

A chunk cluster is considered a duplicate (and collapsed to one canonical entry) when:

  1. Pairwise word-level n-gram Jaccard >= similarity_threshold on the body text (breadcrumb stripped). N-grams are computed after lowercasing and stripping non-alphanumeric punctuation, so superficial differences (capitalization, typography) don't hide duplication.
  2. The cluster spans >= min_files_for_dup distinct source files. Two-file repeats are kept on the assumption they may be legitimately doc-specific; >=3 is the bar for boilerplate.

For each qualifying cluster, the chunk from the most-specific source path wins (deepest dir, longest basename, lexicographic tiebreak); the rest are dropped.

Returns (surviving_chunks, report). The chunker, embedder and retrieval logic are NOT modified — this function is a pure pre- processing pass over the chunk list before embedding.

log_dedupe_report(report, *, level=logging.INFO)

Emit a human-readable summary of a dedupe pass at level.