hybrid_search
diapason.connectors.hybrid_search
¶
Hybrid retrieval over the KnowledgeStore: metadata filter + BM25 + vector cosine.
A single search entrypoint that the agentic research loop calls as a tool.
Structured WHERE-clause filters (person, time range, sources) narrow the
candidate set, then BM25 (FTS5) and dense cosine similarity score the
survivors. The two ranks are fused with Reciprocal Rank Fusion, which is
robust to the very different score scales the two signals produce
(BM25 ~ [0, 20], cosine ~ [0.4, 0.9] for nomic-embed-text).
Each result is enriched with its thread context: when a hit belongs to a
thread_id, the surrounding chunks are attached so the synthesis model
sees the conversation, not an isolated fragment.
Brute-force vector scan is fine at the current corpus size (~5k chunks × 768 dims fits in ~15 MB and matmuls in <50 ms). Swap in an ANN index when that stops being true.
HybridSearch
¶
Hybrid BM25 + dense-cosine retrieval over a KnowledgeStore.
Parameters¶
store:
The store to query.
embedder:
Embedding client used to encode the query. When None, search
falls back to BM25 only and reports vector_score=0.
bm25_weight, vector_weight:
Weights on the two RRF terms. Defaults to 0.5 / 0.5; raise either to
bias retrieval toward lexical or semantic matches.
rrf_k:
RRF damping constant. Larger values flatten the contribution of
deeper ranks; 60 is the canonical value from the original paper.
recall_k:
How deep each individual ranker recalls before fusion. Should be at
least a few times limit so the fuser has overlap to work with.
search(query, *, person=None, time_range=None, sources=None, limit=20)
¶
Run the hybrid pipeline and return up to limit hits.
See module docstring for ranking semantics. query may be empty
when callers want a pure metadata filter (e.g. "all mail from X in
May") — in that case only the vector leg runs (and only if an
embedder is configured); if neither leg yields anything the
structured filter is applied directly. Upcoming calendar timelines are
returned nearest-first; other fallbacks return the most recent rows.
SearchHit
dataclass
¶
A single hybrid-search result with enough context for citation.