Skip to content

scan_chunks

scan_chunks

ScanChunksTool — semantic grep via LM-powered chunk scanning.

Pulls chunks from the KnowledgeStore by filter, batches them, and asks the LM to extract information relevant to a question. Catches semantic matches that keyword-based BM25 search misses.

Classes

ScanChunksTool 

ScanChunksTool(
    store: Optional[KnowledgeStore] = None,
    engine: Optional[InferenceEngine] = None,
    model: str = "",
)

Bases: BaseTool

Semantic grep — feeds chunks to the LM to find information BM25 misses.

Source code in src/diapason/tools/scan_chunks.py 
28
29
30
31
32
33
34
35
36
def __init__(
    self,
    store: Optional[KnowledgeStore] = None,
    engine: Optional[InferenceEngine] = None,
    model: str = "",
) -> None:
    self._store = store
    self._engine = engine
    self._model = model