Skip to content

sync_engine

diapason.connectors.sync_engine

SyncEngine — checkpoint/resume orchestration for connector syncs.

Wraps IngestionPipeline with a lightweight SQLite state database so that long-running syncs can be interrupted and resumed from the last saved cursor.

Typical usage::

store = KnowledgeStore(db_path=":memory:")
pipeline = IngestionPipeline(store)
engine = SyncEngine(pipeline)

items = engine.sync(connector)        # first run
items = engine.sync(connector)        # resumes from saved cursor
cp = engine.get_checkpoint(connector.connector_id)

SyncEngine

Orchestrate connector syncs with checkpoint/resume tracking.

Parameters

pipeline: The IngestionPipeline that documents are fed into. state_db: Path to the SQLite database used for checkpoint state. If empty, defaults to DEFAULT_CONFIG_DIR / "sync_state.db".

get_checkpoint(connector_id)

Return the last checkpoint, or None if never synced.

sync(connector)

Run a full sync for connector and return the number of items ingested.

Resumes from the last saved cursor if one exists. Documents are batched in groups of 100 before being handed to the pipeline; a checkpoint is saved after every batch and once more at the end.

On error the checkpoint is updated with the error message and the exception is re-raised so callers can handle it.