sync_engine
diapason.connectors.sync_engine
¶
SyncEngine — checkpoint/resume orchestration for connector syncs.
Wraps IngestionPipeline with a lightweight SQLite state database so that
long-running syncs can be interrupted and resumed from the last saved cursor.
Typical usage::
store = KnowledgeStore(db_path=":memory:")
pipeline = IngestionPipeline(store)
engine = SyncEngine(pipeline)
items = engine.sync(connector) # first run
items = engine.sync(connector) # resumes from saved cursor
cp = engine.get_checkpoint(connector.connector_id)
SyncEngine
¶
Orchestrate connector syncs with checkpoint/resume tracking.
Parameters¶
pipeline:
The IngestionPipeline that documents are fed into.
state_db:
Path to the SQLite database used for checkpoint state. If empty,
defaults to DEFAULT_CONFIG_DIR / "sync_state.db".
get_checkpoint(connector_id)
¶
Return the last checkpoint, or None if never synced.
sync(connector)
¶
Run a full sync for connector and return the number of items ingested.
Resumes from the last saved cursor if one exists. Documents are batched in groups of 100 before being handed to the pipeline; a checkpoint is saved after every batch and once more at the end.
On error the checkpoint is updated with the error message and the exception is re-raised so callers can handle it.