REGARDS dataprovider microservice conception
This guide talks about Product in the Dataprovider point of view. Ingest product is the real product at Regards point of view.
Dataprovider is used to acquire some products from scanning folders, through acquisition chains. Chains are configured to generate products with one or multiple scanned files.
An acquisition chain has several steps :
Start an acquisition chain
Acquisition chains can be triggered
- periodically, with totally configurable intervals
- manually, from admin UI or directly with Rest API.
- restarted (restart only errors detected from a previous acquisition), from admin UI or directly with Rest API
Scan step
Scan is the action to explore a folder (and a complete tree if needed, depending on which plugin you use), and analyse each file.
Multiple scan path can be indicated. These scan paths are associated to a "since date". Files with an earlier date will not be processed.
To scan a folder, a scan plugin job is used. There are several ways to scan a folder.
After a scan, the since date associated with the scan path is updated to the last file scanned date. Meaning that these files will not be scanned again if the chain is relaunched. Only modified files will be scanned in the future.
Scanned files will be used in the next steps.