Skip to main content
Version: 1.15

Deploy required microservices

Update inventory with microservices you need

info

The next section will describe all REGARDS microservices and how to enable them.

Disable/enable a REGARDS microservice

If you look at your inventory file regards-cnes/group_vars/regards_nodes/main.yml, you may see that many microservices are commented.

# Stack
group_docker_mservices:
access_project:
tag: "{{ group_docker_tags.back }}"
# notifier:
# tag: "{{ group_docker_tags.back }}"
# fem:
# tag: "{{ group_docker_tags.back }}"
# storage:
# tag: "{{ group_docker_tags.back }}"
# order:
# tag: "{{ group_docker_tags.back }}"
# ingest:
# tag: "{{ group_docker_tags.back }}"
# dataprovider:
# tag: "{{ group_docker_tags.back }}"
# processing:
# tag: "{{ group_docker_tags.back }}"
# worker_manager:
# tag: "{{ group_docker_tags.back }}"
front:
protocol: http

If you uncomment a microservice, like so:

# Stack
# fem:
# tag: "{{ group_docker_tags.back }}"
storage:
tag: "{{ group_docker_tags.back }}"
# order:
# tag: "{{ group_docker_tags.back }}"

It means to the playbook that the microservice storage will be activate on the next deployment.

Internal catalogs

There is 3 types of crawler that lets REGARDS populate its meta-catalog :

All types of crawlers can be active at the same time, but you may not need them on your instance at the same time.

Activate microservice carrefully

If you activate only the microservices you need, the admin HMI will not show you the configuration and pages related to everything you have desactivated !
Activate them only if you need them.

Here is the difference between these two internal catalogs:

OAIS Manager (Ingest)Feature Manager (FEM)
Populate using RESTYY
Populate using AMQP (faster)YY
Product deletionYY
Product updateNo
To update a product, use product versionning
Y
Product versionningYY
Product creation toolsScan yours files using
Data acquisition chains
in order to generate SIP
DIY
Product creation feedback
using AMQP
YY
Product creation feedback
using REST API
YY
Product creation feedback
using HMI
YY
Creation payload formatStrict
List of SIP
matching OAIS standard
Open
List of Feature
that are valid REGARDS feature
Creation payload validationYY
Model validationNo validationFeature must be valid against REGARDS model
Product enhancementOptional enhancement
using custom plugins
Features can be enhancement by workers and then referenced or updated
Enrichment is delegated to the worker system.
Sending product to
microservice Notifier
YY
GoalPerennisationA flow between many applications
with complex product lifecycle
Activate the Ingest microservice :
# Replace
# ingest:
# tag: "{{ group_docker_tags.back }}"
# By
ingest:
tag: "{{ group_docker_tags.back }}"

Activate the Feature Manager microservice :

# Replace
# fem:
# tag: "{{ group_docker_tags.back }}"
# By
fem:
tag: "{{ group_docker_tags.back }}"
info

If you want to crawl an external datasource, do not activate any of these internal catalogs.

Create Ingest product using DataProvider

The microservice DataProvider allows you to create your products using Data acquisition chains. It scans files on disk and creates SIP according to these files.
Here are some core features of DataProvider :

  • Regroup several files inside a product (SIP)
  • Scan the folder periodically in order to create new product when they are available
  • When necessary, product generation can be fully customized using dedicated plugin to extract metadata from the filename or the file content

Activate the DataProvider microservice :

# Replace
# dataprovider:
# tag: "{{ group_docker_tags.back }}"
# By
dataprovider:
tag: "{{ group_docker_tags.back }}"

This microservice requires : Ingest

File management using Storage

The Storage microservice can store on different types of storage system:

  • Online storages : local disk, NFS...
  • Nearline storages : Tape Backup, S3 Glacier...

But it also manages external files that are already available through an existing REST HTTP(S) server.

File already available
through existing REST HTTP(S) server
File NOT available
through existing REST HTTP(S) server
On file indexed inside
internal catalog
Reference the provided
remote server
file link
Two possibilities :
- copy the file to one/many storage sytems
- reference provided file link, as it is
already associated to an existing storage system
Expose file
through STORAGE REST API
No
Storage does not proxy over
the existing REST HTTP(S) server
Y
Support nearline storage systemNY

Once a file is managed by the Storage microservice, its products can be ordered by users.

info

Any product coming from an internal catalog will use the Storage microservice to reference or store its files

Activate the Storage microservice :

# Replace
# storage:
# tag: "{{ group_docker_tags.back }}"
# By
storage:
tag: "{{ group_docker_tags.back }}"

This microservice requires : FEM or Ingest

Internal and external notification management

The microservice Notifier receives all FEM and Ingest products everytime they change.
It allows you to emit a notification to any system with a very granular syntax to decide whenever a product must be sent to some destination.
The Notifier plugin system allows you to adapt the content of the notification sent to match your needs.

Activate the Notifier microservice :

# Replace
# notifier:
# tag: "{{ group_docker_tags.back }}"
# By
notifier:
tag: "{{ group_docker_tags.back }}"

This microservice requires : FEM or Ingest

Workers manager

Workers manager is a powerful system designed to handle any type of task on your workflow.
For example, you can raise a product notification using Notifier microservice to ask a worker to enhance the product.
The worker can read the file and send an update request into the internal catalog.

As there is no Worker publicly available, you do not need to enable this service yet.

Orders management

The microservice Order allows users to make orders.
If this system is not active, orders are not available.

Activate the Order microservice :

# Replace
# order:
# tag: "{{ group_docker_tags.back }}"
# By
order:
tag: "{{ group_docker_tags.back }}"

This microservice requires : Storage

Process orders

The microservice Processing allows you to process ordered datafiles.
Once configured, the user can choose to activate a processing on its ordered data :

  • Convert the project of a NetCDF file
  • Crop a file
  • Call external processing system
  • ...

Activate the Processing microservice :

# Replace
# processing:
# tag: "{{ group_docker_tags.back }}"
# By
processing:
tag: "{{ group_docker_tags.back }}"

This microservice requires : Order

Run changes

Once you have edited your inventory, you must execute the following command to update the stack:

# cd regards-docker-master/
ansible-playbook -i inventories/<inventory name> regards.yml <additional parameters>

That's it!

Full setup conclusion

Congratulations, your REGARDS installation is over.

Execute ./health.sh all and wait all microservices are 200 :

$ ./health.sh all
[200] fem
[200] admin
[200] dam
[200] processing
[200] storage
[200] access_project
[200] ingest
[200] admin_instance
[200] order
[200] catalog
[200] access_instance
[200] worker_manager
[200] dataprovider
[200] notifier
[200] authentication