Version: 1.13.0

Feature Manager AMQP API

Overview

Feature management microservice is a data repository in GeoJSON format.

Feature Management reponsabilities:

Extract data references thanks to [IFeatureFactoryPlugins],
Create, patch or delete data references,
Re-notify stakeholders of existing data references,
Delegate the storage of files (if any) to Storage Management.

To edit this repository, a data producer has to send requests.

At the moment, 2 API are available :

Messaging API (AMQP) allows to publish creation, reference, patch, deletion and notification requests on specific exchanges.
HTTP REST API allows to submit creation requests (as POST HTTP requests), update requests (as PATCH HTTP requests) or deletion requests (as DELETE HTTP requests).

Feature management architectural concepts

Under the hood, those reponsabilities are divided between two modules: featureprovider and feature. featureprovider is only responsible for handling data references extraction requests, that is extraction of information needed from physical files to create a data reference that is then handled by the feature module.

API are documented in detail below.

Request payload

Regardless of the API used, payload of each API is expected in GeoJSON format.

The basic structure is as follows :

A required id,
A required type with value Feature,
An optional geometry in GeoJSON format,
An optional set of properties.

{
    "id": "FeatureId",
    "type": "Feature",
    "geometry": {
        "type": "Point",
        "coordinates": [
            125.6,
            10.1
        ]
    },
    "properties": {
        "name": "Dinagat Islands"
    }
}

For the purposes of this microservice, GeoJSON structure is extended with following properties :

An urn (uniform resource name as unique identifier) generated by the microservice when creating a new reference and expected only when updating a reference.
A required model representing the name of the model defining the expected properties field structure (and previously configured).
A required entityType defining the reference business type.
An optional files property with a fixed structure that allows to store or reference physical data (service delegated to another microservice called Storage Management).

{
    "id": "FeatureId",
    "urn": "UniqueFeatureId",
    "model": "RelatedModelName",
    "entityType": "DATA",
    "type": "Feature",
    "geometry": {
        "type": "Point",
        "coordinates": [
            125.6,
            10.1
        ]
    },
    "properties": {
        "name": "Dinagat Islands"
    },
    "files": [
        {
            "locations": [
                {
                    "storage": "DISK",
                    "url": "file://home/user/regards/file.zip"
                }
            ],
            "attributes": {
                "dataType": "RAWDATA",
                "mimeType": "application/zip",
                "filename": "file.zip",
                "filesize": "8013",
                "algorithm": "MD5",
                "checksum": "4e188bd8a6288164c25c3728ce394927"
            }
        }
    ]
}

Payload detailed properties

Feature

Path	Type	Description	Optional
id	`String`	Id from provider
urn	`String`	Unique feature identifer based on provider identifier with TEST:REQUEST:2342

danger

Urn is only expected in update and deletion requests!

Files

Path	Type	Description	Optional
locations[].storage	`String`	Storage	true
locations[].url	`String`	Url location
attributes.dataType	`String`	RAWDATA, QUICKLOOK_SD, QUICKLOOK_MD, QUICKLOOK_HD, DOCUMENT, THUMBNAIL, DESCRIPTION
attributes.mimeType	`String`	MIME type
attributes.filename	`String`	Filename
attributes.filesize	`Number`	File size	true
attributes.algorithm	`String`	Checksum algorithm	true
attributes.checksum	`String`	Checksum	true

danger

Algorithm & cheksum are required if data have to be stored by Storage Management.

Request metadata

As the payload, regardless of the API used, metadata is often associated with a request.

Metadata detailed properties

Path	Type	Description	Optional
metadata.override	`Boolean`	Indicate wether the previous version should be deleted	true
metadata.session	`String`	Arbitrary session name to classify data for human operators
metadata.sessionOwner	`String`	Arbitrary session owner to classify data for human operators
metadata.storages	`Array`	Target storages if there are files to store (may be empty!)	true
metadata.storages[].pluginBusinessId	`String`	Storage plugin identifier (previously configured in `Storage Management`
metadata.storages[].targetTypes	`Array`	List of data object types accepted by this storage location	true
metadata.storages[].storePath	`String`	Directory in which to store the file	true
metadata.priority	`String`	HIGH, NORMAL, LOW

danger

override should only be specified with Extraction or Creation requests.

AMQP API

Feature creation request

Request has to be published on this exchange : regards.broadcast.fr.cnes.regards.modules.feature.dto.event.in.FeatureCreationRequestEvent

With following properties:

Property	Type
metadata	(look at description above)
feature	(look at description above)

And following headers:

Header	Value
regards.tenant	The tenant
regards.request.id	String of max 36 characters long
regards.request.date	ISO 8601 date
regards.request.owner	String of max 128 characters long

Example

# Headers
regards.tenant=project1
regards.request.id=12345463-0f6b-4488-b58c-52af3f7e9563
regards.request.owner=owner
regards.request.date=2019-07-04T01:03:00

{
    "metadata": {
        "sessionOwner": "owner",
        "session": "session",
        "storages": [
            {
                "pluginBusinessId": "disk"
            }
        ],
        "priority": "NORMAL"
    },
    "feature": {
        "entityType": "DATA",
        "model": "FEATURE01",
        "files": [
            {
                "locations": [
                    {
                        "storage": null,
                        "url": "http://www.test.com/filename.xml"
                    }
                ],
                "attributes": {
                    "dataType": "RAWDATA",
                    "mimeType": "application/xml",
                    "filename": "filename",
                    "filesize": 100,
                    "algorithm": "MD5",
                    "checksum": "checksum"
                }
            }
        ],
        "id": "MyId",
        "geometry": {
            "coordinates": [
                10.0,
                20.0
            ],
            "type": "Point",
            "bbox": null,
            "crs": null
        },
        "normalizedGeometry": null,
        "properties": {
            "data_type": "TYPE01",
            "file_characterization": {
                "valid": true
            }
        },
        "type": "Feature"
    }
}

Feature creation request by file reference

This API allows to generate feature by extracting metadata from the passed location.

Request has to be published on this exchange : regards.broadcast.fr.cnes.regards.modules.featureprovider.domain.FeatureExtractionRequestEvent

With following properties:

Property	Type
metadata	(look at description above)
parameters	Free JSON parameters to be used by related factory
factory	Plugin business identifier representing the feature factory to use

info

Factory must have been configured beforehand.

And following headers:

Header	Value
regards.tenant	The tenant
regards.request.id	String of max 36 characters long
regards.request.date	ISO 8601 date
regards.request.owner	String of max 128 characters long

Example

# Headers
regards.tenant=project1
regards.request.id=12345463-0f6b-4488-b58c-52af3f7e9563
regards.request.owner=owner
regards.request.date=2019-07-04T01:03:00

{
    "metadata": {
        "sessionOwner": "owner",
        "session": "session",
        "storages": [
            {
                "pluginBusinessId": "disk"
            }
        ],
        "priority": "NORMAL"
    },
    "factory": "{factory identifier}",
    "parameters": {
        "location":"my/file/location"
    }
}

Feature patch request

Request has to be published on this exchange : regards.broadcast.fr.cnes.regards.modules.feature.dto.event.in.FeatureUpdateRequestEvent

With following properties:

Property	Type
metadata.priority	(look at description above)
metadata.storages	(look at description above)
feature	(look at description above)

info

Only properties to be updated can be passed on ... they will be merged with existing ones.

And following headers:

Header	Value
regards.tenant	The tenant
regards.request.id	String of max 36 characters long
regards.request.date	ISO 8601 date
regards.request.owner	String of max 128 characters long

Example

# Headers
regards.tenant=project1
regards.request.id=12345463-0f6b-4488-b58c-52af3f7e9563
regards.request.owner=owner
regards.request.date=2019-07-04T01:03:00

{
    "metadata": {
        "storages": [],
        "priority": "NORMAL"
    },
    "feature": {
        "urn": "URN:FEATURE:DATA:tenant:87fdda0e-27d1-494f-a1c0-c57f2f0810f7:V1",
        "entityType": "DATA",
        "model": "FEATURE01",
        "files": [],
        "id": "MyId",
        "geometry": null,
        "normalizedGeometry": null,
        "properties": {
            "file_characterization": {
                "invalidation_date": "2019-12-03T12:31:42.466Z",
                "valid": false
            }
        },
        "type": "Feature"
    }
}

Feature deletion request

Request has to be published on this exchange : regards.broadcast.fr.cnes.regards.modules.feature.dto.event.in.FeatureDeletionRequestEvent

Property	Type
priority	(look at description above)
urn	Unique identifier of the feature

With following headers:

Header	Value
regards.tenant	The tenant
regards.request.id	String of max 36 characters long
regards.request.date	ISO 8601 date
regards.request.owner	String of max 128 characters long

Example

# Headers
regards.tenant=project1
regards.request.id=78938463-023a-4488-b58c-52af3f7e9446
regards.request.owner=owner
regards.request.date=2020-03-17T12:24:05.995Z

{
    "priority": "NORMAL",
    "urn": "URN:FEATURE:DATA:tenant:87fdda0e-27d1-494f-a1c0-c57f2f0810f7:V1"
}

AMQP monitoring API...

Feature manager microservice publishes AMQP messages to monitor request lifecycle.

Messages are published to two different exchanges because under the hood Feature manager is in reality composed of two modules: feature and featureprovider. Whatever the exchange to which those monitoring messages are comming from, they have the same structure, that is:

The requestId (corresponds to regards.request.id header),
The requestOwner (corresponds to regards.request.owner header),
The related feature id (not specified for EXTRACTION requests because feature is not yet created),
The related feature urn (not specified for EXTRACTION requests because feature is not yet created),
The request type (EXTRACTION,CREATION,PATCH,DELETION,NOTIFICATION,FILE_COPY,SAVE_METADATA)
The state of the request (GRANTED,DENIED,ERROR or SUCCESS),
A list of errors if any.

info

To receive these messages, your have to subscribe to this exchange.

info

EXTRACTION process does not create features. So if you want to know the created feature id or urn, you have to listen to responses from the CREATION process which has the same requestId than the corresponding EXTRACTION request.

caution

If AMQP virtual host mode is set to SINGLE, this exchange will receive all messages of all tenants! So the receiver may have to filter them according to the tenant he wishes to manage.

caution

Requests without requestId can be routed to AMQP Dead Letter Queue (DLQ) so no response will be published. However, a notification is sent to the administrators of the project in order to tell them to inspect DLQ.

Example of DENIED request

{
    "requestId": "{requestId}",
    "requestOwner": "{requestOwner}",
    "id": "{featureId}",
    "type": "{requestType}",
    "state": "DENIED",
    "errors": ["error1", "error2"]
}

Example of GRANTED request

{
    "requestId": "{requestId}",
    "requestOwner": "{requestOwner}",
    "id": "{featureId}",
    "type": "{requestType}",
    "state": "GRANTED"
}

Example of SUCCESS request

{
    "requestId": "{requestId}",
    "requestOwner": "{requestOwner}",
    "id": "{featureId}",
    "urn": "{featureGeneratedUrn}",
    "type": "{requestType}",
    "state": "SUCCESS"
}

Example of ERROR request

{
    "requestId": "{requestId}",
    "requestOwner": "{requestOwner}",
    "id": "{featureId}",
    "urn": "{featureGeneratedUrn}",
    "type": "{requestType}",
    "state": "ERROR",
    "errors": ["error1", "error2"]
}

...For feature module

Messages are published on this exchange : regards.broadcast.fr.cnes.regards.modules.feature.dto.event.out.FeatureRequestEvent

...For featureprovider module

Messages are published on this exchange : regards.broadcast.fr.cnes.regards.modules.featureprovider.domain.FeatureExtractionResponseEvent

In other words, it means that to migrate from V1.3.x to V1.4.0, you need to change the exchange to which you subscribe to receive extraction monitoring messages. Or you might be able to configure your AMQP broker to automatically redirect messages from this exchange to the old one (regards.broadcast.fr.cnes.regards.modules.feature.dto.event.out.FeatureRequestEvent).

Overview​

Request payload​

Payload detailed properties​

Feature​

Files​

Request metadata​

Metadata detailed properties​

AMQP API​

Feature creation request​

Example​

Feature creation request by file reference​

Example​

Feature patch request​

Example​

Feature deletion request​

Example​

AMQP monitoring API...​

Example of DENIED request​

Example of GRANTED request​

Example of SUCCESS request​

Example of ERROR request​

...For feature module​

...For featureprovider module​

Overview

Request payload

Payload detailed properties

Feature

Files

Request metadata

Metadata detailed properties

AMQP API

Feature creation request

Example

Feature creation request by file reference

Example

Feature patch request

Example

Feature deletion request

Example

AMQP monitoring API...

Example of DENIED request

Example of GRANTED request

Example of SUCCESS request

Example of ERROR request

...For feature module

...For featureprovider module