Skip to main content
Version: 1.13.0

Feature Manager AMQP API

Overview

Feature management microservice is a data repository in GeoJSON format.

Feature Management reponsabilities:

  • Extract data references thanks to [IFeatureFactoryPlugins],
  • Create, patch or delete data references,
  • Re-notify stakeholders of existing data references,
  • Delegate the storage of files (if any) to Storage Management.

To edit this repository, a data producer has to send requests.

At the moment, 2 API are available :

  • Messaging API (AMQP) allows to publish creation, reference, patch, deletion and notification requests on specific exchanges.
  • HTTP REST API allows to submit creation requests (as POST HTTP requests), update requests (as PATCH HTTP requests) or deletion requests (as DELETE HTTP requests).

Feature management architectural concepts

Under the hood, those reponsabilities are divided between two modules: featureprovider and feature. featureprovider is only responsible for handling data references extraction requests, that is extraction of information needed from physical files to create a data reference that is then handled by the feature module.

API are documented in detail below.

Request payload

Regardless of the API used, payload of each API is expected in GeoJSON format.

The basic structure is as follows :

{
"id": "FeatureId",
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [
125.6,
10.1
]
},
"properties": {
"name": "Dinagat Islands"
}
}

For the purposes of this microservice, GeoJSON structure is extended with following properties :

  • An urn (uniform resource name as unique identifier) generated by the microservice when creating a new reference and expected only when updating a reference.
  • A required model representing the name of the model defining the expected properties field structure (and previously configured).
  • A required entityType defining the reference business type.
  • An optional files property with a fixed structure that allows to store or reference physical data (service delegated to another microservice called Storage Management).
{
"id": "FeatureId",
"urn": "UniqueFeatureId",
"model": "RelatedModelName",
"entityType": "DATA",
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [
125.6,
10.1
]
},
"properties": {
"name": "Dinagat Islands"
},
"files": [
{
"locations": [
{
"storage": "DISK",
"url": "file://home/user/regards/file.zip"
}
],
"attributes": {
"dataType": "RAWDATA",
"mimeType": "application/zip",
"filename": "file.zip",
"filesize": "8013",
"algorithm": "MD5",
"checksum": "4e188bd8a6288164c25c3728ce394927"
}
}
]
}

Payload detailed properties

Feature

PathTypeDescriptionOptional
idStringId from provider
urnStringUnique feature identifer based on provider identifier with TEST:REQUEST:2342
danger

Urn is only expected in update and deletion requests!

Files

PathTypeDescriptionOptional
locations[].storageStringStoragetrue
locations[].urlStringUrl location
attributes.dataTypeStringRAWDATA, QUICKLOOK_SD, QUICKLOOK_MD, QUICKLOOK_HD, DOCUMENT, THUMBNAIL, DESCRIPTION
attributes.mimeTypeStringMIME type
attributes.filenameStringFilename
attributes.filesizeNumberFile sizetrue
attributes.algorithmStringChecksum algorithmtrue
attributes.checksumStringChecksumtrue
danger

Algorithm & cheksum are required if data have to be stored by Storage Management.

Request metadata

As the payload, regardless of the API used, metadata is often associated with a request.

Metadata detailed properties

PathTypeDescriptionOptional
metadata.overrideBooleanIndicate wether the previous version should be deletedtrue
metadata.sessionStringArbitrary session name to classify data for human operators
metadata.sessionOwnerStringArbitrary session owner to classify data for human operators
metadata.storagesArrayTarget storages if there are files to store (may be empty!)true
metadata.storages[].pluginBusinessIdStringStorage plugin identifier (previously configured in Storage Management
metadata.storages[].targetTypesArrayList of data object types accepted by this storage locationtrue
metadata.storages[].storePathStringDirectory in which to store the filetrue
metadata.priorityStringHIGH, NORMAL, LOW
danger

override should only be specified with Extraction or Creation requests.

AMQP API

Feature creation request

Request has to be published on this exchange : regards.broadcast.fr.cnes.regards.modules.feature.dto.event.in.FeatureCreationRequestEvent

With following properties:

PropertyType
metadata(look at description above)
feature(look at description above)

And following headers:

HeaderValue
regards.tenantThe tenant
regards.request.idString of max 36 characters long
regards.request.dateISO 8601 date
regards.request.ownerString of max 128 characters long

Example

# Headers
regards.tenant=project1
regards.request.id=12345463-0f6b-4488-b58c-52af3f7e9563
regards.request.owner=owner
regards.request.date=2019-07-04T01:03:00
{
"metadata": {
"sessionOwner": "owner",
"session": "session",
"storages": [
{
"pluginBusinessId": "disk"
}
],
"priority": "NORMAL"
},
"feature": {
"entityType": "DATA",
"model": "FEATURE01",
"files": [
{
"locations": [
{
"storage": null,
"url": "http://www.test.com/filename.xml"
}
],
"attributes": {
"dataType": "RAWDATA",
"mimeType": "application/xml",
"filename": "filename",
"filesize": 100,
"algorithm": "MD5",
"checksum": "checksum"
}
}
],
"id": "MyId",
"geometry": {
"coordinates": [
10.0,
20.0
],
"type": "Point",
"bbox": null,
"crs": null
},
"normalizedGeometry": null,
"properties": {
"data_type": "TYPE01",
"file_characterization": {
"valid": true
}
},
"type": "Feature"
}
}

Feature creation request by file reference

This API allows to generate feature by extracting metadata from the passed location.

Request has to be published on this exchange : regards.broadcast.fr.cnes.regards.modules.featureprovider.domain.FeatureExtractionRequestEvent

With following properties:

PropertyType
metadata(look at description above)
parametersFree JSON parameters to be used by related factory
factoryPlugin business identifier representing the feature factory to use
info

Factory must have been configured beforehand.

And following headers:

HeaderValue
regards.tenantThe tenant
regards.request.idString of max 36 characters long
regards.request.dateISO 8601 date
regards.request.ownerString of max 128 characters long

Example

# Headers
regards.tenant=project1
regards.request.id=12345463-0f6b-4488-b58c-52af3f7e9563
regards.request.owner=owner
regards.request.date=2019-07-04T01:03:00
{
"metadata": {
"sessionOwner": "owner",
"session": "session",
"storages": [
{
"pluginBusinessId": "disk"
}
],
"priority": "NORMAL"
},
"factory": "{factory identifier}",
"parameters": {
"location":"my/file/location"
}
}

Feature patch request

Request has to be published on this exchange : regards.broadcast.fr.cnes.regards.modules.feature.dto.event.in.FeatureUpdateRequestEvent

With following properties:

PropertyType
metadata.priority(look at description above)
metadata.storages(look at description above)
feature(look at description above)
info

Only properties to be updated can be passed on ... they will be merged with existing ones.

And following headers:

HeaderValue
regards.tenantThe tenant
regards.request.idString of max 36 characters long
regards.request.dateISO 8601 date
regards.request.ownerString of max 128 characters long

Example

# Headers
regards.tenant=project1
regards.request.id=12345463-0f6b-4488-b58c-52af3f7e9563
regards.request.owner=owner
regards.request.date=2019-07-04T01:03:00
{
"metadata": {
"storages": [],
"priority": "NORMAL"
},
"feature": {
"urn": "URN:FEATURE:DATA:tenant:87fdda0e-27d1-494f-a1c0-c57f2f0810f7:V1",
"entityType": "DATA",
"model": "FEATURE01",
"files": [],
"id": "MyId",
"geometry": null,
"normalizedGeometry": null,
"properties": {
"file_characterization": {
"invalidation_date": "2019-12-03T12:31:42.466Z",
"valid": false
}
},
"type": "Feature"
}
}

Feature deletion request

Request has to be published on this exchange : regards.broadcast.fr.cnes.regards.modules.feature.dto.event.in.FeatureDeletionRequestEvent

PropertyType
priority(look at description above)
urnUnique identifier of the feature

With following headers:

HeaderValue
regards.tenantThe tenant
regards.request.idString of max 36 characters long
regards.request.dateISO 8601 date
regards.request.ownerString of max 128 characters long

Example

# Headers
regards.tenant=project1
regards.request.id=78938463-023a-4488-b58c-52af3f7e9446
regards.request.owner=owner
regards.request.date=2020-03-17T12:24:05.995Z
{
"priority": "NORMAL",
"urn": "URN:FEATURE:DATA:tenant:87fdda0e-27d1-494f-a1c0-c57f2f0810f7:V1"
}

AMQP monitoring API...

Feature manager microservice publishes AMQP messages to monitor request lifecycle.

Messages are published to two different exchanges because under the hood Feature manager is in reality composed of two modules: feature and featureprovider. Whatever the exchange to which those monitoring messages are comming from, they have the same structure, that is:

  • The requestId (corresponds to regards.request.id header),
  • The requestOwner (corresponds to regards.request.owner header),
  • The related feature id (not specified for EXTRACTION requests because feature is not yet created),
  • The related feature urn (not specified for EXTRACTION requests because feature is not yet created),
  • The request type (EXTRACTION,CREATION,PATCH,DELETION,NOTIFICATION,FILE_COPY,SAVE_METADATA)
  • The state of the request (GRANTED,DENIED,ERROR or SUCCESS),
  • A list of errors if any.
info

To receive these messages, your have to subscribe to this exchange.

info

EXTRACTION process does not create features. So if you want to know the created feature id or urn, you have to listen to responses from the CREATION process which has the same requestId than the corresponding EXTRACTION request.

caution

If AMQP virtual host mode is set to SINGLE, this exchange will receive all messages of all tenants! So the receiver may have to filter them according to the tenant he wishes to manage.

caution

Requests without requestId can be routed to AMQP Dead Letter Queue (DLQ) so no response will be published. However, a notification is sent to the administrators of the project in order to tell them to inspect DLQ.

Example of DENIED request

|:

{
"requestId": "{requestId}",
"requestOwner": "{requestOwner}",
"id": "{featureId}",
"type": "{requestType}",
"state": "DENIED",
"errors": ["error1", "error2"]
}

Example of GRANTED request

{
"requestId": "{requestId}",
"requestOwner": "{requestOwner}",
"id": "{featureId}",
"type": "{requestType}",
"state": "GRANTED"
}

Example of SUCCESS request

{
"requestId": "{requestId}",
"requestOwner": "{requestOwner}",
"id": "{featureId}",
"urn": "{featureGeneratedUrn}",
"type": "{requestType}",
"state": "SUCCESS"
}

Example of ERROR request

{
"requestId": "{requestId}",
"requestOwner": "{requestOwner}",
"id": "{featureId}",
"urn": "{featureGeneratedUrn}",
"type": "{requestType}",
"state": "ERROR",
"errors": ["error1", "error2"]
}

...For feature module

Messages are published on this exchange : regards.broadcast.fr.cnes.regards.modules.feature.dto.event.out.FeatureRequestEvent

...For featureprovider module

Messages are published on this exchange : regards.broadcast.fr.cnes.regards.modules.featureprovider.domain.FeatureExtractionResponseEvent

In other words, it means that to migrate from V1.3.x to V1.4.0, you need to change the exchange to which you subscribe to receive extraction monitoring messages. Or you might be able to configure your AMQP broker to automatically redirect messages from this exchange to the old one (regards.broadcast.fr.cnes.regards.modules.feature.dto.event.out.FeatureRequestEvent).