Feature Manager AMQP API
Overview
Feature management
microservice is a data repository in GeoJSON format.
Feature Management
reponsabilities:
- Extract data references thanks to [IFeatureFactoryPlugins],
- Create, patch or delete data references,
- Re-notify stakeholders of existing data references,
- Delegate the storage of files (if any) to
Storage Management
.
To edit this repository, a data producer has to send requests.
At the moment, 2 API are available :
- Messaging API (AMQP) allows to publish creation, reference, patch, deletion and notification requests on specific exchanges.
- HTTP REST API allows to submit creation requests (as
POST
HTTP requests), update requests (asPATCH
HTTP requests) or deletion requests (asDELETE
HTTP requests).
Under the hood, those reponsabilities are divided between two modules: featureprovider
and feature
. featureprovider
is only responsible for handling data references extraction requests, that is extraction of information needed from physical files to create a data reference that is then handled by the feature
module.
API are documented in detail below.
Request payload
Regardless of the API used, payload of each API is expected in GeoJSON format.
The basic structure is as follows :
- A required
id
, - A required
type
with valueFeature
, - An optional geometry in GeoJSON format,
- An optional set of
properties
.
{
"id": "FeatureId",
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [
125.6,
10.1
]
},
"properties": {
"name": "Dinagat Islands"
}
}
For the purposes of this microservice, GeoJSON structure is extended with following properties :
- An
urn
(uniform resource name as unique identifier) generated by the microservice when creating a new reference and expected only when updating a reference. - A required
model
representing the name of the model defining the expectedproperties
field structure (and previously configured). - A required
entityType
defining the reference business type. - An optional
files
property with a fixed structure that allows to store or reference physical data (service delegated to another microservice calledStorage Management
).
{
"id": "FeatureId",
"urn": "UniqueFeatureId",
"model": "RelatedModelName",
"entityType": "DATA",
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [
125.6,
10.1
]
},
"properties": {
"name": "Dinagat Islands"
},
"files": [
{
"locations": [
{
"storage": "DISK",
"url": "file://home/user/regards/file.zip"
}
],
"attributes": {
"dataType": "RAWDATA",
"mimeType": "application/zip",
"filename": "file.zip",
"filesize": "8013",
"algorithm": "MD5",
"checksum": "4e188bd8a6288164c25c3728ce394927"
}
}
]
}
Payload detailed properties
Feature
Path | Type | Description | Optional |
---|---|---|---|
id | String | Id from provider | |
urn | String | Unique feature identifer based on provider identifier with TEST:REQUEST:2342 |
Urn is only expected in update and deletion requests!
Files
Path | Type | Description | Optional |
---|---|---|---|
locations[].storage | String | Storage | true |
locations[].url | String | Url location | |
attributes.dataType | String | RAWDATA, QUICKLOOK_SD, QUICKLOOK_MD, QUICKLOOK_HD, DOCUMENT, THUMBNAIL, DESCRIPTION | |
attributes.mimeType | String | MIME type | |
attributes.filename | String | Filename | |
attributes.filesize | Number | File size | true |
attributes.algorithm | String | Checksum algorithm | true |
attributes.checksum | String | Checksum | true |
Algorithm & cheksum are required if data have to be stored by Storage Management
.
Request metadata
As the payload, regardless of the API used, metadata is often associated with a request.
Metadata detailed properties
Path | Type | Description | Optional |
---|---|---|---|
metadata.override | Boolean | Indicate wether the previous version should be deleted | true |
metadata.session | String | Arbitrary session name to classify data for human operators | |
metadata.sessionOwner | String | Arbitrary session owner to classify data for human operators | |
metadata.storages | Array | Target storages if there are files to store (may be empty!) | true |
metadata.storages[].pluginBusinessId | String | Storage plugin identifier (previously configured in Storage Management | |
metadata.storages[].targetTypes | Array | List of data object types accepted by this storage location | true |
metadata.storages[].storePath | String | Directory in which to store the file | true |
metadata.priority | String | HIGH, NORMAL, LOW |
override should only be specified with Extraction or Creation requests.
AMQP API
Feature creation request
Request has to be published on this exchange : regards.broadcast.fr.cnes.regards.modules.feature.dto.event.in.FeatureCreationRequestEvent
With following properties:
Property | Type |
---|---|
metadata | (look at description above) |
feature | (look at description above) |
And following headers:
Header | Value |
---|---|
regards.tenant | The tenant |
regards.request.id | String of max 36 characters long |
regards.request.date | ISO 8601 date |
regards.request.owner | String of max 128 characters long |
Example
# Headers
regards.tenant=project1
regards.request.id=12345463-0f6b-4488-b58c-52af3f7e9563
regards.request.owner=owner
regards.request.date=2019-07-04T01:03:00
{
"metadata": {
"sessionOwner": "owner",
"session": "session",
"storages": [
{
"pluginBusinessId": "disk"
}
],
"priority": "NORMAL"
},
"feature": {
"entityType": "DATA",
"model": "FEATURE01",
"files": [
{
"locations": [
{
"storage": null,
"url": "http://www.test.com/filename.xml"
}
],
"attributes": {
"dataType": "RAWDATA",
"mimeType": "application/xml",
"filename": "filename",
"filesize": 100,
"algorithm": "MD5",
"checksum": "checksum"
}
}
],
"id": "MyId",
"geometry": {
"coordinates": [
10.0,
20.0
],
"type": "Point",
"bbox": null,
"crs": null
},
"normalizedGeometry": null,
"properties": {
"data_type": "TYPE01",
"file_characterization": {
"valid": true
}
},
"type": "Feature"
}
}
Feature creation request by file reference
This API allows to generate feature by extracting metadata from the passed location.
Request has to be published on this exchange : regards.broadcast.fr.cnes.regards.modules.featureprovider.domain.FeatureExtractionRequestEvent
With following properties:
Property | Type |
---|---|
metadata | (look at description above) |
parameters | Free JSON parameters to be used by related factory |
factory | Plugin business identifier representing the feature factory to use |
Factory must have been configured beforehand.
And following headers:
Header | Value |
---|---|
regards.tenant | The tenant |
regards.request.id | String of max 36 characters long |
regards.request.date | ISO 8601 date |
regards.request.owner | String of max 128 characters long |
Example
# Headers
regards.tenant=project1
regards.request.id=12345463-0f6b-4488-b58c-52af3f7e9563
regards.request.owner=owner
regards.request.date=2019-07-04T01:03:00
{
"metadata": {
"sessionOwner": "owner",
"session": "session",
"storages": [
{
"pluginBusinessId": "disk"
}
],
"priority": "NORMAL"
},
"factory": "{factory identifier}",
"parameters": {
"location":"my/file/location"
}
}
Feature patch request
Request has to be published on this exchange : regards.broadcast.fr.cnes.regards.modules.feature.dto.event.in.FeatureUpdateRequestEvent
With following properties:
Property | Type |
---|---|
metadata.priority | (look at description above) |
metadata.storages | (look at description above) |
feature | (look at description above) |
Only properties to be updated can be passed on ... they will be merged with existing ones.
And following headers:
Header | Value |
---|---|
regards.tenant | The tenant |
regards.request.id | String of max 36 characters long |
regards.request.date | ISO 8601 date |
regards.request.owner | String of max 128 characters long |
Example
# Headers
regards.tenant=project1
regards.request.id=12345463-0f6b-4488-b58c-52af3f7e9563
regards.request.owner=owner
regards.request.date=2019-07-04T01:03:00
{
"metadata": {
"storages": [],
"priority": "NORMAL"
},
"feature": {
"urn": "URN:FEATURE:DATA:tenant:87fdda0e-27d1-494f-a1c0-c57f2f0810f7:V1",
"entityType": "DATA",
"model": "FEATURE01",
"files": [],
"id": "MyId",
"geometry": null,
"normalizedGeometry": null,
"properties": {
"file_characterization": {
"invalidation_date": "2019-12-03T12:31:42.466Z",
"valid": false
}
},
"type": "Feature"
}
}
Feature deletion request
Request has to be published on this exchange : regards.broadcast.fr.cnes.regards.modules.feature.dto.event.in.FeatureDeletionRequestEvent
Property | Type |
---|---|
priority | (look at description above) |
urn | Unique identifier of the feature |
With following headers:
Header | Value |
---|---|
regards.tenant | The tenant |
regards.request.id | String of max 36 characters long |
regards.request.date | ISO 8601 date |
regards.request.owner | String of max 128 characters long |
Example
# Headers
regards.tenant=project1
regards.request.id=78938463-023a-4488-b58c-52af3f7e9446
regards.request.owner=owner
regards.request.date=2020-03-17T12:24:05.995Z
{
"priority": "NORMAL",
"urn": "URN:FEATURE:DATA:tenant:87fdda0e-27d1-494f-a1c0-c57f2f0810f7:V1"
}
AMQP monitoring API...
Feature manager
microservice publishes AMQP messages to monitor request lifecycle.
Messages are published to two different exchanges because under the hood Feature manager
is in reality composed of two modules: feature
and featureprovider
. Whatever the exchange to which those monitoring messages are comming from, they have the same structure, that is:
- The
requestId
(corresponds toregards.request.id
header), - The
requestOwner
(corresponds toregards.request.owner
header), - The related feature
id
(not specified forEXTRACTION
requests because feature is not yet created), - The related feature
urn
(not specified forEXTRACTION
requests because feature is not yet created), - The request type (
EXTRACTION
,CREATION
,PATCH
,DELETION
,NOTIFICATION
,FILE_COPY
,SAVE_METADATA
) - The state of the request (
GRANTED
,DENIED
,ERROR
orSUCCESS
), - A list of
errors
if any.
To receive these messages, your have to subscribe to this exchange.
EXTRACTION
process does not create features. So if you want to know the created feature id or urn, you have to listen to responses from the CREATION
process which has the same requestId
than the corresponding EXTRACTION
request.
If AMQP virtual host mode is set to SINGLE
, this exchange will receive all messages of all tenants! So the receiver may have to filter them according to the tenant he wishes to manage.
Requests without requestId
can be routed to AMQP Dead Letter Queue (DLQ) so no response will be published. However, a notification is sent to the administrators of the project in order to tell them to inspect DLQ.
Example of DENIED request
|:
{
"requestId": "{requestId}",
"requestOwner": "{requestOwner}",
"id": "{featureId}",
"type": "{requestType}",
"state": "DENIED",
"errors": ["error1", "error2"]
}
Example of GRANTED request
{
"requestId": "{requestId}",
"requestOwner": "{requestOwner}",
"id": "{featureId}",
"type": "{requestType}",
"state": "GRANTED"
}
Example of SUCCESS request
{
"requestId": "{requestId}",
"requestOwner": "{requestOwner}",
"id": "{featureId}",
"urn": "{featureGeneratedUrn}",
"type": "{requestType}",
"state": "SUCCESS"
}
Example of ERROR request
{
"requestId": "{requestId}",
"requestOwner": "{requestOwner}",
"id": "{featureId}",
"urn": "{featureGeneratedUrn}",
"type": "{requestType}",
"state": "ERROR",
"errors": ["error1", "error2"]
}
...For feature module
Messages are published on this exchange : regards.broadcast.fr.cnes.regards.modules.feature.dto.event.out.FeatureRequestEvent
...For featureprovider module
Messages are published on this exchange : regards.broadcast.fr.cnes.regards.modules.featureprovider.domain.FeatureExtractionResponseEvent
In other words, it means that to migrate from V1.3.x to V1.4.0, you need to change the exchange to which you subscribe to receive extraction monitoring messages. Or you might be able to configure your AMQP broker to automatically redirect messages from this exchange to the old one (regards.broadcast.fr.cnes.regards.modules.feature.dto.event.out.FeatureRequestEvent
).