Each Single View needs a dedicated Microservice. This service will listen on the Projections changes that affect the Single View and consequently updates its data. This service is the
Single View Creator.
To start configuring your own Single View Creator, you can begin from the Single View Creator Template.
If you are searching for a ready-to-use plugin that only requires some configuration files to update your single views, please visit this link.
The service starts in
First of all, the template uses the Custom Plugin Lib to instantiate a service.
Inside its callback, the
single-view-creator-lib is initialized to deal with the complexity of the Fast Data components.
config is an object whose fields represent the Microservice environment variables.
Some environment variables will be pre-compiled when you create the service from template, others won't, but they will still have a placeholder as value. Replace it with the correct value.
Here some tips:
TYPE: should be the name of the single view which your single view creator is responsible for
SINGLE_VIEWS_COLLECTION: should be the name of the single view which your single view creator is responsible for
PROJECTIONS_CHANGES_COLLECTION: if you have set a custom projection change collection name from advanced, then set its name. Otherwise it is
SYSTEM_IDis the id of the System of Records this single view creator is responsible for.
SINGLE_VIEWS_PORTFOLIO_ORIGIN: should be equals to the
SYSTEM_IDyou have set in
SINGLE_VIEWS_ERRORS_COLLECTION: it is the name of a MongoDB Crud you want to use as collection for single view errors.
KAFKA_BA_TOPIC: topic where to send the
before-after, which is the single view document before and after a change
SEND_BA_TO_KAFKA: true if you want to send to Kafka the
before-afterinformation about the update changes of the single view
KAFKA_SVC_EVENTS_TOPIC: topic used to queue Single View Creator state changes (e.g. single view creation)
Now, we start the single-view-creator:
strategyis the function that performs the aggregation over the projections
mapperis the function that takes as input the raw aggregation result and maps the data to the final Single View
validatoris the validation function which determines if the Single View is valid (and thus inserted or updated in Mongo) or not (and thus deleted)
singleViewKeyGetteris the function that, given the projections changes identifier, returns the data used as selector to find the single view document on mongo to update or delete
upsertSingleViewis the function that updates or inserts the Single View to the Single Views collection on Mongo
deleteSingleViewis the function that deletes the Single View from the Single Views collection on Mongo
deleteSV are two utility functions that the library exports that handle the upsert and the delete of the single view.
deleteSV function makes a real delete of the document on MongoDb. So, unlike the projections deletion, it does not make a virtual delete.
The Single View creator needs to be stopped when the process is stopping. To do that, we use the
You can use the template and all the Mia-Platform libraries only under license. For further information contact your Mia Platform referent
This documentation refers to the
The core of your work on this service are the files inside the
singleViewKey.js: It takes as input the identifier of the projection change and returns the key object used to select the document of the Single View collection that needs to be updated. This key corresponds to the query object fed to mongodb, therefore you can return any legal mongo query.
In the example below, we expect to have the field
myId as primary key of the Single View collection.
pipeline.js: It takes as input a MongoDB instance and returns a function. This function takes as input the projection change identifier and returns an array.
If it is empty, a delete operation will be executed on the single view identified by the
If it is not empty, an upsert operation will be executed on the single view identified by the
If the pipeline returns an array with more than one element, only the first element will be used for the upsert.
mapper.js: It is a function that takes as argument the first element (if defined) of the result of the pipeline, and returns an object containing the value updated for the single view. The object returned should match the schema of the single view.
Inside the mapper a renaming and repositioning of the fields can be applied.
We suggest to implement inside the mapper all the aggregation logic that can be reused for all the clients that will read the Single Views, they should be as generic as possible. It is a good practice to have some calculation and aggregation logic inside Single View Creator as far as it is reusable. If you have to apply some custom logic try to do it inside and API Adapter specific for the client.
startCustom function accepts a function in the configuration object called
validator, which is the validation function.
The validation of a Single View determines what to do with the current update. If the single view is determined as "non-valid", the delete function will be called. Otherwise, if the result of the validation is positive, it will be updated or inserted in the Single Views collection, through the upsert function. Delete function and upsert function will be explained in the next paragraph.
For this reason, the validation procedure should not be too strict, since a Single View declared as "invalid" would not be updated or inserted to the database. Rather, the validation is a check operation to determine if the current Single View should be handled with the upsert or delete functions.
By default, in this template we set as validator a function that returns always true. So we accept all kinds of single views, but, if you need it, you can replace that function with your own custom validator.
The input fields of the validation function are the logger and the Single View, while the output is a boolean containing the result of the validation.
If you want, you can replace both
deleteSV with your own custom functions to perform those operations.
These functions represents the last step of the creation (or deletion) of a Single View, in which the Single View collection is actually modified.
In case the validation is succeeded, the upsert function will be called with the following arguments:
loggeris the logger
singleViewCollectionis the the Mongo collection object
singleViewis the result of the mapping operation
singleViewKeyis the Single View key
On the other hand, if the validation has a negative result, the delete function will be called with the same arguments, except for the
singleView, which will not be handled by the delete function.
In both cases, some operation should be done on
singleViewCollection in order to modify the Single View with the current
singleViewKey, with the idea of "merging" the current result with the one already present in the database.
For example, we have a Customer Single View with a list of products the customer bought from different e-commerce and we receive an update for a new object on a specific shop, in that case we don't want to replace the list of bought products with the last one arrived but we want to push the product in the list in order to have the complete history of purchases.
For both functions, the output is composed of an object containing two fields:
oldwhich contains the old Single View
newwhich contains the new Single View
These values will be the respectively the
after of the message sent to the
KAFKA_BA_TOPIC topic, that is the topic responsible for tracking any result of the Single View creator.
When generating a Single View, every error that occurs is saved in MongoDb, with a format that satisfies the schema requirements of the crud service, so that you can handle those errors using the Console. The fields of the error messages when they are first created are:
_id: a unique identifier of the record, automatically generated
portfolioOrigin: a value concerning the origin of the error, defaults to
type: the Single View type
identifier: id of the projection changes
errorType: the error details
createdAt: time of creation
creatorId: set to
__STATE__: set to
updaterId: set to
updatedAt: time of creation
It is highly recommended to use a TTL index to enable the automatic deletion of older messages, which can be done directly using the Console, as explained here.