Version: 9.5.x

Single View Creator concepts

Single View Creator is the service in charge of keeping updated the Single View with the data retrieved from the projections. This service is available as a plugin or as a template:

plugin: allows you to use the single view creator as a black-box. You just need to configure it through the Config Maps and environment variables
template: gives you access to the source code of the single view creator because of it will be created a repository on Git with its code. You will need to update its dependencies by yourself and maintains its code.

We strongly recommend using the plugin. Template is supposed to be used only for advanced use-case where the plugin can not be used.

Single View Creator plugin can be used in two modes:

Low Code: allows you to perform aggregation through JSON without writing any Javascript code in most cases. If you need custom behavior you will still be able to writing your own code for just that piece of aggregation.
Manual: allows you to perform aggregation writing your own Javascript code.

We recommend using the Low Code since it allows you to be faster and safer in aggregating your data. You will just need to think about the data and not the code for doing so.
Manual mode is supposed to be used only for cases where Low Code can not be used, but it should rarely happen, since if you are using Low Code you will still be able to writing custom Javascript function for the specific piece of aggregation that Low Code can not perform.

Getting started

Plugin

Go to the Marketplace and create a Single View Creator or Single View Creator - Low Code plugin. Go to the microservice page of the one created and set the correct values to the environment variables containing a placeholder.

Template

Search in the Marketplace for a Single View Creator - Template and create it. Then go to the microservice page of the one created and set the correct values to the environment variables containing a placeholder. Click on the repository link in the microservice page and clone on your computer the repository.

info

You can use the template and all the Mia-Platform libraries only under license. For further information contact your Mia Platform referent

Code overview

The service starts in index.js file. First, the template uses the Custom Plugin Lib to instantiate a service. Inside its callback, the single-view-creator-lib is initialized to deal with the complexity of the Fast Data components.

const singleViewCreator = getSingleViewCreator(log, config, customMetrics)

await singleViewCreator.initEnvironment() // connect Mongo, Kafka and create the patient instance
service.decorate('patient', singleViewCreator.k8sPatient)

Where config is an object whose fields represent the Microservice environment variables.

Some environment variables will be pre-compiled when you create the service from template, others won't, but they will still have a placeholder as value. Replace it with the correct value.

Now, we start the single-view-creator:

const resolvedOnStop = singleViewCreator.startCustom({
  strategy: aggregatorBuilder(projectionsDB),
  mapper,
  validator,
  singleViewKeyGetter: singleViewKey,
  upsertSingleView: upsertFnSv(),
  deleteSingleView: deleteSV(),
})

strategy is the function that performs the aggregation over the projections
mapper is the function that takes as input the raw aggregation result and maps the data to the final Single View
validator is the validation function which determines if the Single View is valid (and thus inserted or updated in Mongo) or not (and thus deleted)
singleViewKeyGetter is the function that, given the projections changes identifier, returns the data used as selector to find the single view document on Mongo to update or delete
upsertFnSv is the function that updates or inserts the Single View to the Single Views collection on Mongo
deleteSingleView is the function that deletes the Single View from the Single Views collection on Mongo. It's used the deleteSV exported by the library.

note

The deleteSV function makes a real delete of the document on MongoDB. So, unlike the projections deletion, it does not make a virtual delete.

The value of upsertFnSv is based on the UPSERT_STRATEGIES environment variable. If its value is update, then the updateOrInsertSV function exported by the library is used, otherwise the function replaceOrInsertSV is used instead. The default upsert strategy is replace.

note

In the versions of the template prior to the v3.1.0, the UPSERT_STRATEGIES was missing, and it was used an alias function (upsertSV) of the replaceOrInsertSV.

The Single View creator needs to be stopped when the process is stopping. To do that, we use the onClose hook:

service.addHook('onClose', async() => {
  log.fatal({ type: 'END' }, 'Single View Creator is stopping...')
  await singleViewCreator.stop()

  // this is a promise resolved when the infinite loop which processes the single views ends.
  // Here we wait for the resolving of the promise. You don't need to call it.
  await resolvedOnStop
  log.fatal({ type: 'END' }, 'Single View Creator stopped')
  await mongoClient.close()
})

info

You can use the template and all the Mia-Platform libraries only under license. For further information contact your Mia Platform referent

Environment Variables

Name	Required	Description	Default value
CONFIGURATION_FOLDER	false	Folder where configuration files are mounted	/home/node/app/src/
LOG_LEVEL	true	Level to use for logging	-
HTTP_PORT	false	Port exposed by the service	3000
TYPE	true	Identifies the type of projection changes that need to be read. It should be the same as the Single View name you want to update.	-
SCHEDULING_TIME	false	a quantity of time in milliseconds, every X milliseconds the service wake up and check if there are some projections changes in `NEW` state to work on. The service continue working until no more new projections changes are found, if so he goes to sleep for X milliseconds.	60000
PROJECTIONS_MONGODB_URL	true	MongoDB connection string where projections are stored. Must be a valid uri	-
SINGLE_VIEWS_MONGODB_URL	true	MongoDB connection string where single view must be stored. Must be a valid uri	-
PROJECTIONS_CHANGES_MONGODB_URL	false	The db from where projections changes are read.	value of PROJECTIONS_MONGODB_URL
PROJECTIONS_CHANGES_DATABASE	true	The db from where projections changes are read.	-
PROJECTIONS_DATABASE	true	The db from where projections are read.	value of PROJECTIONS_CHANGES_DATABASE
PROJECTIONS_CHANGES_COLLECTION	false	if you have set a custom projection change collection name from advanced, then set its name. Otherwise, it is `fd-pc-SYSTEM_ID` where `SYSTEM_ID` is the id of the System of Records this single view creator is responsible for.	-
SINGLE_VIEWS_DATABASE	true	The db from where single views are written.	-
SINGLE_VIEWS_COLLECTION	true	It must be equals to the Single View name the service is in charge of keeping updated.	-
SINGLE_VIEWS_PORTFOLIO_ORIGIN	true	should be equals to the `SYSTEM_ID` you have set in `PROJECTIONS_CHANGES_COLLECTION`	-
SINGLE_VIEWS_ERRORS_COLLECTION	true	Name of a MongoDB CRUD you want to use as collection for single view errors.	-
KAFKA_CONSUMER_GROUP_ID	false	@deprecated - in favor of KAFKA_GROUP_ID. The Kafka consumer group identifier	-
KAFKA_GROUP_ID	true	defines the Kafka group id (it is suggested to use a syntax like `{tenant}.{environment}.{projectName}.{system}.{singleViewName}.single-view-creator`)	-
KAFKA_CLIENT_ID	false	The Kafka client identifier	-
KAFKA_BROKERS_LIST	false	@deprecated - in favor of KAFKA_BROKERS. list of brokers the service needs to connect to	-
KAFKA_BROKERS	false	list of brokers the service needs to connect to	-
KAFKA_SASL_MECHANISM	false	The Kafka SASL mechanism to be used. Can be one of the following: "plain", "PLAIN", "scram-sha-256", "SCRAM-SHA-256", "scram-sha-512", "SCRAM-SHA-512"	plain
KAFKA_SASL_USERNAME	false	username to use for logging into Kafka	-
KAFKA_SASL_PASSWORD	false	password to use for logging into Kafka	-
KAFKA_SVC_EVENTS_TOPIC	false	topic used to queue Single View Creator state changes (e.g. single view creation). This feature is deprecated in favor of KAFKA_SV_UPDATE_TOPIC and it will be removed soon	-
SEND_BA_TO_KAFKA	false	true if you want to send to Kafka the `before-after` information about the update changes of the single view. This feature is deprecated in favor of ADD_BEFORE_AFTER_CONTENT using the 'sv-update' event and it will be removed soon	false
KAFKA_BA_TOPIC	false	topic where to send the `before-after` messages which represent the single view document before and after a change. This feature is deprecated in favor of ADD_BEFORE_AFTER_CONTENT using the 'sv-update' event and it will be removed soon	-
SEND_SV_UPDATE_TO_KAFKA	false	true if you want to send to Kafka the `sv-update` message about the update changes of the single view	false
ADD_BEFORE_AFTER_CONTENT	false	true if you want to add the _before_ and _after_ content to the `sv-update` message, works only if `SEND_SV_UPDATE_TO_KAFKA` is set to true	false
KAFKA_SV_UPDATE_TOPIC	false	topic where to send the `sv-update` message	-
UPSERT_STRATEGY	false	(v3.1.0 or higher) If it is set to "replace", the whole Single View document will be replaced with the new one. If it is set to "update", the existing one will be updated with the new one, but fields not present in the latter will be kept. Otherwise the custom function placed at the path equals to the value of this env will be used.	replace
DELETE_STRATEGY	false	(v3.1.0 or higher) If it is set to "delete", the whole Single View document will be deleted. Otherwise the custom function placed at the path equals to the value of this env will be used.	delete
SINGLE_VIEWS_MAX_PROCESSING_MINUTES	false	(v3.4.2 or higher) time to wait before processing again a Projection with state IN_PROGRESS	30
CA_CERT_PATH	false	the path to the CA certificate, which should include the file name as well, e.g. `/home/my-ca.pem`	-
ER_SCHEMA_FOLDER	false	the path to the ER Schema folder, e.g. `/home/node/app/erSchema`	-
AGGREGATION_FOLDER	false	the path to the Aggregation folder, e.g. `/home/node/app/aggregation`	-
USE_AUTOMATIC	false	wheather to use the low code architecture for the Single View Creator service or not	-
PROJECTIONS_CHANGES_SOURCE	false	System to use to handle the Projection Changes, supported methods are KAFKA or MONGO	MONGO

If you do not want to use Kafka in the Single View Creator, you can just not set the environment variable KAFKA_CLIENT_ID or KAFKA_BROKERS. If one of them is missing, Kafka will not be configured by the service (requires single-view-creator-lib v9.1.0 or higher)

Single View Key

Single View Key is the Single View Creator part in charge of identifying the single view document that needs to be updated as consequence of the event that the service has consumed.

If you are using Low Code, please visit here, otherwise you can go to the manual documentation

Aggregation

Aggregation is the Single View Creator part in charge of aggregating data of projections and generating the Single View that is going to be updated.

If you are using Low Code, please visit here, otherwise you can go to the manual documentation

note

Since version v5.0.0 of Single View Creator service and v12.0.0 of the @mia-platform-internal/single-view-creator-lib, returning a single view with the field __STATE__ from the aggregation will update the Single View to that state (among the others changes).
This means, for instance, that if you set the __STATE__ to DRAFT in the aggregation.json in LowCode (or in the pipeline.js in the Manual), the single view updated will have the STATE equals to DRAFT. Previously, the __STATE__ you returned was ignored, and the Single View would always have the STATE equals to PUBLIC.

Validator

The validation of a Single View determines what to do with the current update. If the single view is determined as "non-valid", the delete function will be called. Otherwise, if the result of the validation is positive, it will be updated or inserted in the Single Views collection, through the upsert function. Delete function and upsert function will be explained in the next paragraph.

For this reason, the validation procedure should not be too strict, since a Single View declared as "invalid" would not be updated or inserted to the database. Rather, the validation is a check operation to determine if the current Single View should be handled with the upsert or delete functions.

By default, the validator returns always true. So we accept all kinds of single views, but, if you need it, you can set your own custom validator.

// (logger: BasicLogger, singleView: Document) => Boolean
function singleViewValidator(logger, singleView) {
  ... checks on singleView

  // returns a boolean
  return validationResult
}

danger

When the update of an existing Single View is triggered and the validation has a negative outcome, the Single View won't be updated, and instead it will be deleted.

Plugin

Since version v3.5.0, it is possible to specify a custom validator function inside the configuration folder (CONFIGURATION_FOLDER).

The file must be named validator.js and must export a function that will take as arguments the same as the default validator explained above.

validator.js
module.exports = function validator(logger, singleView) {
  ... custom validation logic on singleView

  // returns a boolean
  return customValidationResult
}

Template

The startCustom function accepts a function in the configuration object called validator, which is the validation function.

The input fields of the validation function are the logger and the Single View, while the output is a boolean containing the result of the validation.

Customize Upsert and Delete functions

If you want, you can replace both upsert and delete functions with your own custom functions to perform those operations.

These functions represent the last step of the creation (or deletion) of a Single View, in which the Single View collection is actually modified.

In case the validation is succeeded, the upsert function will be called with the following arguments:

logger is the logger
singleViewCollection is the Mongo collection object
singleView is the result of the mapping operation
singleViewKey is the Single View key

On the other hand, if the validation has a negative result, the delete function will be called with the same arguments, except for the singleView, which will not be handled by the delete function.

In both cases, some operation should be done on singleViewCollection in order to modify the Single View with the current singleViewKey, with the idea of "merging" the current result with the one already present in the database.

For example, we have a Customer Single View with a list of products the customer bought from different e-commerce, and we receive an update for a new object on a specific shop, in that case we don't want to replace the list of bought products with the last one arrived, but we want to push the product in the list in order to have the complete history of purchases.

For both functions, the output is composed of an object containing two fields:

old which contains the old Single View
new which contains the new Single View

These values will respectively be the before and the after of the message sent to the KAFKA_BA_TOPIC topic, that is the topic responsible for tracking any result of the Single View creator. The naming convention for this topic can be found here.

async function upsertSingleViewFunction(
  logger,
  singleViewCollection,
  singleView,
  singleViewKey)
{
  logger.trace('Upserting Single View...')
  const oldSingleView = await singleViewCollection.findOne(singleViewKey)

  await singleViewCollection.replaceOne(
    singleViewKey,
    singleView,
    { upsert: true }
  )

  logger.trace({ isNew: Boolean(oldSingleView) }, 'Updated Single View')
  return {
    old: oldSingleView,
    new: singleView,
  }
}

async function deleteSingleViewFunction(
  logger,
  singleViewCollection,
  singleViewKey)
{
  logger.trace('Deleting Single View...')
  const oldSingleView = await singleViewCollection.findOne(singleViewKey)

  if (oldSingleView !== null) {
    try {
      await singleViewCollection.deleteOne(singleViewKey)
    } catch (ex) {
      logger.error(`Error during Single View delete: ${ex}`)
    }
  }

  logger.trace('Single view deletion procedure terminated')
  return {
    old: oldSingleView,
    new: null,
  }
}

Plugin

Add a config map to your service and put the Javascript files into it containing the custom function you want to use as upsert or delete function.

For instance:

myDeleteFunction.js
module.exports = async function myDeleteFunction(
  logger,
  singleViewCollection,
  singleViewKey)
{
  logger.trace('Checking if it can be deleted...')
  const oldSingleView = await singleViewCollection.findOne(singleViewKey)

  // my custom logic
  // do something...

  if (oldSingleView !== null) {
    
    try {
      await singleViewCollection.deleteOne(singleViewKey)
    } catch (ex) {
      logger.error(`Error during Single View delete: ${ex}`)
    }
  }

  logger.trace('Single view deletion procedure terminated')
  return {
    old: oldSingleView,
    new: null,
  }
}

Let's suppose that I put this file in a config map mounted on path /home/node/app/my-functions. Then, in order to use that, I need to set DELETE_STRATEGY environment variables equals to /home/node/app/my-functions/myDeleteFunction.js.

The same logic can be applied to upsert function, but setting the file path to the environment variable UPSERT_STRATEGY.

Template

You can whether to choose if applying the same pattern used in plugin (by setting the environment variables) or by passing your custom functions directly to the startCustom method.

index.js
const resolvedOnStop = singleViewCreator.startCustom({  strategy: aggregatorBuilder(projectionsDB),  mapper,  validator,  singleViewKeyGetter: singleViewKey,  upsertSingleView: upsertFnSV,  deleteSingleView: deleteSV,})

Error handling

When generating a Single View, every error that occurs is saved in MongoDB, with a format that satisfies the schema requirements of the CRUD service, so that you can handle those errors using the Console. The fields of the error messages when they are first created are:

_id: a unique identifier of the record, automatically generated
portfolioOrigin: a value concerning the origin of the error, defaults to UNKNOWN_PORTFOLIO_ORIGIN
type: the Single View type
identifier: id of the projection changes
errorType: the error details
createdAt: time of creation
creatorId: set to single-view-creator
__STATE__: set to PUBLIC
updaterId: set to single-view-creator
updatedAt: time of creation

It is highly recommended using a TTL index to enable the automatic deletion of older messages, which can be done directly using the Console, as explained here.

CA certs

Since service version 3.9.0, you can set your CA certs by providing a path to the certification file in the environment variable CA_CERT_PATH.

Single View Creator concepts

Getting started​

Plugin​

Template​

Code overview​

Environment Variables​

Single View Key​

Aggregation​

Validator​

Plugin​

Template​

Customize Upsert and Delete functions​

Plugin​

Template​

Error handling​

CA certs​

Getting started

Plugin

Template

Code overview

Environment Variables

Single View Key

Aggregation

Validator

Plugin

Template

Customize Upsert and Delete functions

Plugin

Template

Error handling

CA certs