This Plugin is a BETA Plugin and, as such, is currently under active development. Pay attention using it.
The Single View Trigger Generator is a new component of the Fast Data architecture. Its introduction allows splitting the import phase from strategies execution, which are currently handled both by the Real-Time Updater.
Here below a diagram showing how the Single View Trigger Generator service integrates with Fast Data flow is provided:
In this particular Fast Data configuration, Real-Time Updater is set to not execute strategies, but to rather emit a Projection Update event (
pr-update) for each modified projection.
The Single View Trigger Generator reacts to these events executing the corresponding strategy. The strategy produces one or more identifiers of the Single View that needs to be updated. This information is then passed over to the downstream component through Single View Trigger events (
sv-trigger) or Projection changes (
Eventually, the Single View Creator consumes these events triggering the corresponding logic for aggregating the interested Single View.
Advantages of adopting the Single View Trigger Generator
The main change occurring due to the introduction of the Single View Trigger Generator is the decoupling of projections import logic from the one in charge identifying which Single View should be updated. This effect leads to the following advantages:
- operations occurring during initial load phase are easier to handle and carry out
- processing load can be better spread across the two services depending on the requirements, so that it is possible to optimize the different number of services' replicas depending on the ongoing situation
Migration Guide for adopting Single View Trigger Generator
In order to introduce the Single View Trigger Generator in the Fast Data flow it is necessary to update the configuration of the Real-Time Updater and the Single View Creator. Below are reported the steps for each service.
- set environment variable
trueto enable the production of
- set environment variable
/home/node/app/kafkaProjectionUpdates. This allows the Real-Time Updater to load a configuration file automatically generated that instructs the service how to map each projection to the corresponding projection-update topic of the event streaming platform. This mapping can be changed in Console in each projection detail section, besides the configuration of the ingestion topic. In case this config map is not already available, please commit your work. It should appear afterward.
- disable the computation of projections changes by setting environment variable
Additional details on how to configure the Real-Time Updater to produce Projection Update events can be found in the service configuration page.
Single View Creator
The following steps are only required if you choose to generate
sv-trigger messages over Projection changes
- Configure the service to consume from Kafka (see the Consuming from Kafka section)
- Set environment variable
KAFKA_PROJECTION_CHANGES_TOPICSto the topic of the event streaming platform on which the Single View Trigger Generator service will publish the
sv-triggerevents. In case the topic does not already exist, we recommend adopting our naming convention for defining the topic name.
Single View Trigger vs Projection Changes
3.0.0 the Single View Trigger Generator can also produce Projection Changes (
pc) on MongoDB, just like the Real-Time Updater.
sv-trigger messages are sent and consumed on Event Streaming Platforms making the full cycle of Fast Data completely event driven. This offers a greater separation of concers since it means that business data like Projections and Single Views will be on your DB, while all the events that make Fast Data work will be on your Event Streaming Platform.
sv-trigger messages sound like the better option so far,
pc records usually offer a better performance of the aggregation process. This is because
pc records are unique by their Single View Identifier, so if a Projection record gets updated twice in a very short period of time the Single View Creator will most likely process the aggregation of the Single Views related to that Projection record only once. This is a broad generalization and it depends also on the size and frequency of the incoming flow of data you may have, if you choose to generate
pc records make sure you have the recommended indexes set up.
The Single View Trigger Generator can be configured to to process
pr-update events and to produce
sv-trigger events or Projection changes.
An in-depth explanation on how to configure the service is provided in the dedicated page.
Currently it is necessary to manually manage the Single View Trigger Generator config maps from the microservices section of Console, since it does not cover its configuration yet.
It is also important to notice that the Single View Trigger Generator couples up with a specific Single View and not with the System of Records. Consequently, for each configured Single View Creator it is necessary to introduce a Single View Trigger Generator.