Real-Time Updater Configurations
The Real-Time Updater needs some environment variables and some configurations file to work.
Environment variables
- LOG_LEVEL (required): defines the level of the logger
- MONGODB_URL (required): defines the mongodb url to contact
- PROJECTIONS_DATABASE_NAME (required): defines the name of the projections database
- PROJECTIONS_CHANGES_COLLECTION_NAME (required): defines the name of the projections changes collection
- PROJECTIONS_CHANGES_ENABLED: defines whether you want to generate projections changes, default is true,
- LC39_HTTP_PORT (required): defines the lc39 http port
- STRATEGIES_MAX_EXEC_TIME_MS (required): defines the maximum time for which a strategy is executed
- KAFKA_BROKERS (required): defines the kafka brokers
- KAFKA_GROUP_ID (required): defines the kafka group id
- KAFKA_SASL_USERNAME (required): defines the kafka sasl username
- KAFKA_SASL_PASSWORD (required): defines the kafka sasl password
- KAFKA_MESSAGE_ADAPTER (required): defines the kafka message adapter
- LIVENESS_INTERVAL_MS (required) defines the liveness interval in milliseconds
- INVARIANT_TOPIC_MAP (required): defines an object that maps the topic to the projection
- KAFKA_USE_LATEST_DEQUEUE_STRATEGY: defines latest dequeue strategy or not
- KAFKA_ADAPTER_FOLDER: defines the path to the kafka adapter folder
- CAST_FUNCTIONS_FOLDER: defines the path to the cast-functions folder
- MAP_TABLE_FOLDER: defines the path to the map table folder
- STRATEGIES_FOLDER: defines the path to the strategies folder
- KAFKA_SASL_MECHANISM: defines the authentication mechanism. It can be one of:
plain
,scram-sha-256
orscram-sha-512
. The default value isplain
.
Custom Projection Changes Collection
You can choose to use a collection you have already created in the CRUD section.
In order to do that, your collection is supposed to have the following fields (apart from the default ones):
[
{"name":"type","type":"string","required":false,"nullable":false},
{"name":"changes","type":"Array_RawObject","required":false,"nullable":false},
{"name":"identifier","type":"RawObject","required":true,"nullable":false},
{"name":"doneAt","type":"Date","required":false,"nullable":false}
]
You also need to have the following additional indexes:
Add an index with name type_change_state
, type normal
, unique false
.
You need to add the following index fields:
- name
type
, orderASCENDENT
- name
changes.state
, orderASCENDENT
Add another index with name type_identifier
, type normal
, unique true
.
You need to add the following index fields:
- name
type
, orderASCENDENT
- name
identifier
, orderASCENDENT
After that, you need to set your collection as the one to be used by the Real-Time Updater. To do so, set the name of the collection you want to use as value of the PROJECTIONS_CHANGES_COLLECTION_NAME
environment variable of the service.
Configuration files
The Real-Time Updater accepts the following configurations:
KAFKA_ADAPTER configurations
The mount path used for these configurations is: /home/node/app/configurations/kafkaAdapterFolder
.
This folder contains the configurations for your kafka adapters.
CAST_FUNCTION configurations
The mount path used for these configurations is: /home/node/app/configurations/castFunctionsFolder
.
In this folder you have all the generated Cast Functions definitions.
STRATEGIES configuration
The default mount path used for these configuration is: /home/node/app/configurations/strategies
.
In this folder you have all the generated Strategies which you have defined in your gitlab project inside the fast-data-files/strategies
directory.
MAP_TABLE configurations
The mount path used for these configurations is: /home/node/app/configurations/mapTableFolder
.
Two mappings will be placed in this folder: one between cast functions and fields and another one between strategies and projections.
An example:
{
"projection-name": {
"destination": "projection-name",
"conversion": {
"UNIQUE_ID": {
"fieldName": "UNIQUE_ID",
"cast": "__fromFile__[defaultCastToString]"
},
"NAME": {
"fieldName": "NAME",
"cast": "__fromFile__[defaultCastToString]"
},
"ADDRESS": {
"fieldName": "ADDRESS",
"cast": "__fromFile__[defaultCastToString]"
}
},
"fieldsInKey": ["UNIQUE_ID"],
"changes": {
"sv_single_views": "__fromFile__[getIdUniqueFromSingleView]"
}
}
}