Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[3.4.1] 2025-04-09

Fixed

• Topics not handled by the control plane are resumed and work independently

[3.4.0] 2025-01-23

Added

• maxBatchSize property in producer configuration to configure the number of messages that can be written in a single request
• forceResume setting field to controlPlane.settings.channel that can either be a boolean or an object with a numeric field timeout.ms, defining the amount of time for control plane pipelines to be resumed after an unsuccessful connection.

Changed

• controlPlane.settings object now has both state and feedback marked as optional

Removed

• gRPC Control Plane Feedback client
• Kafka Control Plane client

[3.3.4] 2025-01-16

Fixed

• Reconnect Control Plane GRPC client when Control Plane Operator restarts

[3.3.3] 2024-12-10

Added

• Added image signing and sbom generation in pipeline

[3.3.2] 2024-07-29

Fixed

• Non recoverable errors now make the service crash

[3.3.1] 2024-04-29

Added

• add support for secret variables in the event-store config map

[3.3.0] 2024-04-23

Updated

• control plane connection, which can be configured also using gRPC protocol supported by control plane backend.

  Two possible configurations can be set over the JSON file located at the path specified by CONTROL_PLANE_CONFIG_PATH:

  • full-duplex: the service will open a bidirectional streaming channel to send its state and receive updates

    {
      "feedback": {
        "type": "grpc"
      },
      "state": {
        "type": "grpc"
      },
      "settings": {
        "channel": {
          // interval for which service will send its state
          // defaults to 2500ms if not specified
          "heartbet.ms": 2500 
        },
        "grpc": {
          // tells the service the hostname of your control plane istance
          "host": "your-control-plane-host",
          // port where the service is exposed, defaults to grpc default 50051
          "port": 50051 
        }
      }
    }
  • feedback-only: the service will open a request streaming channel to send its state, while the control plane will send updates using a kafka topic

    {
      "feedback": {
        "type": "grpc"
      },
      "state": {
        "type": "kafka",
        "configuration": { /** kafka connection configuration */ }
      },
      "settings": { /** same as full-duplex */ }
    }

[3.2.0] 2024-04-11

Added

• control plane feature, where:
  • pr-update topics can paused/resumed over commands sent by the control plane service.
  • the service updates periodically the control-plane about the state of its topics.

To enable these features, the following environment variables are needed:

• CONTROL_PLANE_CONFIG_PATH: absolute path of the JSON file containing the connection details to the control plane.
• CONTROL_PLANE_BINDINGS_PATH: absolute path of the JSON file containing the link between control plane artifacts and pr-update topics. This file is auto-generated by Mia Console

For more details on how to configure the control plane, visit its dedicated section

[3.1.6] 2024-02-13

Added

• introduce support for snappy compression both on the consumer and producer sides. While compression coded employed on the consumer is automatically recognized, on the producer it is necessary to be configured explicitly. This can be achieved by adding compressionName property to event store producer configuration, which can get one of the following values:
  • none (default when no value provided)
  • snappy (recommended)
  • gzip

[3.1.5] 2024-02-09

Updated

• fast-data-automation-lib to v3.1.5

[3.1.4] 2023-12-15

Fixed

• Service won't shutdown without logs when the kafka consumer cannot parse a message
• improve description of FD_SVTG_E0003 error code

[3.1.3] 2023-11-03

Added

• Added Single View name to the sv-trigger message

Changed

• updated Docker image to employ NodeJS v20

Fixed

• __internal_kafka_info property can now correctly accepts number type as key value when pr-updates v1.0.0 are read.

[3.1.2] 2023-10-10

Fixed

• review support of custom CA certificate for Kafka Producer client

[3.1.1] 2023-10-02

Added

• message header to sv-trigger events to specify which schema it is employed

Changed

• upgraded fast-data-automation-lib to v3.1.3 to fix an issue on pr-update events with after property in message payload set to null
• updated development demo to test service functioning with multiple projections
• updated definition of __internal__kafkaInfo in pr-update event to avoid conflicts with types

[3.1.0] 2023-09-25

Added

• add support for v2.0.0 of pr-update

Changed

• update NodeJS version to v18.18.0

[3.0.4] 2023-09-22

Added

• Added check on foreign keys to know if running strategy on both before and after is necessary

Changed

• upgraded fast-data-automation-lib to v3.1.2
• remove unused environment variables READ_TOPIC_FROM_BEGINNING and CA_CERT_PATH. Their actual configuration can be found within the event store config file
• update NodeJS version to v18.17.1
• updated library dependencies

Fixed

• additional CA files can be added to the ssl config of the event store
• removed printing invalid event store configuration to avoid sharing sensitive information

[3.0.3] 2023-07-25

Added

• added new metrics to monitor Kafka consumer that reads from pr-update topics
• added a docker-compose file and an example of configuration, which allows to automatically configure a demo of the Single View Trigger Generator
• added a Makefile to simplify the interaction with docker-compose and starting tests using dependencies defined in the docker-compose

Changed

• updated fast-data-automation-lib to v3.1.1
• updated service dependencies

[3.0.2] 2023-06-30

Fixed

• Kafka log level is mapped to the kafkajs string, not number

[3.0.1] 2023-06-27

Fixed

• Kafka brokers is a comma separated string, not an array of strings

[3.0.0] 2023-06-27

Added

• Added support for the generation of MongoDB projection changes instead of sv-trigger kafka messages

Breaking Changes

• kafkaProjectionChanges configMap does not support __lookup__ strategy type anymore since pr-updates events for PATCH operations are supported also by the Single View Creator
• consequently, the service won't emit sv-trigger messages of type patch, in fact the property type has been removed from sv-trigger messages altogether
• kafka configuration is not passed through environment variables but through its own config map Input Output Config
• TRIGGER_TOPIC environment variable has been generalized and renamed to EVENT_STORE_TARGET to have meaning also with mongo collections

[2.0.1] 2023-04-14

Changed

• review error codes to be aligned with the troubleshooting guidelines

[2.0.0] 2023-04-13

Breaking Changes

• environmental variables
  • renamed STRATEGIES_FOLDER into MANUAL_STRATEGIES_FOLDER
  • renamed CUSTOM_FUNCTIONS_FOLDER into TRIGGER_CUSTOM_FUNCTIONS_FOLDER
  • removed USE_AUTOMATIC
  • SINGLE_VIEW_NAME, ER_SCHEMA_FOLDER, PROJECTION_CHANGES_SCHEMA_FOLDER are now always required
• manual and custom strategy functions are expected to be AsyncGenerator of IdentifierObject instead of returning a Promise containing a list of IdentifierObject

Added

• added support to MongoDB v6.0

Changed

• upgraded service dependencies, among which the most relevant are
  • @mia-platform-internal/fast-data-automation-lib@v3
  • mongodb@v5
• removed dependencies not needed anymore
• refactored the service to use Mia eslint standard
• simplified the process of configurations loading removing unnecessary repetition or use cases management
• expanded tests
• refactored how the core logic and service dependencies are composed, removing the need of fastify-pluging
• swapped ts-node with swc for transpling tests source code
• upgraded Docker image to node v18.15.0
• reviewed pipeline removing unneeded packages installation
• reduced the size of the trigger message buffer from 500 to 100 to prevent issues with JS garbage collector

Removed

• removed support to MongoDB v4.0

[1.0.1] 2023-02-24

Changed

• Added env var READ_TOPIC_FROM_BEGINNING to read pr-update messages topic from the beginning
• Readiness route working, improve on handling of kafka consumer heartbeat, faster service start up by removing await on consumer.run

[1.0.0] 2023-01-20

Breaking Change

• Changed kafka projections to kafka projection updates to match the RTU env var

[0.3.2] 2022-12-20

Fixed

• Strategy resolution matched all documents when both fields in a erSchema condition were inexistent on the starting document. Now the query does not match any documents in that case. This fix comes with @mia-platform-internal/fast-data-automation-lib: 2.3.4

[0.3.1] 2022-11-29

• Projection update messages are correctly resolved after the sending the sv-trigger messages

[0.3.0] 2022-11-24

Added

• Added the neededFieldsMap system in the strategy execution

[0.2.0] 2022-11-21

Changed

• Messages sent to TRIGGER_TOPIC are grouped and sent all at once
• Included algorithm of grouping messages based on the message key

[0.1.1] 2022-11-11

Changed

• Removed lodash and ramda functions
• Updated bucket values for the "strategy execution time" prometheus metric
• Updated @mia-platform-internal/fast-data-automation-lib to v2.2.1

[0.1.0] 2022-11-03

Added

• Added the strategy execution time metric
• Added the health routes (health, ready, check-up)

Changed

• updated getStrategyFunctionFromAutomatic inside getStrategies to support the new automator function with UpdateEvent as parameter

Fixed

• Moved resolveOffset method from the finally block to the try block and added error rethrow inside catch of the consumer.run method

[0.0.2-rc.1] 2022-09-30

[0.0.2-rc.0] 2022-09-30

Changed

• Refactor of the main test and other files
• Renaming triggerer with trigger generator

[0.0.1] 2022-08-31