Skip to main content
Version: 13.x (Current)

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[7.9.0] 2024-07-29​

Added​

  • Add DELETE_MECHANISM env var to control hard/soft delete behavior

Fixed​

  • USE_UPSERT is overwritten to true when message adapter is db2 or basic since the DB2 message does not contain operation type info

[7.8.0] 2024-06-06​

Added​

    • support for new cast function signature, that is:
      export default function(messageObject, fieldName, logger) {
        const myValue = messageObject[fieldName]
        // do some custom logic...
        return myValue
      }
    • kafka message adapter can be loaded as ESM modules, for example:
      export default function myEsmMessageAdapter(message, primaryKeys, logger) {
        // ...adapter logic
      }

[7.7.0] 2024-04-23​

Updated​

  • control plane connection, which can be configured also using gRPC protocol supported by control plane backend.

    Two possible configurations can be set over the JSON file located at the path specified by CONTROL_PLANE_CONFIG_PATH:

    • full-duplex: the service will open a bidirectional streaming channel to send its state and receive updates
      {
        "feedback": {
          "type": "grpc"
        },
        "state": {
          "type": "grpc"
        },
        "settings": {
          "channel": {
            // interval for which service will send its state
            // defaults to 2500ms if not specified
            "heartbet.ms": 2500 
          },
          "grpc": {
            // tells the service the hostname of your control plane istance
            "host": "your-control-plane-host",
            // port where the service is exposed, defaults to grpc default 50051
            "port": 50051 
          }
        }
      }
    • feedback-only: the service will open a request streaming channel to send its state, while the control plane will send updates using a kafka topic
      {
        "feedback": {
          "type": "grpc"
        },
        "state": {
          "type": "kafka",
          "configuration": { /** kafka connection configuration */ }
        },
        "settings": { /** same as full-duplex */ }
      }

Fixed​

  • bug that prevented health probes to work properly if control plane was not configured

[7.6.0] 2024-04-11​

Added​

  • control plane feature, where:
    • ingestion topics can paused/resumed over commands sent by the control plane service.
    • the service updates periodically the control-plane state about the state of its topics.

To enable these features, the following environment variables are needed:

  • CONTROL_PLANE_CONFIG_PATH: absolute path of the JSON file containing the connection details to the control plane.
  • CONTROL_PLANE_BINDINGS_PATH: absolute path of the JSON file containing the link between control plane artifacts and ingestion topics. This file is auto-generated by Mia Console.

For more details on how to configure the control plane, visit its dedicated section

Updated​

  • kafka-importer-lib to v7.6.0: control plane features and fix to have a limit of 500 messages in a pr-update batch.

[7.5.8] 2024-02-13​

Added​

  • add PRODUCER_COMPRESSION environment variable, to enable the encoding of kafka pr-update messages with the following encoding standards:
    • snappy
    • gzip
    • none (default if no value has been provided)

Updated​

  • kafka-importer-lib to v7.5.8: pr-update messages are now sent in batch, instead of one message per update

[7.5.7] 2023-12-15​

Fixed​

  • improved some logging (upgraded fast-data-automation-lib to v3.1.4)
  • when using db2 (basic) or debezium message adapter the service is now able to properly process messages whose payload is empty (a buffer with length of zero), instead of skipping them

Added​

  • documentation folder

Changed​

  • updated Node JS to v20
  • updated service demo
  • updated kafka-importer-lib to v7.5.7

[7.5.6] 2023-09-07​

Fixed​

  • Strategies not executed when message has old timestamp (internalcounter)

Updated​

  • updated kafka-importer-lib to v7.5.5

[7.5.5] 2023-07-21​

Added​

  • added a docker-compose file and an example of configuration, which allows to automatically configure a demo of the Real-Time Updater
  • added a Makefile to simplify the interaction with docker-compose and starting tests using dependencies defined in the docker-compose

Changed​

  • updated kafka-importer-lib to include the kafka consumer metrics tracking logic
  • updated NodeJS image to v18.17.0 in Dockerfile
  • updated the README to provide some insights on the RTU demo
  • db2 can be used now as alias of basic when selecting the message adapter

[7.5.4] 2023-06-26​

Fixed​

  • upgrade kafka-importer-lib to v7.5.3 to fix the logic handling Debezium tombstone messages

[7.5.3] 2023-06-23​

Fixed​

  • upgrade kafka-importer-lib to v7.5.2 to fix the Debezium Message Adapter implementation. Now the Real-Time Updater properly support Debezium CDC change events with and without the change schema.

Changed​

  • upgraded nodejs image, both in Dockerfile and Gitlab-CI to v18.16.1
  • upgraded service dependencies

[7.5.2] 2023-05-25​

Fixed​

  • kafkaProjectionUpdates config map accepts topics without updatesTopic property, in the format that the console autogenerates.

[7.5.1] 2023-05-24​

Fixed​

  • When strategy throws an error the ingestion message is not committed anymore

[7.5.0] 2023-03-28​

Added​

  • Added support for Debezium kafka message adapter

[7.4.1] 2023-03-14​

Changed​

  • Re-designed the way the projection changes are stored in mongo: now they get upserted with a bulkWrite operation instead of doing four separate queries for each projection change. The new way is not based on the existence of a unique index to work properly, like it was with the previous implementation. Introduced with kafka-importer-lib@7.4.1

Fixed​

  • Projection changes creation does not depend on collection's unique index anymore, fixing the duplicate projections changes bug

Added​

  • Upgraded fast-data-automation-lib@2.3.5 and added neededFieldsMap to the strategy execution to make it compatible with new version and improve performance

[7.4.0] 2023-02-28​

Added​

  • golden-gate message adapter provides additional operations metadata onto which further logics can be enabled to ensure the correct functioning of the data flow. In particular, the pos field (generated by CDC system) is employed by default as value of operationPosition property to verify that only newer changes are written to the database. Starting from this release, the operationPosition property in golden-gate message adapter can also be set to adopt the Kafka message's timestamp (as it happens with the other message adapters). This change can be implemented by disabling the default behavior setting the USE_POS_AS_COUNTER environment variable to false

Fixed​

  • fixed a behavior regression that caused the service using this library to restart completely whenever a Kafka consumer rebalance event occurred. Consequently, due to the restarts, more rebalance events were triggered, causing even further restarts that could only be solved by stopping all the services using the same consumer group and starting only one of them.

[7.3.0] 2023-02-17​

Added​

  • Added K8s health routes using status exposed by Kafka Importer Lib
  • Added custom-plugin-lib replacing k8-patient-lib with kafka-importer-lib@7.3.0

[7.2.0] 2023-01-17​

Added​

  • Add possibility to use OAuthbearer authentication for connecting to Kafka (only grant type password is supported)

Changed​

  • upgrade kafka-importer-lib to v7.2.0 to
    • support OAuthBearer authentication
    • fix a situation where DELETE operations of __records__ with a conflicting __internal_counter resulted in writing a new record on the database whose key was null. This issue occurred when Real-Time Updater is configured in upsert mode.

[7.1.5] 2022-12-30​

Changed​

  • upgrade kafka-importer-lib to v7.1.5 to solve multiple bugs on peculiar service configurations

[7.1.4] 2022-11-18​

Added​

  • On error processing the message you can now pause the topic instead of tearing down the service. Added the PAUSE_TOPIC_CONSUMPTION_ON_ERROR env var

[7.1.3] 2022-11-09​

  • upgrade kafka-importer-lib to v7.1.3 to solve an error during the update of primary keys of type date

[7.1.2] 2022-11-07​

Changed​

  • upgrade kafka-importer-lib to v7.1.2

[7.1.1] 2022-09-27​

Changed​

  • upgrade ajv to v8.11.0
  • upgrade kafkajs to v2.2.0

[7.1.0] 2022-09-27​

Changed​

  • upgrade kafka-importer-lib to v7.1.0
  • upgrade fast-data-automation-lib to v2.1.0
  • review log verbosity

[7.0.0] 2022-07-29​

BREAKING CHANGES​

  • Added new required Env var SYSTEM_OF_RECORDS

Updated​

  • kafka-importer-lib@7.0.0 npm audit

[6.1.0] 2024-05-28​

Added​

    • support for new cast function signature, that is:
      export default function(messageObject, fieldName, logger) {
        const myValue = messageObject[fieldName]
        // do some custom logic...
        return myValue
      }
    • kafka message adapter can be loaded as ESM modules, for example:
      export default function myEsmMessageAdapter(message, primaryKeys, logger) {
        // ...adapter logic
      }

[6.0.1] 2022-07-04​

Updated​

  • kafka-importer-lib@6.2.2 npm audit

[6.0.0] 2022-07-04​

BREAKING CHANGES​

  • uniform handling of MongoDB queries in LowCode

Updated​

  • @mia-platform-internal/fast-data-automation-lib 2.0.0

[5.5.0] 2024-05-28​

Added​

    • support for new cast function signature, that is:
      export default function(messageObject, fieldName, logger) {
        const myValue = messageObject[fieldName]
        // do some custom logic...
        return myValue
      }
    • kafka message adapter can be loaded as ESM modules, for example:
      export default function myEsmMessageAdapter(message, primaryKeys, logger) {
        // ...adapter logic
      }

[5.4.1] 2022-05-09​

Fixed​

  • Strategies - correctly handle one-to-many relationships

Updated​

  • @mia-platform-internal/fast-data-automation-lib 1.4.3
  • @mia-platform-internal/kafka-importer-lib ^6.2.1

[5.4.0] 2022-04-26​

Added​

  • support to CA certs

Updated​

  • @mia-platform-internal/kafka-importer-lib ^6.2.0

[5.3.3] 2022-04-20​

Updated​

Fixed​

  • Fixed a bug where in case of an insert after delete projection, no projection update message would have been generated

[5.3.2] 2022-04-07​

Updated​

  • @mia-platform-internal/kafka-importer-lib ^6.1.3

Fixed​

  • wrong key used for kafka message projection updates

[5.3.1] 2022-03-28​

Updated​

  • @mia-platform-internal/fast-data-automation-lib ^1.3.0

[5.3.0] 2022-03-28​

Updated​

  • @mia-platform-internal/kafka-importer-lib ^6.1.0

Added​

  • send messages to a dedicated kafka topic after changes on projections on MongoDB, add related envs and update config map to add new topic

[5.2.1] 2022-03-25​

Fixed​

  • For low code the folder used to gather strategies is PROJECTION_CHANGES_SCHEMA_FOLDER instead of STRATEGIES_FOLDER

[5.2.0] 2022-03-22​

Added​

  • envs KAFKA_CONNECTION_TIMEOUT_MS, KAFKA_SESSION_TIMEOUT_MS, KAFKA_HEARTBEAT_INTERVAL_MS to configure kafka timeouts and interval

[5.1.0] 2022-03-17 (error: this is not really a minor, it was supposed to be just a patch)​

Updated​

  • @mia-platform-internal/fast-data-automation-lib": "^1.2.0
  • validate aggregation schema against a JSON Schema

[5.0.2] 2022-03-17​

Changed​

  • updated MongoDb npm driver to version 4.4.1

Fixed​

  • set FORCE_CHECK_ON_OFFSET default value to true

[5.0.1] 2022-03-07​

Updated​

  • @mia-platform-internal/kafka-importer-lib ^6.0.2

Fixed​

  • flat identifier in MongoDB filter in order to NOT fail when identifier has multiple fields in any order

[5.0.0] 2022-02-25​

BREAKING CHANGES​

  • update mongodb package from ^3.6.0 to ^4.3.1. Please, refer to official mongodb changelog to get more information.
  • @mia-platform-internal/kafka-importer-lib ^6.0.1 which renames kafkaInfo field into internalkafkaInfo in Kafka Projection Changes

Updated​

[4.4.0] 2022-02-14​

Added​

  • support to string, boolean and integer to ER Schema
  • LOGGING_UNMAPPED_FIELDS to decide whether to enable or disable the logging of the unmapped fields
  • COLLECTION_NAME_UNMAPPED_FIELDS_STORAGE to decide where to save information about the unmapped fields

[4.3.1] 2022-01-26​

Fixed​

  • updated kafka-importer-lib to fix logs issues

[4.3.0] 2022-01-25​

Added​

  • add PROJECTION_CHANGES_SCHEMA_FOLDER environment variable to specify the location of projectionChangesSchema file.
  • ER_SCHEMA_FOLDER and USE_AUTOMATIC_STRATEGIES to support automation on strategies execution by JSON configuration

[4.2.0] 2022-01-25​

Added​

  • logger for custom kafka message adapter

Updated​

[4.1.1] 2022-01-13​

Fixed​

  • key saved in projection and pc kafkaInfo is the original key and not the parsed one

Updated​

[4.1.0] 2022-01-11​

Added​

  • Snappy Compression Codec support for Kafka messages

Updated​

[4.0.0] 2021-12-27​

BREAKING CHANGES​

  • when heartbeat fails, generate PC and try to commit anyway if projection has already been updated.
  • do NOT check rebalancing for each bulk operation

Updated​

Added​

  • FORCE_CHECK_ON_OFFSET environment variable to specify whether to check message offset is greater than the one saved on the projection or not.
  • COMMIT_MESSAGE_LOGGING_INTERVAL to specify the interval of logging for committed messages

[3.5.1] 2021-11-29​

Updated​

  • @mia-platform-internal/kafka-importer-lib ^4.5.2 to fix mismatch of names of kafka info fields in projections and pc

[3.5.0] 2021-11-24​

Added​

  • KAFKA_CONSUMER_MAX_WAIT_TIME environment variable to define the maximum wait time for the Kafka Consumer for new data in batch

[3.4.0] 2021-11-24​

Added​

  • GENERATE_KAFKA_PROJECTION_CHANGES environment variables to decide whether to send projection changes on Kafka
  • KAFKA_PROJECTION_CHANGES_FOLDER environment variables to define the mount where has been mounted the configuration needed to specify the topic to send the projection changes

Updated​

Fixed​

  • logs format on errors

[3.3.1] 2021-11-18​

  • do not upsert on update if the document target is missing if useUpsert is false

[3.3.0] 2021-11-15​

  • @mia-platform-internal/kafka-importer-lib ^4.3.0 - to enable upsert on insert to be set with an environment variable, and importing goldenGateAdapter (renamed as operationTypeBasedAdapter) from the library.

[3.2.0] 2021-11-05​

Updated​

  • @mia-platform-internal/kafka-importer-lib ^4.2.0 - to have additional info in projection changes and projections

[3.1.3] 2021-09-28​

Updated​

  • @mia-platform-internal/kafka-importer-lib ^4.1.0 - to have createdAt in new projection document

[3.1.2] 2021-07-21​

Updated​

  • @mia-platform-internal/kafka-importer-lib ^4.0.2

[3.1.1] 2021-07-15​

Fixed​

  • fixed cast of envs with jsonSchema

[3.1.0] 2021-06-24​

Updated​

Added​

  • added support to Object and Array Of Object fields

[3.0.1] 2021-04-28​

Updated​

[3.0.0] 2021-04-22​

Updated​

  • @mia-platform-internal/kafka-importer-lib ^3.1.1

BREAKING​

  • Kafka Message Adapter CANNOT return keyObject null

[2.0.1] 2021-04-12​

Fixed​

  • Sorts fieldsInKey of mapTable
  • does not set kafka authentication if both KAFKA_SASL_USERNAME and KAFKA_SASL_PASSWORD are empty

[2.0.0] 2021-03-01​

Added​

  • GoldenGate adapter

BREAKING CHANGE​

  • KAFKA_MESSAGE_ADAPTER env to choose the adapter to use
  • use virtualDelete

Updated​

  • @mia-platform-internal/kafka-importer-lib v2.5.0

[1.1.0] 2021-02-22​

Added​

  • configuration of kafka_sasl_mechanism with environment variable

[1.0.0] 2021-02-11​

[0.1.0] 2021-01-29​

Updated​

  • @mia-platform-internal/kafka-importer-lib ^2.3.1 to support CRUD fields and update logs format

Added​

  • configuration files