Skip to main content
Version: 13.x (Current)

Configuration

Salesforce Connector Service​

The environment variables used by the service (* = optional) are:

  • HTTP_PORT: port, defaulted to 8080
  • KAFKA_BROKERS: comma separated list of broker addresses
  • KAFKA_CLIENT_ID: client id of the producer that will write on projection topics
  • KAFKA_SASL_MECHANISM*: authentication mechanism, use scram-sha-256 to enable scram authentication, defaulted to plain
  • KAFKA_SASL_USERNAME*: scram username
  • KAFKA_SASL_PASSWORD*: scram password
  • SALESFORCE_HTTP_BASE_PATH: base path for salesforce access token generation (mustn't end with /)
  • SALESFORCE_CLIENT_ID: client id of your salesforce account
  • SALESFORCE_CLIENT_SECRET: secret of your salesforce account
  • SALESFORCE_USERNAME: username of your salesforce account
  • SALESFORCE_PASSWORD: concatenation of password and secret token of your salesforce account
  • CRUD_URL: url of your crud service
  • KAFKA_DLQ_TOPIC: Kafka topic used to implement the DLQ
  • CONFIG_FILE_PATH: path to the yaml configuration file
  • PARALLELISM*: size of the coroutine pool which the events are dispatched to. It should be a prime number to reduce collisions and distribute the workload evenly. The default value is 97
  • CKP_UPSERT_PERIOD_MS*: period in milliseconds specifying the frequency of checkpoint save, default 5000
  • DEFAULT_REPLAY_POLICY*: replay id policy used in case a checkpoint is invalid for topic connection. Use REPLAY_FROM_TIP if you want the topic to start processing from the current instant, use REPLAY_FROM_EARLIER if you want to process all events received during the last 24 hours (might put the service under heavy load, but also
    might help to not lose changes)
  • RESTART_TIMES*: comma separated list of HH:mm formatted times at which the service will restart
  • RESTART_TIME_ZONE*: zone id of the time zone of reference of restart times, defaulted to Etc/UTC
  • INACTIVITY_TIMEOUT_SECONDS*: duration of the timeout that resets a topic connection after a period of inactivity, namely a period where no updates are received
The parallelism variable can be tuned to improve parallelization capabilities if you have topics with high traffic.

The default value should be fine for more cases, however the higher it gets, the more cpu resources will be required. Please feel free to tune this parameter if needed, using only prime numbers.:::

The yaml configuration file must have the following structure:

- salesforceTableName: Table1
  salesforceTopic: /data/Table1ChangeEvent
  projectionTopic: fd-p-sforce-table1-json
  idPropertyName: CustomId
- salesforceTableName: Table2
  salesforceTopic: /topic/Table2
  projectionTopic: fd-p-sforce-table2-json
  idPropertyName: Id
- ...
  • salesforceTableName: name of the table created on Salesforce DB
  • salesforceTopic: name of the StreamingAPI topic
  • projectionTopic: name of the Kafka topic of the projection
  • idPropertyName: optional name of the property representing the id of a record, defaulted to "Id"

Every item of the list corresponds to a new connection to a StreamingAPI topic. Be sure these data are correct as otherwise your service might get stuck in a restart loop, as health checks will fail.

Salesforce Connector DLQ Service​

The environment variables used by the service (* = optional) are:

  • HTTP_PORT: port, defaulted to 8080
  • KAFKA_BROKERS: comma separated list of broker addresses
  • KAFKA_CLIENT_ID: client id of the producer that will write on projection topics
  • KAFKA_GROUP_ID: group id of the DLQ consumer
  • KAFKA_SASL_MECHANISM*: authentication mechanism, use scram-sha-256 to enable scram authentication, defaulted to plain
  • KAFKA_SASL_USERNAME*: scram username
  • KAFKA_SASL_PASSWORD*: scram password
  • SALESFORCE_HTTP_BASE_PATH: base path for Salesforce access token generation (mustn't end with /)
  • SALESFORCE_CLIENT_ID: client id of your Salesforce account
  • SALESFORCE_CLIENT_SECRET: secret of your Salesforce account
  • SALESFORCE_USERNAME: username of your Salesforce account
  • SALESFORCE_PASSWORD: concatenation of password and secret token of your Salesforce account
  • KAFKA_DLQ_TOPIC: DLQ Kafka topic
  • FIELD_MAPPING_FILE_PATH*: path to the yaml configuration file
  • SALESFORCE_ACCESS_TOKEN_RETRY_ATTEMPTS*: number of retry attempts that will be done before declaring the service as unhealthy, defaulted to 5

The yaml configuration file must have the following structure:

Table1:
  Id: CustomId
  Field: CustomField
Table2:
  Field: CustomField
...

In the above example, we are going to rename all "Id" fields of Table1 to "CustomId" before sending them to the projection.