Control Plane Configuration
Fast Data Control Plane allows to:
- keep track of the state of the different pipelines of a runtime;
- receive user interaction through the dedicated frontend, and to update pipeline state accordingly;
- receive feedbacks from workloads, to acknowledge that they are up-and-running in the pipelines they have been linked to.
State Persistence
The Control Plane stores for each runtime the latest state of all pipelines (either domain or single view) and its artifacts in a MongoDB collection, that is queried directly by the service at startup to retrieve the latest available state.
To discover more about state persistence configuration, you can check its dedicated section
This collection can change on the basis of the user interactions through the frontend.
Runtime Communication
Communications to/from runtime's workloads happen through two different channels, namely State Channel and Feedback Channel.
To discover more about channels configuration, you can check its dedicated section
State Channel
The State Channel is where the Control Plane sends the current state of each pipeline. Each pipeline has a general state, that can be overridden.
For each pipeline, there could be different overrides
that can change individually the state of a particular artifact.
Also, each artifact can have additional overrides
for each stage of the pipeline where is employed.
The state that can be set at pipeline level, or artifact level, or stage level is an Activation State: it can either pause
or resume
,
which will be used by workloads to stop/resume consumption of data.
- Schema Viewer
- Raw JSON Schema
{
"$schema": "http://json-schema.org/draft-07/schema",
"$id": "fast-data-state.schema.json",
"title": "FastDataState",
"examples": [
{
"key": {
"id": "854c2285-334e-41d1-bcfc-8c483fafca08"
},
"value": {
"pipelines": {
"state": "resume"
}
}
},
{
"key": {
"id": "d2d4fb77-94e9-45cb-ad2a-4ad09040b796"
},
"value": {
"pipelines": {
"state": "pause"
}
}
},
{
"key": {
"id": "bd1643e1-7f53-4b84-93ce-a979d2a5f9cb"
},
"value": {
"pipelines": {
"state": "pause",
"override": {
"b52c98ec-4910-456a-9ac4-13ad82dac404": {
"state": "resume",
"name": "sv_orders"
}
}
}
}
},
{
"key": {
"id": "854c2285-334e-41d1-bcfc-8c483fafca08"
},
"value": {
"pipelines": {
"state": "pause",
"override": {
"b52c98ec-4910-456a-9ac4-13ad82dac404": {
"state": "pause",
"name": "sv_orders",
"override": {
"7e738df3-d963-4554-a8aa-3ca3c55bcf22": {
"state": "resume",
"name": "base-prj"
}
}
}
}
}
}
},
{
"key": {
"id": "854c2285-334e-41d1-bcfc-8c483fafca08"
},
"value": {
"pipelines": {
"state": "pause",
"override": {
"b52c98ec-4910-456a-9ac4-13ad82dac404": {
"state": "pause",
"name": "sv_orders",
"override": {
"7e738df3-d963-4554-a8aa-3ca3c55bcf22": {
"state": "pause",
"name": "base-prj",
"override": {
"b9dc9fc8-2d06-4be5-9950-dd2088cfc034": {
"state": "resume",
"name": "aggregation"
}
}
}
}
}
}
}
}
},
{
"key": {
"id": "854c2285-334e-41d1-bcfc-8c483fafca08"
},
"value": {
"pipelines": {
"state": "pause",
"override": {
"b52c98ec-4910-456a-9ac4-13ad82dac404": {
"state": "pause",
"name": "orders-domain",
"override": {
"ab2703e8-b13b-42a3-af2c-c3c2f7d97792": {
"state": "pause",
"name": "proj-a",
"override": {
"daeb6959-ddf2-4c03-b237-ff57b2e1ea4d": {
"state": "resume",
"name": "mapping"
},
"a161dd38-7979-4156-8551-3c8c6386bfc4": {
"state": "resume",
"name": "backup"
}
}
}
}
}
}
}
}
}
],
"definitions": {
"activation": {
"type": "string",
"title": "Activation",
"description": "describe the values a pipeline state can take",
"enum": [
"resume",
"pause"
]
},
"stateful": {
"type": "object",
"title": "StatefulItem",
"required": [
"state"
],
"properties": {
"state": {
"$ref": "#/definitions/activation"
}
}
},
"named": {
"type": "object",
"title": "NamedItem",
"required": [
"name"
],
"properties": {
"name": {
"type": "string"
}
}
},
"artifactItem": {
"title": "ArtifactItem",
"allOf": [
{
"$ref": "#/definitions/stateful"
},
{
"$ref": "#/definitions/named"
},
{
"type": "object",
"properties": {
"override": {
"type": "object",
"description": "mapping between pipeline exec types and their corresponding state override",
"additionalProperties": {
"allOf": [
{
"$ref": "#/definitions/stateful"
},
{
"$ref": "#/definitions/named"
}
]
}
}
}
}
]
},
"eventPayload": {
"type": "object",
"title": "FastDataStateValue",
"required": [
"pipelines"
],
"properties": {
"pipelines": {
"allOf": [
{
"$ref": "#/definitions/stateful"
},
{
"type": "object",
"properties": {
"override": {
"type": "object",
"description": "mapping between pipelines UUID and their corresponding state override",
"additionalProperties": {
"allOf": [
{
"$ref": "#/definitions/stateful"
},
{
"$ref": "#/definitions/named"
},
{
"type": "object",
"properties": {
"override": {
"type": "object",
"description": "mapping between pipeline artifact reference and their corresponding state override",
"additionalProperties": {
"$ref": "#/definitions/artifactItem"
}
}
}
}
]
}
}
}
}
]
}
}
}
},
"type": "object",
"required": [
"key",
"value"
],
"additionalProperties": true,
"properties": {
"key": {
"type": "object",
"required": [
"id"
],
"properties": {
"id": {
"type": "string",
"description": "UUID representing a reference to the runtime"
}
}
},
"value": {
"$ref": "#/definitions/eventPayload"
}
}
}
Feedback Channel
The Feedback Channel is where the workloads will notify the Control Plane about the state of their artifacts. This is needed to inform the Control Plane that:
- the artifacts are up-and-running;
- the workloads have their state updated correctly after a message from the state channel
The workloads will send messages through the feedback channel periodically.
- Schema Viewer
- Raw JSON Schema
{
"$schema": "http://json-schema.org/draft-07/schema",
"$id": "service-feedback.schema.json",
"title": "ServiceFeedback",
"examples": [
{
"key": {
"pipeline": {
"id": "ec0ae4b1-e8aa-4a82-b5d8-d260e0eed340",
"name": "sv pipeline 1"
}
},
"value": {
"edges": [
{
"artifact": {
"id": "7a171212-2da0-4c60-9d7c-67f80eca91cb",
"name": "books_proj"
},
"exec": {
"id": "06e4f2e2-0e0f-4c41-85af-1c6a6075965f",
"name": "mapping"
},
"state": "running"
},
{
"artifact": {
"id": "8de96793-7cef-4867-9ce3-3a2fa23f9cdc",
"name": "orders_proj"
},
"exec": {
"id": "10c1165e-e5b2-4240-8990-c9b8b078c66e",
"name": "mapping"
},
"state": "running"
},
{
"artifact": {
"id": "8de96793-7cef-4867-9ce3-3a2fa23f9cdc",
"name": "orders_proj"
},
"exec": {
"id": "b8c1b719-56bb-421a-b37b-cce472ccfb8f",
"name": "backup"
},
"state": "paused"
}
]
},
"headers": {
"version": "1"
}
},
{
"key": {
"pipeline": {
"id": "ec0ae4b1-e8aa-4a82-b5d8-d260e0eed340",
"name": "sv pipeline 1"
}
},
"value": {
"edges": [
{
"artifact": {
"id": "f719e097-5f06-48e8-9eca-b43b634bf450",
"name": "books_proj"
},
"exec": {
"id": "83f18349-2d2b-4682-ac1c-440b88bf1643",
"name": "aggregation"
},
"state": "paused"
}
]
},
"headers": {
"version": "1"
}
},
{
"key": {
"pipeline": {
"id": "ec0ae4b1-e8aa-4a82-b5d8-d260e0eed340",
"name": "sv pipeline 1"
}
},
"value": {
"edges": [
{
"artifact": {
"id": "f719e097-5f06-48e8-9eca-b43b634bf450",
"name": "books_proj"
},
"exec": {
"id": "83f18349-2d2b-4682-ac1c-440b88bf1643",
"name": "aggregation"
},
"state": "paused",
"metadata": {
"throughput": "{\"value\":342,\"unit\":\"msg/s\"}"
}
}
]
},
"headers": {
"version": "1"
}
}
],
"definitions": {
"state": {
"type": "string",
"title": "RuntimeFeedback",
"description": "the actual operation state, extracted from the deployed service that is sending the feedback",
"enum": [
"running",
"paused"
]
},
"item": {
"type": "object",
"title": "ItemFeedback",
"required": [
"id",
"name"
],
"properties": {
"id": {
"type": "string",
"description": "unique identifier of the item"
},
"name": {
"type": "string",
"description": "human-readable name of the item - based on old Fast Data projection and single view collection identifiers"
}
}
}
},
"type": "object",
"required": [
"key",
"value"
],
"properties": {
"key": {
"type": "object",
"title": "HeartbeatKey",
"required": [
"pipeline"
],
"properties": {
"pipeline": {
"type": "object",
"required": [
"id",
"name"
],
"properties": {
"id": {
"type": "string"
},
"name": {
"type": "string"
}
}
}
}
},
"value": {
"type": "object",
"title": "HeartbeatPayload",
"required": [
"edges"
],
"properties": {
"edges": {
"type": "array",
"items": {
"type": "object",
"required": [
"artifact",
"exec",
"state"
],
"properties": {
"artifact": {
"$ref": "#/definitions/item"
},
"exec": {
"$ref": "#/definitions/item"
},
"state": {
"$ref": "#/definitions/state"
},
"metadata": {
"type": "object",
"additionalProperties": {
"type": "string"
}
}
}
}
}
}
},
"headers": {
"type": "object",
"title": "HeartbeatHeaders",
"properties": {
"version": {
"const": "1"
}
},
"additionalProperties": true
}
}
}
Endpoints
Here's a list of the APIs made available by the Control Plane with the following endpoints.
Endpoint | Type | Method | Description |
---|---|---|---|
/fast-data/ | REST | POST | Receives JSON-RPC from the frontend. |
/fast-data/configurations/:id | REST | GET | Returns the data of a given pipeline of the runtime where Control Plane has been deployed. |
/fast-data/configurations | REST | GET | Returns the data from all pipelines of the runtime where Control Plane has been deployed. |
/fast-data/feedback/:id | Websocket | GET | Opens a websocket connection with the client to receive updates for a given pipeline. |
The default prefix for Control Plane endpoints is /fast-data
and can be customized in the Settings configuration.
To make the endpoint work properly, additional manual operations must be performed in the Advanced Section of your gateway.
These operations are needed to:
- introduce the support to Websocket connection upgrade;
- override the default headers that are added to the HTTP response, so that
X-Frame-Options
is removed and the frontend can be embedded as an iFrame.
Envoy
This configuration edit needs to be inserted in the file patches.yml
.
It is important to observe that these modifications affect the first filter chain and subsequently the first filter definition.
- listener_name: frontend
# https://www.envoyproxy.io/docs/envoy/v1.29.3/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#envoy-v3-api-msg-extensions-filters-network-http-connection-manager-v3-httpconnectionmanager-upgradeconfig
'filter_chains.0.filters.0.typed_config.upgrade_configs':
upgrade_type: "websocket"
'filter_chains.0.filters.0.typed_config.route_config.response_headers_to_add': [
{
"header": {
"key": "X-Content-Type-Options",
"value": "nosniff"
},
"append": false
},
{
"header": {
"key": "X-Download-Options",
"value": "noopen"
},
"append": false
},
{
"header": {
"key": "Referrer-Policy",
"value": "strict-origin-when-cross-origin"
},
"append": false
}
]
Please notice that the above configurations are specific for the frontend
listener of Envoy. In case it has been
decided to expose the Fast Data Runtime Management system under a different listener,
please update the configuration accordingly.
Nginx
If you are using the standard Nginx API Gateway, you have to:
- add this custom location to
api-gateway/server-extension.conf
using the websocket endpoint as location (e.g./fast-data/feedback
) - if you want to use the Control Plane as iFrame, you need to expose the frontend on a different location (see full example below) and overwrite its headers in the Advanced section.
API Gateway Nginx Full Configuration
Consider a use case where:
- Control Plane backend is exposed over
/fast-data
- Control Plane frontend is exposed over
/control-plane
(check the frontend documentation on how to overwrite the default base path)
Then, we have to write the following advanced configuration:
location /fast-data/feedback/ {
proxy_pass http://$proxy_name$proxy_url;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# WebSocket support
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection upgrade;
}
location /control-plane {
proxy_pass http://$proxy_name$proxy_url;
add_header X-Content-Type-Options nosniff;
add_header X-Content-Type-Options noopen;
add_header Referrer-Policy strict-origin-when-cross-origin;
}
Configuration
Configuration of Control Plane is a straightforward process that involves setting up a ConfigMap and specifying essential environment variables.
Environment Variables
Control Plane can be customized using the following environment variables:
Name | Required | Description | Default Value |
---|---|---|---|
HTTP_PORT | - | This variable determines the TCP port where the HTTP controller binds its listener. | 3000 |
LOG_LEVEL | - | Specify the centralized application log level, choosing from options such as debug , error , fatal , info , silent , trace , or warn . | info |
FD_CONTROL_PLANE_CONFIGURATION_FILEPATH | - | Set the location of the configuration file. | ~/.fd/control-plane/configuration.json |
Config Map
The configuration of the service is handled by a JSON file whose location is defined by the FD_CONTROL_PLANE_CONFIGURATION_FILEPATH
.
The Control Plane applications provide a sample of this JSON in a dedicated Config Map, named control-plane-configuration
.
- Schema Viewer
- Raw JSON Schema
{
"title": "Control Plane Configuration JSON Schema",
"type": "object",
"properties": {
"persistence": {
"type": "object",
"properties": {
"collection": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"updates": {
"type": "object",
"properties": {
"millis": {
"type": "number"
}
},
"additionalProperties": false
}
},
"required": [
"name"
],
"additionalProperties": false
},
"configuration": {
"type": "object",
"properties": {
"database": {
"type": "string"
},
"url": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"title": "Secret From Variable",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"title": "Secret From File",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
}
},
"required": [
"url"
],
"additionalProperties": false
},
"type": {
"type": "string",
"const": "mongodb"
}
},
"required": [
"collection",
"configuration",
"type"
],
"additionalProperties": false
},
"runtime": {
"type": "object",
"properties": {
"feedback": {
"anyOf": [
{
"type": "object",
"properties": {
"configuration": {},
"type": {
"type": "string",
"const": "grpc"
}
},
"required": [
"type"
],
"additionalProperties": false,
"default": {
"type": "grpc"
}
},
{
"allOf": [
{
"type": "object",
"properties": {
"configuration": {
"title": "Kafka Configuration",
"type": "object",
"properties": {
"allow.auto.create.topics": {
"type": "string",
"enum": [
"true",
"false"
]
},
"auto.offset.reset": {
"type": "string",
"enum": [
"earliest",
"latest"
]
},
"bootstrap.servers": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
},
"client.id": {
"type": "string"
},
"enable.auto.commit": {
"type": "string",
"enum": [
"true",
"false"
]
},
"sasl.mechanism": {
"type": "string",
"enum": [
"PLAIN",
"SCRAM-SHA-256",
"SCRAM-SHA-512",
"OAUTHBEARER"
]
},
"sasl.password": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
},
"sasl.username": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
},
"security.protocol": {
"type": "string",
"enum": [
"PLAINTEXT",
"SSL",
"SASL_PLAINTEXT",
"SASL_SSL"
]
},
"session.timeout.ms": {
"type": "number"
},
"ssl.key.password": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
},
"ssl.keystore.certificate.chain": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
},
"ssl.keystore.key": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
},
"ssl.keystore.location": {
"type": "string"
},
"ssl.keystore.type": {
"type": "string",
"const": "PEM",
"default": "PEM"
},
"ssl.truststore.certificates": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
},
"ssl.truststore.location": {
"type": "string"
},
"ssl.truststore.type": {
"type": "string",
"const": "PEM",
"default": "PEM"
}
},
"required": [
"bootstrap.servers"
],
"additionalProperties": false
},
"type": {
"type": "string",
"const": "kafka"
}
},
"required": [
"configuration",
"type"
]
},
{
"type": "object",
"properties": {
"consumer": {
"type": "object",
"properties": {
"group.id": {
"type": "string"
},
"topic": {
"type": "string"
}
},
"required": [
"group.id",
"topic"
],
"additionalProperties": false
}
},
"required": [
"consumer"
]
}
]
}
]
},
"state": {
"anyOf": [
{
"type": "object",
"properties": {
"configuration": {},
"type": {
"type": "string",
"const": "grpc"
}
},
"required": [
"type"
],
"additionalProperties": false,
"default": {
"type": "grpc"
}
},
{
"allOf": [
{
"type": "object",
"properties": {
"configuration": {
"type": "object",
"properties": {
"allow.auto.create.topics": {
"type": "string",
"enum": [
"true",
"false"
]
},
"auto.offset.reset": {
"type": "string",
"enum": [
"earliest",
"latest"
]
},
"bootstrap.servers": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
},
"client.id": {
"type": "string"
},
"enable.auto.commit": {
"type": "string",
"enum": [
"true",
"false"
]
},
"sasl.mechanism": {
"type": "string",
"enum": [
"PLAIN",
"SCRAM-SHA-256",
"SCRAM-SHA-512",
"OAUTHBEARER"
]
},
"sasl.password": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
},
"sasl.username": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
},
"security.protocol": {
"type": "string",
"enum": [
"PLAINTEXT",
"SSL",
"SASL_PLAINTEXT",
"SASL_SSL"
]
},
"session.timeout.ms": {
"type": "number"
},
"ssl.key.password": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
},
"ssl.keystore.certificate.chain": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
},
"ssl.keystore.key": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
},
"ssl.keystore.location": {
"type": "string"
},
"ssl.keystore.type": {
"type": "string",
"const": "PEM",
"default": "PEM"
},
"ssl.truststore.certificates": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
},
"ssl.truststore.location": {
"type": "string"
},
"ssl.truststore.type": {
"type": "string",
"const": "PEM",
"default": "PEM"
}
},
"required": [
"bootstrap.servers"
],
"additionalProperties": false
},
"type": {
"type": "string",
"const": "kafka"
}
},
"required": [
"configuration",
"type"
]
},
{
"type": "object",
"properties": {
"producer": {
"type": "object",
"properties": {
"topic": {
"type": "string"
}
},
"required": [
"topic"
],
"additionalProperties": false
}
},
"required": [
"producer"
]
}
]
}
]
}
},
"additionalProperties": false
},
"settings": {
"type": "object",
"properties": {
"channels": {
"type": "object",
"properties": {
"buffer": {
"type": "object",
"properties": {
"millis": {
"type": "number"
}
},
"additionalProperties": false
},
"grpc": {
"type": "object",
"properties": {
"mode": {
"type": "string",
"enum": [
"json",
"protobuf"
],
"default": "protobuf"
},
"port": {
"type": "number",
"default": 50051
}
},
"additionalProperties": false
},
"runtimeHeartbeat": {
"type": "object",
"properties": {},
"additionalProperties": false
}
},
"additionalProperties": false
},
"server": {
"type": "object",
"properties": {
"apis": {
"type": "object",
"properties": {
"controllers": {
"type": "object",
"properties": {
"fast-data": {
"type": "string",
"default": "/fast-data"
},
"prefix": {
"type": "string"
},
"settings": {
"type": "object",
"properties": {
"feedbackBuffer": {
"type": "object",
"properties": {
"millis": {
"type": "number"
}
},
"additionalProperties": false
},
"policies": {
"type": "object",
"properties": {
"edit": {
"type": "string"
},
"view": {
"type": "string"
}
},
"additionalProperties": false
}
},
"additionalProperties": false
}
},
"additionalProperties": false
},
"probes": {
"type": "object",
"properties": {
"healthcheck": {
"type": "string",
"default": "/healthz"
},
"metrics": {
"type": "string",
"default": "/metrics"
},
"prefix": {
"type": "string",
"default": "/-/"
},
"readiness": {
"type": "string",
"default": "/ready"
}
},
"additionalProperties": false
}
},
"additionalProperties": false
},
"authorization": {
"type": "object",
"properties": {
"apiKey": {
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"type": "string",
"const": "env"
}
},
"required": [
"key",
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"type": "string",
"const": "file"
}
},
"required": [
"path",
"type"
],
"additionalProperties": false
}
]
}
},
"required": [
"apiKey"
],
"additionalProperties": false
},
"prefix": {
"type": "string",
"default": "/"
},
"swagger": {
"type": "object",
"properties": {
"prefix": {
"type": "string",
"default": "/documentation"
}
},
"additionalProperties": false
},
"website": {
"type": "object",
"properties": {
"root": {
"type": "string",
"default": "./website"
},
"security": {
"type": "object",
"properties": {
"x-frame-options": {
"type": "string",
"enum": [
"sameorigin",
"deny"
]
}
},
"additionalProperties": false
},
"url": {
"type": "string",
"default": "/website/"
}
},
"additionalProperties": false
}
},
"additionalProperties": false
}
},
"additionalProperties": false
}
},
"required": [
"persistence"
],
"additionalProperties": false
}
Initially, only the persistence layer is required to be available at startup, as there could be no Fast Data runtime available.
- Persistence
- Runtime
- Settings
Persistence
The object persistence
configures the layer of runtime state persistence where Control Plane stores the next desired state for its runtime.
It must point to a MongoDB
collection, available in a replicaSet
enabled database.
Available interfaces include:
MongoDB
client
Example
{
// ...other control plane configurations
"persistence": {
"type": "mongodb",
"configuration": {
"url": "mongodb://localhost:27017/fast-data?replicaSet=local"
},
"collection": {
"name": "runtime-state"
}
}
// ...other control plane configurations
}
Runtime
The object runtime
configures:
- the state channel that the Control Plane uses to dispatch the next desired state;
- the feedback channel where Control Plane receives workloads feedbacks about their inner states.
Available interfaces are:
Kafka
Both feedback and state channels can be managed by a Kafka broker. In particular:
- the state channel will have a Kafka Producer that sends desired states to a dedicated topic (referred to as Action Topic)
- the feedback channel will have a Kafka Consumer that reads workloads feedbacks from a dedicated topic (referred to as Heartbeat Topic)
Example
{
// ...other control plane configurations
"runtime": {
"feedback": {
"configuration": {
// to see all the available configuration options, go over the JSON schema
"allow.auto.create.topics": "true",
"bootstrap.servers": "localhost:9094"
},
"consumer": {
"topic": "heartbeat-topic",
"group.id": "control-plane-group-id"
},
"type": "kafka"
},
"state": {
"configuration": {
// to see all the available configuration options, go over the JSON schema
"allow.auto.create.topics": "true",
"bootstrap.servers": "localhost:9094"
},
"producer": {
"topic": "action-topic"
},
"type": "kafka"
}
}
// ...other control plane configurations
}
gRPC
Both feedback and state channels can be managed by a gRPC server instance managed by the Control Plane, which creates a bi-directional streaming remote procedure call that:
- receives feedbacks from workloads
- dispatch states to services
Example
{
// ...other control plane configurations
"runtime": {
"feedback": {
"type": "grpc"
},
"state": {
"type": "grpc"
}
}
// ...other control plane configurations
}
By default, the gRPC server is exposed over the default gRPC protocol port (50051
).
This value can be overridden in the settings
section
gRPC + Kafka
It is also possible to have:
- a state channel managed by a topic in a Kafka broker;
- a feedback channel managed by a gRPC server with a client-side streaming request.
Example
{
// ...other control plane configurations
"runtime": {
"feedback": {
"type": "grpc"
},
"state": {
"configuration": {
// to see all the available configuration options, go over the JSON schema
"allow.auto.create.topics": "true",
"bootstrap.servers": "localhost:9094"
},
"producer": {
"topic": "action-topic"
},
"type": "kafka"
}
}
// ...other control plane configurations
}
Settings
The overall setup of Control Plane is tuned in the settings
object, allowing you to configure several aspects of the services, such as:
- channels setup, including:
- buffer timeout used to delay state channel notifications
- gRPC server configuration
- runtime heartbeat in milliseconds, used to tell whether a workload is alive or not
- server APIs, including:
- Fast Data control endpoint
- Metrics
- Probes
- Frontend/Website static files location
Example
{
// ...other control plane configurations
"settings": {
"server": {
"prefix": "/control-plane/",
"website": {
"url": "/"
}
}
}
// ...other control plane configurations
}
To have a full overview of the settings
properties, you can check the JSON Schema specifications
provided in this section.
Secret Resolution
The following fields contains secrets that can be managed using the Fast Data Secrets Resolution mechanism:
persistence.configuration.url
runtime.feedback.configuration.['bootstrap.servers']
runtime.feedback.configuration.['sasl.username']
runtime.feedback.configuration.['sasl.password']
runtime.feedback.configuration.['ssl.truststore.certificates']
runtime.feedback.configuration.['ssl.keystore.key']
runtime.feedback.configuration.['ssl.key.password']
runtime.feedback.configuration.['ssl.keystore.certificate.chain']
runtime.state.configuration.['bootstrap.servers']
runtime.state.configuration.['sasl.username']
runtime.state.configuration.['sasl.password']
runtime.state.configuration.['ssl.truststore.certificates']
runtime.state.configuration.['ssl.keystore.key']
runtime.state.configuration.['ssl.key.password']
runtime.state.configuration.['ssl.keystore.certificate.chain']
settings.server.authorization.apiKey
Examples
Here are listed some configuration examples that can be useful as a starting point for your setup.
Control Plane with Kafka Channels
{
"runtime": {
"state": {
"type": "kafka",
"configuration": {
"bootstrap.servers": "localhost:9094",
"allow.auto.create.topics": "true"
},
"producer": {
"topic": "state"
}
},
"feedback": {
"type": "kafka",
"configuration": {
"bootstrap.servers": "localhost:9094",
"allow.auto.create.topics": "true"
},
"producer": {
"topic": "feedback",
"group.id": "feedback-group"
}
}
},
"persistence": {
"type": "mongodb",
"configuration": {
"url": "mongodb://localhost:27017/fast-data?replicaSet=local"
},
"collection": {
"name": "runtime-state"
}
},
"settings": {
"server": {
"prefix": "/control-plane/",
"apis": {
"controllers": {
"fast-data": "/my-awesome-controller"
}
},
"website": {
"url": "/__frontend/"
}
}
}
}
Control Plane with gRPC channels
{
"runtime": {
"state": {
"type": "grpc"
},
"feedback": {
"type": "grpc"
}
},
"persistence": {
"type": "mongodb",
"configuration": {
"url": "mongodb://localhost:27017/fast-data?replicaSet=local"
},
"collection": {
"name": "runtime-state"
}
},
"settings": {
"server": {
"prefix": "/control-plane/",
"apis": {
"controllers": {
"fast-data": "/my-awesome-controller"
}
},
"website": {
"url": "/__frontend/"
}
}
}
}