Skip to main content
Version: 13.x (Current)

Control Plane Configuration

Fast Data Control Plane allows to:

  • keep track of the state of the different pipelines of a runtime;
  • receive user interaction through the dedicated frontend, and to update pipeline state accordingly;
  • receive feedbacks from workloads, to acknowledge that they are up-and-running in the pipelines they have been linked to.

State Persistence​

The Control Plane stores for each runtime the latest state of all pipelines (either domain or single view) and its artifacts in a MongoDB collection, that is queried directly by the service at startup to retrieve the latest available state.

info

To discover more about state persistence configuration, you can check its dedicated section

This collection can change on the basis of the user interactions through the frontend.

Runtime Communication​

Communications to/from runtime's workloads happen through two different channels, namely State Channel and Feedback Channel.

info

To discover more about channels configuration, you can check its dedicated section

State Channel​

The State Channel is where the Control Plane sends the current state of each pipeline. Each pipeline has a general state, that can be overridden.

For each pipeline, there could be different overrides that can change individually the state of a particular artifact.

Also, each artifact can have additional overrides for each stage of the pipeline where is employed.

The state that can be set at pipeline level, or artifact level, or stage level is an Activation State: it can either pause or resume, which will be used by workloads to stop/resume consumption of data.

Loading ....

Feedback Channel​

The Feedback Channel is where the workloads will notify the Control Plane about the state of their artifacts. This is needed to inform the Control Plane that:

  • the artifacts are up-and-running;
  • the workloads have their state updated correctly after a message from the state channel

The workloads will send messages through the feedback channel periodically.

Loading ....

Endpoints​

Here's a list of the APIs made available by the Control Plane with the following endpoints.

EndpointTypeMethodDescription
/fast-data/RESTPOSTReceives JSON-RPC from the frontend.
/fast-data/configurations/:idRESTGETReturns the data of a given pipeline of the runtime where Control Plane has been deployed.
/fast-data/configurationsRESTGETReturns the data from all pipelines of the runtime where Control Plane has been deployed.
/fast-data/feedback/:idWebsocketGETOpens a websocket connection with the client to receive updates for a given pipeline.
tip

The default prefix for Control Plane endpoints is /fast-data and can be customized in the Settings configuration.

caution

To make the endpoint work properly, additional manual operations must be performed in the Advanced Section of your gateway.

These operations are needed to:

  • introduce the support to Websocket connection upgrade;
  • override the default headers that are added to the HTTP response, so that X-Frame-Options is removed and the frontend can be embedded as an iFrame.

Envoy​

This configuration edit needs to be inserted in the file patches.yml.

It is important to observe that these modifications affect the first filter chain and subsequently the first filter definition.

patches.yml
- listener_name: frontend
  # https://www.envoyproxy.io/docs/envoy/v1.29.3/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#envoy-v3-api-msg-extensions-filters-network-http-connection-manager-v3-httpconnectionmanager-upgradeconfig
  'filter_chains.0.filters.0.typed_config.upgrade_configs':
    upgrade_type: "websocket"
  'filter_chains.0.filters.0.typed_config.route_config.response_headers_to_add': [
    {
      "header": {
        "key": "X-Content-Type-Options",
        "value": "nosniff"
      },
      "append": false
    },
    {
      "header": {
        "key": "X-Download-Options",
        "value": "noopen"
      },
      "append": false
    },
    {
      "header": {
        "key": "Referrer-Policy",
        "value": "strict-origin-when-cross-origin"
      },
      "append": false
    }
  ]
caution

Please notice that the above configurations are specific for the frontend listener of Envoy. In case it has been decided to expose the Fast Data Runtime Management system under a different listener, please update the configuration accordingly.

Nginx​

If you are using the standard Nginx API Gateway, you have to:

  • add this custom location to api-gateway/server-extension.conf using the websocket endpoint as location (e.g. /fast-data/feedback)
  • if you want to use the Control Plane as iFrame, you need to expose the frontend on a different location (see full example below) and overwrite its headers in the Advanced section.
API Gateway Nginx Full Configuration

Consider a use case where:

  • Control Plane backend is exposed over /fast-data
  • Control Plane frontend is exposed over /control-plane (check the frontend documentation on how to overwrite the default base path)

Then, we have to write the following advanced configuration:

api-gateway/server-extension.conf
location /fast-data/feedback/ {
    proxy_pass http://$proxy_name$proxy_url;

    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    # WebSocket support
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection upgrade;
}

location /control-plane {
    proxy_pass http://$proxy_name$proxy_url;
    add_header X-Content-Type-Options nosniff;
    add_header X-Content-Type-Options noopen;
    add_header Referrer-Policy strict-origin-when-cross-origin;
}

Configuration​

Configuration of Control Plane is a straightforward process that involves setting up a ConfigMap and specifying essential environment variables.

Environment Variables​

Control Plane can be customized using the following environment variables:

NameRequiredDescriptionDefault Value
HTTP_PORT-This variable determines the TCP port where the HTTP controller binds its listener.3000
LOG_LEVEL-Specify the centralized application log level, choosing from options such as debug , error , fatal , info , silent , trace , or warn .info
FD_CONTROL_PLANE_CONFIGURATION_FILEPATH-Set the location of the configuration file.~/.fd/control-plane/configuration.json

Config Map​

The configuration of the service is handled by a JSON file whose location is defined by the FD_CONTROL_PLANE_CONFIGURATION_FILEPATH.

The Control Plane applications provide a sample of this JSON in a dedicated Config Map, named control-plane-configuration.

Loading ....
info

Initially, only the persistence layer is required to be available at startup, as there could be no Fast Data runtime available.

Persistence​

The object persistence configures the layer of runtime state persistence where Control Plane stores the next desired state for its runtime.

It must point to a MongoDB collection, available in a replicaSet enabled database.

Available interfaces include:

  • MongoDB client
Example​
{
    // ...other control plane configurations
    "persistence": {
        "type": "mongodb",
        "configuration": {
            "url": "mongodb://localhost:27017/fast-data?replicaSet=local"
        },
        "collection": {
            "name": "runtime-state"
        }
    }
    // ...other control plane configurations
}

Secret Resolution​

The following fields contains secrets that can be managed using the Fast Data Secrets Resolution mechanism:

  • persistence.configuration.url
  • runtime.feedback.configuration.['bootstrap.servers']
  • runtime.feedback.configuration.['sasl.username']
  • runtime.feedback.configuration.['sasl.password']
  • runtime.feedback.configuration.['ssl.truststore.certificates']
  • runtime.feedback.configuration.['ssl.keystore.key']
  • runtime.feedback.configuration.['ssl.key.password']
  • runtime.feedback.configuration.['ssl.keystore.certificate.chain']
  • runtime.state.configuration.['bootstrap.servers']
  • runtime.state.configuration.['sasl.username']
  • runtime.state.configuration.['sasl.password']
  • runtime.state.configuration.['ssl.truststore.certificates']
  • runtime.state.configuration.['ssl.keystore.key']
  • runtime.state.configuration.['ssl.key.password']
  • runtime.state.configuration.['ssl.keystore.certificate.chain']
  • settings.server.authorization.apiKey

Examples​

Here are listed some configuration examples that can be useful as a starting point for your setup.

Control Plane with Kafka Channels

{
    "runtime": {
        "state": {
            "type": "kafka",
            "configuration": {
                "bootstrap.servers": "localhost:9094",
                "allow.auto.create.topics": "true"
            },
            "producer": {
                "topic": "state"
            }
        },
        "feedback": {
            "type": "kafka",
            "configuration": {
                "bootstrap.servers": "localhost:9094",
                "allow.auto.create.topics": "true"
            },
            "producer": {
                "topic": "feedback",
                "group.id": "feedback-group"
            }
        }
    },
    "persistence": {
        "type": "mongodb",
        "configuration": {
            "url": "mongodb://localhost:27017/fast-data?replicaSet=local"
        },
        "collection": {
            "name": "runtime-state"
        }
    },
    "settings": {
        "server": {
            "prefix": "/control-plane/",
            "apis": {
                "controllers": {
                    "fast-data": "/my-awesome-controller"
                }
            },
            "website": {
                "url": "/__frontend/"
            }
        }
    }
}

Control Plane with gRPC channels

{
    "runtime": {
        "state": {
            "type": "grpc"
        },
        "feedback": {
            "type": "grpc"
        }
    },
    "persistence": {
        "type": "mongodb",
        "configuration": {
            "url": "mongodb://localhost:27017/fast-data?replicaSet=local"
        },
        "collection": {
            "name": "runtime-state"
        }
    },
    "settings": {
        "server": {
            "prefix": "/control-plane/",
            "apis": {
                "controllers": {
                    "fast-data": "/my-awesome-controller"
                }
            },
            "website": {
                "url": "/__frontend/"
            }
        }
    }
}