Skip to main content
Version: 13.x (Current)

Control Plane Operator

Fast Data Control Plane Operator is a service that has to be deployed in each Fast Data Runtime (a Console project released in a specific environment) whose main objectives are:

  • recognize in which runtime it is executing and register itself on the main Control Plane instance to acknowledge its existence within detected runtime;
  • expose an interface towards Fast Data workloads to allow a bidirectional communication with them where it can:
  • send to the main Control Plane instance the workloads state (e.g. whether they are up-and-running, active or paused) of the Fast Data Pipeline they are linked to;

Configuration​

Configuration of Control Plane Operator is a straightforward process that involves setting up a ConfigMap and specifying essential environment variables.

Environment Variables​

Control Plane service can be customized using the following environment variables:

NameRequiredDescriptionDefault Value
GRPC_PORT-This variable determines the TCP port where the gRPC controller binds its listener50051
LOG_LEVEL-Specify the centralized application log level, choosing from options such as debug, error, info, trace or warninfo
FD_OPERATOR_CONFIGURATION_FILEPATH-Set the location of the configuration file~/.fd/control_plane/configuration.json
OTEL_EXPORTER_OTLP_ENDPOINT-The URL to a GRPC endpoint of an OpenTelemetry Collector.
Specifying this value enables the support for OpenTelemetry tracing
MIA_PLATFORM_EU_TENANT✓Identifier of the company that contains the project where this service is configured.
This is retrieved exploiting Downward API configuration
APP_KUBERNETES_IO_PART_OF✓Identifier of the project where this service is configured.
This is retrieved exploiting Downward API configuration
MIA_PLATFORM_EU_STAGE✓Environment where this service is deployed to.
This is retrieved exploiting Downward API configuration
tip

This is how environment variables using Downward API should look like under Control Plane Operator microservice page of Console Design area:

Environment Variables from Downward API

Config Maps​

The configuration of the service is handled by a JSON file whose location is defined by the FD_OPERATOR_CONFIGURATION_FILEPATH. When instantiating Control Plane application from Marketplace, Control Plane service configuration is generated with a dedicated Config Map, named control-plane-operator-config.
This file contains a template configuration that should help in configuring the service.

Loading ....

Upstream​

Control Plane Operator needs to know the address of main Control Plane instance, so that it can send to it the feedbacks collected from the different Fast Data workloads. This can be configured through the url field under upstream property in the configuration file.

An example of a simple configuration is shown here, where it is assumed that Control Plane service is located within data-fabric namespace:

{
  "upstream": {
    "url": "http://control-plane.data-fabric.svc.cluster.local:50051"
  }
}

The url can use both http and https protocol, although the former is used as default. The latter can be configured as default connection protocol setting the property assume_http2 to true.
Furthermore, it is possible to customize the request headers by adding ad-hoc ones within the property headers. This can be useful for setting an Authorization header or an API key in the expected header.

Below is provided a complete example of the configuration:

{
  "upstream": {
    "url": "http://control-plane.data-fabric.svc.cluster.local:50051",
    "assume_http2": true,
    "headers": {
      "secret": {
        "type": "env",
        "key": "CP_OPERATOR_API_KEY"
      }
    }
  }
}
tip

The following properties support secret resolution:

  • upstream.headers.*

Pipelines Configuration​

Control Plane Operator requires an additional configuration file to learn the topology of Fast Data Pipelines deployed within the runtime of execution. Such information is automatically generated and added by Fast Data Configurator to the Control Plane Operator when it recognizes it as a known plugin, that is when Control Plane Operator service has been instantiated from Mia-Platform Marketplace.

danger

To configure Control Plane Operator, always start from the Mia-Marketplace plugin item.

When configuring Control Plane Operator without using the corresponding Mia-Platform Marketplace item, such automatic configuration is not added by Fast Data Configurator, so that the service won't be able to recognize Fast Data Pipelines and match them with state feedbacks collected from Fast Data workload.

Enable gRPC communications​

Control Plane Operator service must be reachable through gRPC from Fast Data Workloads deployed in its namespace. In this manner Fast Data Workloads can receive state updates and send back their actual status as feedback.

Thus, it is necessary to expose the port where the gRPC controller runs in the Control Plane service, which by default is the 50051. This operation can be achieved by setting the proper port to the list of Container Ports that can be found in the Console Design area, under the specific microservice resource. In the image below it is shown how the container ports card should appear:

Control Plane Operator microservice ports

tip

When instantiating Fast Data Control Plane application, Container Ports are pre-filled with all the needed ports using their default value.
In case either the HTTP or the gRPC port chosen through environment variables has been edited, please change the Container Ports accordingly.

Runtime Communication​

Communications to/from runtime's workloads happen through a bidirectional channel whose State and Feedback messages are exchanged in their respective direction.

State Message​

The State Message is employed to notify the desired state of each pipeline to the underlying workloads. Each pipeline has a general state, that can be overridden.
For each pipeline, there could be different overrides that can change individually the state of a particular artifact. Also, each artifact can have additional overrides for each execution stage of the pipeline where is employed.

The state that can be set at pipeline level, or artifact level, or execution stage level is an Activation State, which can be:

  • pause → request workloads to stop consuming from the data stream
  • resume → request workloads to resume consuming the data stream
Loading ....

Feedback Channel​

The Feedback Message defines the format for workloads to notify the Control Plane Operator about the state of their artifacts. The information that can be extracted from these messages is:

  • which artifacts are up-and-running;
  • which workloads have already successfully reconciled their state with the one requested in the latest State Message;

Workloads connected to a Control Plane Operator periodically generate and send these messages through the established bidirectional channel.

Loading ....