Skip to main content
Version: 13.x (Current)

Control Plane

Fast Data Control Plane allows to:

  • manage Fast Data Runtimes, in particular groups them in Runtime Views;
  • keep track of the state of the different pipelines belonging to all the managed runtimes;
  • receive user interactions through Control Plane, and to update pipeline state accordingly;
  • receive feedbacks from the Control Plane Operators that registered to this specific Control Plane instance;

In the following paragraphs is described Control Plane service configuration and communication interfaces.

Configuration​

Configuration of Control Plane is a straightforward process that involves setting up a ConfigMap and specifying essential environment variables.

Environment Variables​

Control Plane service can be customized using the following environment variables:

NameRequiredDescriptionDefault Value
HTTP_PORT-This variable determines the TCP port where the HTTP controller binds its listener3000
GRPC_PORT-This variable determines the TCP port where the gRPC controller binds its listener50051
LOG_LEVEL-Specify the centralized application log level, choosing from options such as debug, error, info, trace or warninfo
FD_CONTROL_PLANE_CONFIGURATION_FILEPATH-Set the location of the configuration file~/.fd/control_plane/configuration.json
OTEL_EXPORTER_OTLP_ENDPOINT-The URL to a GRPC endpoint of an OpenTelemetry Collector.
Specifying this value enables the support for OpenTelemetry tracing

Config Map​

The configuration of the service is handled by a JSON file whose location is defined by the FD_CONTROL_PLANE_CONFIGURATION_FILEPATH. When instantiating Control Plane application from Marketplace, Control Plane service configuration is generated with a dedicated Config Map, named control-plane-configuration.
This file contains a template configuration that should help in configuring the service.

Loading ....

Feedback Cache​

info

Currently only Redis is supported as feedback cache for storing relevant data, such as current pipelines state.

Control Plane receive feedbacks from all the registered Control Plane Operators, which may vary in the number depending on how many Fast Data Runtimes are selected for management. In order to decouple the internal feedback flow from the live updates stream opened towards the Control Plane UI and to support service replication, Control Plane service leverages an external cache to store those Fast Data Runtime states.
Cache details can be declared in the configuration file under the cache.configuration property, whose main fields are:

  • url → the connection string to your Redis instance;

An example of cache configuration can be seen below:

{
  "cache": {
    "type": "redis",
    "configuration": {
      "url": "redis://<redis-instance-address>:6379"
    }
  },
  // ...other control plane configurations
}
tip

The following properties support secret resolution:

  • cache.configuration.url
note

When instantiating Fast Data Control Plane application, a small Redis instance is added to your project, ready for supporting Control Plane operator. In case you would like to adopt your own Redis instance, please update the generated configuration accordingly.

Persistence Layer​

info

Currently only MongoDB is supported as persistence layer for storing relevant data.

caution

The MongoDB database selected for storing Control Plane data must be configured to have replicaSet enabled, since Control Plane exploits features that can be used only when a replicaSet is available.

In order to carry out all its operations, Control Plane requires a persistence layer where relevant information are stored. In particular, it stores the next desired state of each Fast Data Runtime, which the system needs to reconcile with, alongside details regarding Runtime Views and Pipelines components.

This configuration can be set under the persistence.configuration key of the configuration file. The main properties are:

  • url → the connection string to your MongoDB instance with replicaSet enabled;
  • database → the database name where to search for the collections relevant to Control Plane service. Please notice that setting this property will override the database name potentially set in the connection string;

An example of persistence configuration can be seen below:

{
  // ...other control plane configurations
  "persistence": {
    "type": "mongodb",
    "configuration": {
      "url": "mongodb://<server>:27017/<default-database>?replicaSet=local",
      "database": "<data-fabric-database-name>"
    }
  },
  // ...other control plane configurations
}
tip

The following properties support secret resolution:

  • persistence.configuration.url
  • persistence.configuration.database

Service Settings​

Additionally, the Control Plane service itself has a set of properties for changing its behavior. Here are listed the available ones within settings properties:

  • apiPrefix → the base path applied to all the exposed routes. It defaults to /;
  • auditUserHeader → specifies in which HTTP header can be found the user identifier set by the authentication system. The value of this header will be employed to correlate requests stored by the auditing system with the user that performed them. When using Mia-Platform Authentication and Authorization services this property can be set to miauserid.
    In case it is not set the auditing system does not correlate users with requests;

Here can be found a configuration example:

{
  // ...other control plane configurations
  "settings": {
    "apiPrefix": "/",
    "auditUserHeader": "miauserid"
  }
}

Enable gRPC communications​

Control Plane main service must be reachable through gRPC from Control Plane Operators deployed in other namespaces either internally, within your K8s cluster, or externally from outside. In this manner Control Plane Operators can register themselves on the main Control Plane instance and establish a channel where desired pipelines states are received and Fast Data Runtimes states are sent back.

Thus, it is necessary to advertise the port where the gRPC controller is exposed by Control Plane service, which by default is the 50051. This operation can be achieved by adding the proper port to the list of Container Ports that can be found in the Console Design area, under the specific microservice resource. The expected list, based on default configuration, is shown below in the image.

Control Plane microservice ports

tip

When instantiating Fast Data Control Plane application, Container Ports are pre-filled with all the needed ports using their default value.
In case either the HTTP or the gRPC port chosen through environment variables has been edited, please change the Container Ports accordingly.

Reach Control Plane from within your cluster​

When connecting to Control Plane service from another namespace of your cluster, depending on your additional K8s rules and policies, it shouldn't be necessary to apply any further configuration. If in doubt, please contact your K8s cluster administrator to learn whether additional communication restrictions were put in place.

Reach Control Plane from outside your cluster​

In case Control Plane service should be configured to be reachable from outside your cluster, for example because your Fast Data Control Plane application is located in a cluster different from the one where Fast Data Runtimes are deployed, a few extra advanced configuration steps are necessary. These are listed here:

  • Introduce further Envoy cluster to expose the gRPC port of Control Plane service through the API Gateway.
    This can be obtained by opening your Mia-Platform Console Project and selecting the Advanced tab from the Design area's sidebar. Within this area of Console it is possible to extend Envoy API Gateway default configuration, by editing the file api-gateway-envoy/clusters.yaml with the following content:

    api-gateway-envoy/clusters.yaml
    - "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
      name: control-plane
      connect_timeout: 30s
      http2_protocol_options:
        max_concurrent_streams: 100
      type: LOGICAL_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: control-plane
        endpoints:
        - lb_endpoints:
          - endpoint:
            address:
              socket_address:
                address: 'control-plane'
                port_value: 50051

  • Expose new subdomain from which Control Plane gets accessible (assuming traefik is employed as ingress controller)
    In order to carry out this operation you first need to have access to the repository of the Console Project where Control Plane application has been configured. Within the project repository there is a folder named overlays, which should contains a folder for each environment available for the project.
    For each of the environment of interest, open the folder and edit the default.ingressroute.yml by adding the following record under the spec.routes array:

    - match: Host(`grpc-<PROJECT_DOMAIN>`)
      middlewares:
        - name: "ingress-controller-hsts-headers"
          namespace: mia-platform
      kind: Rule
      services:
        - name: api-gateway
          port: 50051
          scheme: h2c
          passHostHeader: true

    where <PROJECT_DOMAIN> should be replaced with the project's domain for currently selected environment. An example could be my-cool-project.test.my-domain.com.

Endpoints​

When connecting internally to Control Plane service, it is not necessary to publicly expose any endpoint, since all the Control Plane routes can be reached internally or via Fabric BFF.

In case the service must be reached from outside the cluster, the endpoints listed in the table below should be defined in Console and assigned to Control Plane service. These endpoints are necessary to allow gRPC Control Plane Operator requests to reach Control Plane from outside the K8s cluster.

EndpointRewrite Base PathMicroservice
/grpc.reflection.v1.ServerReflection/grpc.reflection.v1.ServerReflectioncontrol-plane
/grpc.reflection.v1alpha.ServerReflection/grpc.reflection.v1alpha.ServerReflectioncontrol-plane
/control_plane_fabric.RuntimeManagement/control_plane_fabric.RuntimeManagementcontrol-plane
/control_plane_fabric.ControlPlane/control_plane_fabric.ControlPlanecontrol-plane

Routes​

Here are described which routes Control Plane service serves

EndpointTypeMethodDescription
/fast-data/runtimes/itemsRESTGETList existing runtime views
/fast-data/runtimes/itemsRESTPOSTCreate a new runtime view
/fast-data/runtimes/items/:idRESTGETRetrieve the details of specified runtime view
/fast-data/runtimes/items/:idRESTPATCHUpdate the details of specified runtime view
/fast-data/runtimes/items/:idRESTDELETEDelete specified runtime view
/fast-data/runtimes/items/:id/pipelinesRESTGETReturn the list of pipelines associated to specified rumtime
/fast-data/runtimes/statsRESTGETReturn additional details for each runtime, such as the number of pipelines registered on them
/grpc.reflection.v1.ServerReflectiongRPCgRPC reflection service that describes protobuf-defined API
/grpc.reflection.v1alpha.ServerReflectiongRPCgRPC reflection service that describes protobuf-defined API
/control_plane_fabric.RuntimeManagementgRPCgRPC service that allows Control Plane Operator to register itself on the Control Plane
/control_plane_fabric.ControlPlanegRPCgRPC service for exchanging Fast Data desired state and runtime feedback between Control Plane and Control Plane Operator
/control_plane_fabric.FastDataControlgRPCgRPC service for exchanging change actions and runtime feedback between Control Plane and Fabric BFF