Control Plane Workloads Configuration

This section helps you configure new or existing Fast Data services so that they can communicate with the Control Plane Operator.

Compatibility Matrix

To see if your microservices have support for your Control Plane Operator version, give a look at the compatibility matrix section.

As mentioned in the Overview section, a service configured to interact with the Control Plane will have all its artifacts (topics, projections or single views) in a paused state, meaning that no data will be processed until a resume interaction will be requested by the user.

danger

If you are configuring Control Plane configurations for existing Fast Data workloads, they will be released in a paused state after deployment and they will not process data.

This may cause some delay in your architecture.

To reduce as much possible this delay, you can relay on Bulk actions to quickly resume entire pipeline stages.

Configuration

To enable a Fast Data service to communicate with a Control Plane instance, you have to define a JSON configuration that should match the one given to the Control Plane.

The JSON object has the same properties for all Fast Data microservices and must be compliant to the following JSON Schema.

Schema Viewer

Loading ....

Raw JSON Schema

{
  "$schema": "http://json-schema.org/draft-07/schema",
  "type": "object",
  "title": "Control Plane Client JSON Schema Configuration",
  "examples": [
    {
      "state": {
        "type": "grpc"
      },
      "feedback": {
        "type": "grpc"
      },
      "settings": {
        "grpc": {
          "host": "control-plane-operator"
        }
      }
    }
  ],
  "definitions": {
    "settings": {
      "title": "Settings",
      "description": "additional settings to provide for configuration",
      "properties": {
        "channel": {
          "type": "object",
          "allOf": [
            {
              "$ref": "#/definitions/heartbeat"
            }
          ]
        },
        "grpc": {
          "type": "object",
          "title": "Grpc Configuration",
          "properties": {
            "mode": {
              "type": "string",
              "title": "Grpc Mode",
              "enum": [
                "protobuf",
                "json"
              ]
            },
            "port": {
              "type": "number",
              "default": 50051
            },
            "host": {
              "type": "string",
              "default": "0.0.0.0"
            }
          }
        }
      }
    },
    "heartbeat": {
      "title": "Heartbeat Configuration",
      "type": "object",
      "properties": {
        "heartbeat.ms": {
          "type": "number",
          "description": "Number of milliseconds between each heartbeat message",
          "default": 2500
        }
      }
    },
    "connections": {
      "grpc": {
        "title": "Grpc Connection Configuration",
        "properties": {
          "type": {
            "const": "grpc"
          }
        },
        "required": [
          "type"
        ]
      }
    },
    "state": {
      "title": "State Channel Configuration",
      "type": "object",
      "description": "This section contains the configuration of the state channel that will received commands from Control Plane",
      "oneOf": [
        {
          "$ref": "#/definitions/connections/grpc"
        }
      ]
    },
    "feedback": {
      "title": "Feedback Channel Configuration",
      "type": "object",
      "description": "This section contains the configuration of the feedback channel that will send heartbeat messages to the Control Plane.",
      "oneOf": [
        {
          "$ref": "#/definitions/connections/grpc"
        }
      ]
    }
  },
  "required": [
    "state",
    "feedback"
  ],
  "properties": {
    "state": {
      "$ref": "#/definitions/state"
    },
    "feedback": {
      "$ref": "#/definitions/feedback"
    },
    "settings": {
      "$ref": "#/definitions/settings"
    }
  },
  "additionalProperties": false
}

Example

{
  "state": {
    "type": "grpc"
  },
  "feedback": {
    "type": "grpc"
  },
  "settings": {
    "grpc": {
      "host": "control-plane-operator"
    }
  }
}

Choose below the service you want to configure with Control Plane to see where to place this configuration.

Projection Storer

Projection Storer

To configure the Control Plane connection for the Projection Storer, first you need to attach the microservice to a System of Record.

Then, you can scroll over the Control Plane section and write your JSON configuration. More details on the Control Plane configuration for Projection Storer service can be found in the service configuration page.

Real-Time Updater

Real-Time Updater

The Control Plane Configuration is managed by a dedicated Config Map.

In the Config Map section of the Design Area, create a new Config Map (e.g. control-plane-configuration), then create a JSON file (e.g. config.json) where you will write your own configuration.

To made the service aware of the Control Plane configuration, add a new environment variable called CONTROL_PLANE_CONFIG_PATH where you will use as value the full path of the configuration file inside your microservice.

tip

If the Control Plane is configured with both feedback and state as gRPC channels, the same Config Map can be shared across microservices.

Single View Trigger Generator

Single View Trigger Generator

The Control Plane Configuration is managed by a dedicated Config Map.

In the Config Map section of the Design Area, create a new Config Map (e.g. control-plane-configuration), then create a JSON file (e.g. config.json) where you will write your own configuration.

To made the service aware of the Control Plane configuration, add a new environment variable called CONTROL_PLANE_CONFIG_PATH where you will use as value the full path of the configuration file inside your microservice.

tip

If the Control Plane is configured with both feedback and state as gRPC channels, the same Config Map can be shared across microservices.

Single View Creator

Single View Creator

The Control Plane Configuration is managed by a dedicated Config Map.

In the Config Map section of the Design Area, create a new Config Map (e.g. control-plane-configuration), then create a JSON file (e.g. config.json) where you will write your own configuration.

To made the service aware of the Control Plane configuration, add a new environment variable called CONTROL_PLANE_CONFIG_PATH where you will use as value the full path of the configuration file inside your microservice.

tip

If the Control Plane is configured with both feedback and state as gRPC channels, the same Config Map can be shared across microservices.

gRPC Host Resolution

To enable gRPC communication between the Control Plane Operator and the Fast Data runtime, services within the Fast Data runtime should reference the gRPC server contained in the Control Plane Operator using its corresponding Kubernetes hostname.

For example, consider a use case where we have a Control Plane Operator service named control-plane-operator. Then, workloads should have the settings properties configured as follows:

{
  // ...
  "settings": {
    "grpc": {
      "host": "control-plane-operator" 
      // or control-plane-operator.svc.cluster.local, etc...
    }
  }
  // ...
}

Bind Microservices to Runtime

Once a Control Plane configuration has been defined for one of the workloads above mentioned, upon saving, the Console will create an additional configuration, that we refer to as bindings, that will contain the mapping between the Projections or the Single Views with a specific pipeline and its own artifacts.

This configuration will be read-only and:

• for the Projection Storer, the configuration will be managed internally by the microservice;
• for other microservices, an additional environment variable CONTROL_PLANE_BINDINGS_PATH managed by Mia-Platform Console will be set-up with the full path of the new configuration.