Control Plane Operator
Fast Data Control Plane Operator is a service that has to be deployed in each Fast Data Runtime (a Console project released in a specific environment) whose main objectives are:
- recognize in which runtime it is executing and register itself on the main Control Plane instance to acknowledge its existence within detected runtime;
- expose an interface towards Fast Data workloads to allow a bidirectional communication with them where it can:
- forward on the state channel the desired state that has been requested by users;
- gather actual workloads state from the feedback channel;
- send to the main Control Plane instance the workloads state (e.g. whether they are up-and-running, active or paused) of the Fast Data Pipeline they are linked to;
Configuration
Configuration of Control Plane Operator is a straightforward process that involves setting up a ConfigMap and specifying essential environment variables.
Environment Variables
Control Plane service can be customized using the following environment variables:
Name | Required | Description | Default Value |
---|---|---|---|
GRPC_PORT | - | This variable determines the TCP port where the gRPC controller binds its listener | 50051 |
LOG_LEVEL | - | Specify the centralized application log level, choosing from options such as debug , error , info , trace or warn | info |
FD_OPERATOR_CONFIGURATION_FILEPATH | - | Set the location of the configuration file | ~/.fd/control_plane/configuration.json |
OTEL_EXPORTER_OTLP_ENDPOINT | - | The URL to a GRPC endpoint of an OpenTelemetry Collector. Specifying this value enables the support for OpenTelemetry tracing | |
MIA_PLATFORM_EU_TENANT | ✓ | Identifier of the company that contains the project where this service is configured. This is retrieved exploiting Downward API configuration | |
APP_KUBERNETES_IO_PART_OF | ✓ | Identifier of the project where this service is configured. This is retrieved exploiting Downward API configuration | |
MIA_PLATFORM_EU_STAGE | ✓ | Environment where this service is deployed to. This is retrieved exploiting Downward API configuration |
This is how environment variables using Downward API should look like under Control Plane Operator microservice page of Console Design area:
Config Maps
The configuration of the service is handled by a JSON file whose location is defined by the FD_OPERATOR_CONFIGURATION_FILEPATH
. When instantiating
Control Plane application from Marketplace, Control Plane service configuration is generated with
a dedicated Config Map, named control-plane-operator-config
.
This file contains a template configuration that should help in configuring the service.
- Schema Viewer
- Raw JSON Schema
- Example
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Configuration",
"type": "object",
"required": [
"upstream"
],
"properties": {
"upstream": {
"$ref": "#/definitions/Upstream"
}
},
"examples": [
{
"upstream": {
"url": "http://control-plane.<namespace>.svc.cluster.local:50051"
}
},
{
"upstream": {
"url": "https://external-url-to-access-control-plane",
"headers": {
"secret": {
"type": "env",
"key": "CP_OPERATOR_API_KEY"
}
}
}
}
],
"definitions": {
"Upstream": {
"type": "object",
"required": [
"url"
],
"properties": {
"assume_http2": {
"default": false,
"type": "boolean"
},
"headers": {
"default": {},
"type": "object",
"additionalProperties": {
"$ref": "#/definitions/secret"
}
},
"url": {
"type": "string"
}
}
},
"secret": {
"oneOf": [
{
"type": "string"
},
{
"type": "object",
"required": [
"key",
"type"
],
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"type": {
"const": "env"
}
}
},
{
"type": "object",
"required": [
"path",
"type"
],
"properties": {
"encoding": {
"type": "string",
"enum": [
"base64"
]
},
"key": {
"type": "string"
},
"path": {
"type": "string"
},
"type": {
"const": "file"
}
}
}
]
}
}
}
{
"upstream": {
"url": "http://control-plane.<namespace>.svc.cluster.local:50051"
}
}
Upstream
Control Plane Operator needs to know the address of main Control Plane instance, so that it can send to it the feedbacks collected
from the different Fast Data workloads. This can be configured through the url
field under upstream
property in the configuration file.
An example of a simple configuration is shown here, where it is assumed that Control Plane service is located within data-fabric
namespace:
{
"upstream": {
"url": "http://control-plane.data-fabric.svc.cluster.local:50051"
}
}
The url
can use both http
and https
protocol, although the former is used as default. The latter can be configured as default connection
protocol setting the property assume_http2
to true
.
Furthermore, it is possible to customize the request headers by adding ad-hoc ones within the property headers
. This can be useful for setting an
Authorization
header or an API key in the expected header.
Below is provided a complete example of the configuration:
{
"upstream": {
"url": "http://control-plane.data-fabric.svc.cluster.local:50051",
"assume_http2": true,
"headers": {
"secret": {
"type": "env",
"key": "CP_OPERATOR_API_KEY"
}
}
}
}
The following properties support secret resolution:
upstream.headers.*
Pipelines Configuration
Control Plane Operator requires an additional configuration file to learn the topology of Fast Data Pipelines deployed within the runtime of execution. Such information is automatically generated and added by Fast Data Configurator to the Control Plane Operator when it recognizes it as a known plugin, that is when Control Plane Operator service has been instantiated from Mia-Platform Marketplace.
To configure Control Plane Operator, always start from the Mia-Marketplace plugin item.
When configuring Control Plane Operator without using the corresponding Mia-Platform Marketplace item, such automatic configuration is not added by Fast Data Configurator, so that the service won't be able to recognize Fast Data Pipelines and match them with state feedbacks collected from Fast Data workload.
Enable gRPC communications
Control Plane Operator service must be reachable through gRPC from Fast Data Workloads deployed in its namespace. In this manner Fast Data Workloads can receive state updates and send back their actual status as feedback.
Thus, it is necessary to expose the port where the gRPC controller runs in the Control Plane service, which by default is the 50051
.
This operation can be achieved by setting the proper port to the list of Container Ports
that can be found in the Console Design area, under the specific microservice resource. In the image below it is shown how
the container ports card should appear:
When instantiating Fast Data Control Plane application, Container Ports are pre-filled with all the needed ports using their default value.
In case either the HTTP or the gRPC port chosen through environment variables has been edited, please change the Container Ports accordingly.
Runtime Communication
Communications to/from runtime's workloads happen through a bidirectional channel whose State and Feedback messages are exchanged in their respective direction.
State Message
The State Message is employed to notify the desired state of each pipeline
to the underlying workloads. Each pipeline has a general state, that can be overridden.
For each pipeline, there could be different overrides
that can change individually the state of a particular
artifact. Also, each artifact can have additional overrides
for each execution stage of the pipeline where is employed.
The state that can be set at pipeline level, or artifact level, or execution stage level is an Activation State, which can be:
pause
→ request workloads to stop consuming from the data streamresume
→ request workloads to resume consuming the data stream
- Schema Viewer
- Raw JSON Schema
- Example
{
"$schema": "http://json-schema.org/draft-07/schema",
"$id": "fast-data-state.schema.json",
"title": "FastDataState",
"examples": [
{
"key": {
"id": "854c2285-334e-41d1-bcfc-8c483fafca08"
},
"value": {
"pipelines": {
"state": "resume"
}
}
},
{
"key": {
"id": "d2d4fb77-94e9-45cb-ad2a-4ad09040b796"
},
"value": {
"pipelines": {
"state": "pause"
}
}
},
{
"key": {
"id": "bd1643e1-7f53-4b84-93ce-a979d2a5f9cb"
},
"value": {
"pipelines": {
"state": "pause",
"override": {
"b52c98ec-4910-456a-9ac4-13ad82dac404": {
"state": "resume",
"name": "sv_orders"
}
}
}
}
},
{
"key": {
"id": "854c2285-334e-41d1-bcfc-8c483fafca08"
},
"value": {
"pipelines": {
"state": "pause",
"override": {
"b52c98ec-4910-456a-9ac4-13ad82dac404": {
"state": "pause",
"name": "sv_orders",
"override": {
"7e738df3-d963-4554-a8aa-3ca3c55bcf22": {
"state": "resume",
"name": "base-prj"
}
}
}
}
}
}
},
{
"key": {
"id": "854c2285-334e-41d1-bcfc-8c483fafca08"
},
"value": {
"pipelines": {
"state": "pause",
"override": {
"b52c98ec-4910-456a-9ac4-13ad82dac404": {
"state": "pause",
"name": "sv_orders",
"override": {
"7e738df3-d963-4554-a8aa-3ca3c55bcf22": {
"state": "pause",
"name": "base-prj",
"override": {
"b9dc9fc8-2d06-4be5-9950-dd2088cfc034": {
"state": "resume",
"name": "aggregation"
}
}
}
}
}
}
}
}
},
{
"key": {
"id": "854c2285-334e-41d1-bcfc-8c483fafca08"
},
"value": {
"pipelines": {
"state": "pause",
"override": {
"b52c98ec-4910-456a-9ac4-13ad82dac404": {
"state": "pause",
"name": "orders-domain",
"override": {
"ab2703e8-b13b-42a3-af2c-c3c2f7d97792": {
"state": "pause",
"name": "proj-a",
"override": {
"daeb6959-ddf2-4c03-b237-ff57b2e1ea4d": {
"state": "resume",
"name": "mapping"
},
"a161dd38-7979-4156-8551-3c8c6386bfc4": {
"state": "resume",
"name": "backup"
}
}
}
}
}
}
}
}
}
],
"definitions": {
"activation": {
"type": "string",
"title": "Activation",
"description": "describe the values a pipeline state can take",
"enum": [
"resume",
"pause"
]
},
"stateful": {
"type": "object",
"title": "StatefulItem",
"required": [
"state"
],
"properties": {
"state": {
"$ref": "#/definitions/activation"
}
}
},
"named": {
"type": "object",
"title": "NamedItem",
"required": [
"name"
],
"properties": {
"name": {
"type": "string"
}
}
},
"artifactItem": {
"title": "ArtifactItem",
"allOf": [
{
"$ref": "#/definitions/stateful"
},
{
"$ref": "#/definitions/named"
},
{
"type": "object",
"properties": {
"override": {
"type": "object",
"description": "mapping between pipeline exec types and their corresponding state override",
"additionalProperties": {
"allOf": [
{
"$ref": "#/definitions/stateful"
},
{
"$ref": "#/definitions/named"
}
]
}
}
}
}
]
},
"eventPayload": {
"type": "object",
"title": "FastDataStateValue",
"required": [
"pipelines"
],
"properties": {
"pipelines": {
"allOf": [
{
"$ref": "#/definitions/stateful"
},
{
"type": "object",
"properties": {
"override": {
"type": "object",
"description": "mapping between pipelines UUID and their corresponding state override",
"additionalProperties": {
"allOf": [
{
"$ref": "#/definitions/stateful"
},
{
"$ref": "#/definitions/named"
},
{
"type": "object",
"properties": {
"override": {
"type": "object",
"description": "mapping between pipeline artifact reference and their corresponding state override",
"additionalProperties": {
"$ref": "#/definitions/artifactItem"
}
}
}
}
]
}
}
}
}
]
}
}
}
},
"type": "object",
"required": [
"key",
"value"
],
"additionalProperties": true,
"properties": {
"key": {
"type": "object",
"required": [
"id"
],
"properties": {
"id": {
"type": "string",
"description": "UUID representing a reference to the runtime"
}
}
},
"value": {
"$ref": "#/definitions/eventPayload"
}
}
}
{
"key": {
"id": "854c2285-334e-41d1-bcfc-8c483fafca08"
},
"value": {
"pipelines": {
"state": "resume"
}
}
}
Feedback Channel
The Feedback Message defines the format for workloads to notify the Control Plane Operator about the state of their artifacts. The information that can be extracted from these messages is:
- which artifacts are up-and-running;
- which workloads have already successfully reconciled their state with the one requested in the latest State Message;
Workloads connected to a Control Plane Operator periodically generate and send these messages through the established bidirectional channel.
- Schema Viewer
- Raw JSON Schema
- Example
{
"$schema": "http://json-schema.org/draft-07/schema",
"$id": "service-feedback.schema.json",
"title": "ServiceFeedback",
"examples": [
{
"key": {
"pipeline": {
"id": "ec0ae4b1-e8aa-4a82-b5d8-d260e0eed340",
"name": "sv pipeline 1"
}
},
"value": {
"edges": [
{
"artifact": {
"id": "7a171212-2da0-4c60-9d7c-67f80eca91cb",
"name": "books_proj"
},
"exec": {
"id": "06e4f2e2-0e0f-4c41-85af-1c6a6075965f",
"name": "mapping"
},
"state": "running"
},
{
"artifact": {
"id": "8de96793-7cef-4867-9ce3-3a2fa23f9cdc",
"name": "orders_proj"
},
"exec": {
"id": "10c1165e-e5b2-4240-8990-c9b8b078c66e",
"name": "mapping"
},
"state": "running"
},
{
"artifact": {
"id": "8de96793-7cef-4867-9ce3-3a2fa23f9cdc",
"name": "orders_proj"
},
"exec": {
"id": "b8c1b719-56bb-421a-b37b-cce472ccfb8f",
"name": "backup"
},
"state": "paused"
}
]
},
"headers": {
"version": "1"
}
},
{
"key": {
"pipeline": {
"id": "ec0ae4b1-e8aa-4a82-b5d8-d260e0eed340",
"name": "sv pipeline 1"
}
},
"value": {
"edges": [
{
"artifact": {
"id": "f719e097-5f06-48e8-9eca-b43b634bf450",
"name": "books_proj"
},
"exec": {
"id": "83f18349-2d2b-4682-ac1c-440b88bf1643",
"name": "aggregation"
},
"state": "paused"
}
]
},
"headers": {
"version": "1"
}
},
{
"key": {
"pipeline": {
"id": "ec0ae4b1-e8aa-4a82-b5d8-d260e0eed340",
"name": "sv pipeline 1"
}
},
"value": {
"edges": [
{
"artifact": {
"id": "f719e097-5f06-48e8-9eca-b43b634bf450",
"name": "books_proj"
},
"exec": {
"id": "83f18349-2d2b-4682-ac1c-440b88bf1643",
"name": "aggregation"
},
"state": "paused",
"metadata": {
"throughput": "{\"value\":342,\"unit\":\"msg/s\"}"
}
}
]
},
"headers": {
"version": "1"
}
}
],
"definitions": {
"state": {
"type": "string",
"title": "RuntimeFeedback",
"description": "the actual operation state, extracted from the deployed service that is sending the feedback",
"enum": [
"running",
"paused"
]
},
"item": {
"type": "object",
"title": "ItemFeedback",
"required": [
"id",
"name"
],
"properties": {
"id": {
"type": "string",
"description": "unique identifier of the item"
},
"name": {
"type": "string",
"description": "human-readable name of the item - based on old Fast Data projection and single view collection identifiers"
}
}
}
},
"type": "object",
"required": [
"key",
"value"
],
"properties": {
"key": {
"type": "object",
"title": "HeartbeatKey",
"required": [
"pipeline"
],
"properties": {
"pipeline": {
"type": "object",
"required": [
"id",
"name"
],
"properties": {
"id": {
"type": "string"
},
"name": {
"type": "string"
}
}
}
}
},
"value": {
"type": "object",
"title": "HeartbeatPayload",
"required": [
"edges"
],
"properties": {
"edges": {
"type": "array",
"items": {
"type": "object",
"required": [
"artifact",
"exec",
"state"
],
"properties": {
"artifact": {
"$ref": "#/definitions/item"
},
"exec": {
"$ref": "#/definitions/item"
},
"state": {
"$ref": "#/definitions/state"
},
"metadata": {
"type": "object",
"additionalProperties": {
"type": "string"
}
}
}
}
}
}
},
"headers": {
"type": "object",
"title": "HeartbeatHeaders",
"properties": {
"version": {
"const": "1"
}
},
"additionalProperties": true
}
}
}
{
"key": {
"pipeline": {
"id": "ec0ae4b1-e8aa-4a82-b5d8-d260e0eed340",
"name": "sv pipeline 1"
}
},
"value": {
"edges": [
{
"artifact": {
"id": "7a171212-2da0-4c60-9d7c-67f80eca91cb",
"name": "books_proj"
},
"exec": {
"id": "06e4f2e2-0e0f-4c41-85af-1c6a6075965f",
"name": "mapping"
},
"state": "running"
},
{
"artifact": {
"id": "8de96793-7cef-4867-9ce3-3a2fa23f9cdc",
"name": "orders_proj"
},
"exec": {
"id": "10c1165e-e5b2-4240-8990-c9b8b078c66e",
"name": "mapping"
},
"state": "running"
},
{
"artifact": {
"id": "8de96793-7cef-4867-9ce3-3a2fa23f9cdc",
"name": "orders_proj"
},
"exec": {
"id": "b8c1b719-56bb-421a-b37b-cce472ccfb8f",
"name": "backup"
},
"state": "paused"
}
]
},
"headers": {
"version": "1"
}
}