Fabric BFF

Fabric BFF, as suggested by the name, works as Backend For Frontend for Data Catalog UI and it mediates requests between frontend and Data Catalog service.

Configuration

Configuration of Fabric BFF is a straightforward process that involves setting up a ConfigMap and specifying essential environment variables.

Environment Variables

Fabric BFF can be customized using the following environment variables:

Name	Required	Description	Default Value
HTTP_PORT	-	This variable determines the TCP port where the HTTP controller binds its listener	3000
LOG_LEVEL	-	Specify the centralized application log level, choosing from options such as debug, error, info, trace or warn	info
BFF_CONFIGURATION_FILEPATH	-	Set the location of the configuration file	~/.fd/data-fabric-bff/config.json
OTEL_EXPORTER_OTLP_ENDPOINT	-	The URL to a GRPC endpoint of an OpenTelemetry Collector. Specifying this value enables the support for OpenTelemetry tracing

Config Map

The configuration of the service is handled by a JSON file whose location is defined by the BFF_CONFIGURATION_FILEPATH.
When instantiating Data Catalog application, Fabric BFF service configuration is generated with a dedicated Config Map, named fabric-bff-config. This file contains a template configuration that should help in configuring the service.

Schema Viewer

Loading ....

Raw JSON Schema

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Configuration",
  "description": "BFF Service configuration",
  "type": "object",
  "required": [
    "persistence"
  ],
  "properties": {
    "console": {
      "anyOf": [
        {
          "$ref": "#/definitions/ConsoleSettings"
        },
        {
          "type": "null"
        }
      ]
    },
    "controlPlane": {
      "anyOf": [
        {
          "$ref": "#/definitions/ControlPlaneSettings"
        },
        {
          "type": "null"
        }
      ]
    },
    "openLineage": {
      "anyOf": [
        {
          "$ref": "#/definitions/OpenLineageSettings"
        },
        {
          "type": "null"
        }
      ]
    },
    "persistence": {
      "$ref": "#/definitions/Persistence"
    },
    "settings": {
      "$ref": "#/definitions/Settings"
    }
  },
  "definitions": {
    "AuthContext": {
      "oneOf": [
        {
          "type": "object",
          "oneOf": [
            {
              "type": "object",
              "required": [
                "credentials",
                "flow",
                "tokenEndpoint"
              ],
              "properties": {
                "credentials": {
                  "$ref": "#/definitions/ClientCredentials"
                },
                "flow": {
                  "type": "string",
                  "enum": [
                    "client_credentials"
                  ]
                },
                "tokenEndpoint": {
                  "type": "string"
                }
              }
            }
          ],
          "required": [
            "type"
          ],
          "properties": {
            "type": {
              "type": "string",
              "enum": [
                "oauth2"
              ]
            }
          }
        }
      ]
    },
    "BasicCredentials": {
      "type": "object",
      "required": [
        "clientId",
        "clientSecret"
      ],
      "properties": {
        "clientId": {
          "$ref": "#/definitions/secret"
        },
        "clientSecret": {
          "$ref": "#/definitions/secret"
        }
      }
    },
    "ClientCredentials": {
      "anyOf": [
        {
          "$ref": "#/definitions/JWTBearerCredentials"
        },
        {
          "$ref": "#/definitions/BasicCredentials"
        }
      ]
    },
    "ConsoleSettings": {
      "description": "Settings describing the connections with Control Plane service",
      "type": "object",
      "required": [
        "rest"
      ],
      "properties": {
        "rest": {
          "description": "Configuration related to the exposed REST APIs that are proxied",
          "allOf": [
            {
              "$ref": "#/definitions/RestProxy"
            }
          ]
        }
      }
    },
    "ControlPlaneSettings": {
      "description": "Settings describing the connections with Control Plane service",
      "type": "object",
      "required": [
        "grpc",
        "rest"
      ],
      "properties": {
        "grpc": {
          "description": "Configuration related to the inter-service communication with Control Plane service",
          "allOf": [
            {
              "$ref": "#/definitions/GrpcProxy"
            }
          ]
        },
        "rest": {
          "description": "Configuration related to the exposed REST APIs that are proxied",
          "allOf": [
            {
              "$ref": "#/definitions/RestProxy"
            }
          ]
        }
      }
    },
    "GrpcProxy": {
      "description": "Configuration regarding the GRPC services called by the BFF",
      "type": "object",
      "required": [
        "target"
      ],
      "properties": {
        "target": {
          "description": "An structured representation of a URI employed internally to contact target services",
          "type": "string"
        }
      }
    },
    "JWTBearerCredentials": {
      "type": "object",
      "required": [
        "clientId",
        "clientKeyId",
        "privateKey"
      ],
      "properties": {
        "clientId": {
          "$ref": "#/definitions/secret"
        },
        "clientKeyId": {
          "$ref": "#/definitions/secret"
        },
        "privateKey": {
          "$ref": "#/definitions/secret"
        }
      }
    },
    "MongodbPersistence": {
      "type": "object",
      "required": [
        "url"
      ],
      "properties": {
        "database": {
          "description": "Optional database name. It selects which database to be employed for storing data (it overrides the one provided in the connection string when it is set)",
          "default": null,
          "type": [
            "string",
            "null"
          ]
        },
        "url": {
          "description": "MongoDB connection string",
          "allOf": [
            {
              "$ref": "#/definitions/secret"
            }
          ]
        }
      }
    },
    "OpenLineageSettings": {
      "description": "Settings describing the connections with OpenLineage service",
      "type": "object",
      "required": [
        "grpc",
        "rest"
      ],
      "properties": {
        "grpc": {
          "description": "Configuration related to the bulk actions Data Catalog service",
          "allOf": [
            {
              "$ref": "#/definitions/GrpcProxy"
            }
          ]
        },
        "rest": {
          "description": "Configuration related to the exposed REST APIs that are proxied",
          "allOf": [
            {
              "$ref": "#/definitions/RestProxy"
            }
          ]
        }
      }
    },
    "Persistence": {
      "oneOf": [
        {
          "type": "object",
          "required": [
            "configuration",
            "type"
          ],
          "properties": {
            "configuration": {
              "$ref": "#/definitions/MongodbPersistence"
            },
            "type": {
              "type": "string",
              "enum": [
                "mongodb"
              ]
            }
          }
        }
      ]
    },
    "RestProxy": {
      "description": "Configuration regarding the Control Plane APIs that are proxied by the BFF",
      "type": "object",
      "required": [
        "target"
      ],
      "properties": {
        "auth": {
          "anyOf": [
            {
              "$ref": "#/definitions/AuthContext"
            },
            {
              "type": "null"
            }
          ]
        },
        "target": {
          "description": "An structured representation of a URI employed internally to contact target services",
          "type": "string"
        }
      }
    },
    "Settings": {
      "description": "Service specific configurations",
      "type": "object",
      "properties": {
        "apiPrefix": {
          "description": "Prefix path that it is applied to all the exposed APIs, except from the status ones",
          "default": "",
          "type": "string"
        },
        "auditUserHeader": {
          "description": "Header containing the user unique identifier",
          "type": [
            "string",
            "null"
          ]
        }
      }
    },
    "secret": {
      "oneOf": [
        {
          "type": "string"
        },
        {
          "type": "object",
          "required": [
            "key",
            "type"
          ],
          "properties": {
            "encoding": {
              "type": "string",
              "enum": [
                "base64"
              ]
            },
            "key": {
              "type": "string"
            },
            "type": {
              "const": "env"
            }
          }
        },
        {
          "type": "object",
          "required": [
            "path",
            "type"
          ],
          "properties": {
            "encoding": {
              "type": "string",
              "enum": [
                "base64"
              ]
            },
            "key": {
              "type": "string"
            },
            "path": {
              "type": "string"
            },
            "type": {
              "const": "file"
            }
          }
        }
      ]
    }
  }
}

Example

{
  "openLineage": {
    "rest": {
      "target": "http://open-lineage"
    },
    "grpc": {
      "target": "http://open-lineage:50051"
    }
  },
  "persistence": {
    "type": "mongodb",
    "configuration": {
      "url": {
        "type": "file",
        "path": "/run/secrets/data-fabric/fabric-bff.ini",
        "key": "MONGODB_URL"
      },
      "database": "data-fabric-db"
    }
  }
}

In the paragraphs below are explained the main properties of the Fabric BFF configuration file.

Open Lineage Communication

Communication between Fabric BFF and Open Lineage services occurs both via gRPC and HTTP REST requests. For this reason it is necessary to configure on the Fabric BFF the addresses where to reach Open Lineage service. This can be done by setting the properties rest and grpc of openLineage field in the Fabric BFF configuration. In both properties target field should be set to the address where Open Lineage service exposes the corresponding one.

Here can be found an example of configuration that assumes Fabric BFF and Open Lineage services are deployed within the same K8s namespace:

{
  // ...other fabric bff configurations
  "openLineage": {
    "rest": {
      "target": "<open-lineage-service-name>"                // when protocol is http, it is not necessary specifying it. When port is not specified, it is assumed the 80 is employed
    },
    "grpc": {
      "target": "http://<open-lineage-service-name>:50051"   // it is important to notice that GRPC connection use a different port from the REST target
    }
  },
  // ...other fabric bff configurations
}

Persistence Layer

info

Currently only MongoDB is supported as persistence layer for storing relevant data, such as the one related to operations auditing.

caution

The MongoDB database selected for storing Data Catalog data must be configured to have replicaSet enabled, since Fabric BFF exploits features that can be used only when a replicaSet is available.

In order to carry out all its operations, Fabric BFF requires a persistence layer where relevant information, such as auditing details, are stored. This configuration can be set under the persistence.configuration key of the configuration file. The main properties are:

• url → the connection string to your MongoDB instance;
• database → the database name where to search for the collections relevant to Fabric BFF service. Please notice that setting this property will override the database name potentially set in the connection string;

An example of persistence configuration can be seen below:

{
  // ...other fabric bff configurations
  "persistence": {
    "type": "mongodb",
    "configuration": {
      "url": "mongodb://<server>:27017/<default-database>?replicaSet=local",
      "database": "<data-fabric-database-name>"
    }
  },
  // ...other fabric bff configurations
}

tip

The following properties support secret resolution:

• persistence.configuration.url
• persistence.configuration.database

Service Settings

Additionally, the Fabric BFF service itself has a set of properties for changing its behavior. Here are listed the available ones within settings properties:

• apiPrefix → the base path applied to all the exposed routes. It defaults to /;
• auditUserHeader → specifies in which HTTP header can be found the user identifier set by the authentication system. The value of this header will be employed to correlate requests stored by the auditing system with the user that performed them. When using Mia-Platform Authentication and Authorization services this property can be set to miauserid.
  In case it is not set the auditing system does not correlate users with requests;

Here can be found a configuration example:

{
  // ...other fabric bff configurations
  "settings": {
    "apiPrefix": "/",
    "auditUserHeader": "miauserid"
  }
}

Endpoints

In the table below is provided the list of endpoints that should be defined in Console and assigned to Fabric BFF service.

Endpoint	Rewrite Base Path	Microservice	Description
/api/data-catalog	/api/data-catalog	fabric-bff	Groups all the requests related to Open Lineage operations

Routes

Under the endpoint specified above, the following routes are served by Fabric BFF

Route	Type	Method	Description
/data-catalog/bulk-actions	Websocket	*	Route for handling async operations over multiple datasets records
/data-catalog/*	REST	*	Routes prefixed with /data-catalog/ are forwarded towards Open Lineage service. For more details please read corresponding documentation