Skip to main content
Version: 13.x (Current)

Fabric BFF

Fabric BFF, as suggested by the name, works as Backend For Frontend for Data Catalog UI and it mediates requests between frontend and the following services:

Moreover, the BFF offers API to manage Data Catalog Connections from the dedicated UI section.

Configuration​

Configuration of Fabric BFF is a straightforward process that involves setting up a ConfigMap and specifying essential environment variables.

Environment Variables​

Fabric BFF can be customized using the following environment variables:

NameRequiredDescriptionDefault Value
HTTP_PORT-This variable determines the TCP port where the HTTP controller binds its listener3000
LOG_LEVEL-Specify the centralized application log level, choosing from options such as debug, error, info, trace or warninfo
BFF_CONFIGURATION_FILEPATH-Set the location of the configuration file~/.fd/data-fabric-bff/config.json
OTEL_EXPORTER_OTLP_ENDPOINT-The URL to a GRPC endpoint of an OpenTelemetry Collector.
Specifying this value enables the support for OpenTelemetry tracing

Config Map​

The configuration of the service is handled by a JSON file whose location is defined by the BFF_CONFIGURATION_FILEPATH.
When instantiating Data Catalog application, Fabric BFF service configuration is generated with a dedicated Config Map, named fabric-bff-config. This file contains a template configuration that should help in configuring the service.

Loading ....

In the paragraphs below are explained the main properties of the Fabric BFF configuration file.

Open Lineage Communication​

Communication between Fabric BFF and Open Lineage service occurs both via gRPC and HTTP REST requests. For this reason it is necessary to configure on the Fabric BFF the addresses where to reach Open Lineage service. This can be done by setting the properties rest and grpc of openLineage field in the Fabric BFF configuration. In both properties target field should be set to the address where Open Lineage service exposes the corresponding one.

Here can be found an example of configuration that assumes Fabric BFF and Open Lineage service are deployed within the same K8s namespace:

{
  // ...other fabric bff configurations
  "openLineage": {
    "rest": {
      "target": "<open-lineage-service-name>"                // when protocol is http, it is not necessary specifying it. When port is not specified, it is assumed the 80 is employed
    },
    "grpc": {
      "target": "http://<open-lineage-service-name>:50051"   // it is important to notice that GRPC connection use a different port from the REST target
    }
  },
  // ...other fabric bff configurations
}

Job Runner Communication​

Communication between Fabric BFF and Job Runner service occurs via gRPC. For this reason it is necessary to configure on the Fabric BFF the addresses where to reach Job Runner service. This can be done by setting the property grpc of jobRunner field in the Fabric BFF configuration. The target field should be set to the address where Job Runner service exposes the corresponding one.

Here can be found an example of configuration that assumes Fabric BFF and Job Runner service are deployed within the same K8s namespace:

{
  // ...other fabric bff configurations
  "jobRunner": {
    "grpc": {
      "target": "http://<job-runner-service-name>:50051"
    }
  },
  // ...other fabric bff configurations
}

Console Communication​

In order for Data Catalog UI to know which environments can be linked to Mia Platform CRUD connections, the service needs to contact Mia-Platform Console and retrieve the list of Projects that should be accessible from this Data Catalog instance.

To achieve so, it is first necessary that your Company Owner creates a dedicated Service Account on your Mia-Platform Console instance and assign to it the proper permissions for listing the Console projects of interest.

Good practices in permissions assignment

Pay attention to the level of access to the resources that you assign to the Service Account. For Control Plane use case, a good practice may be to assign the role of guest at Company level while granting the reporter role to all the projects that should be visible by Control Plane.
About permissions assignment, it is possible to go even more granular in case you want to allow visibility only to a subset of runtime environments of a specific project. In fact, to do that, you may opt to assign the role of guest even at Project level while granting the reporter role solely to those runtime environments that should be visible by Control Plane.

Once the service account has been registered, your Company Owner needs to hand over to you its credentials, which are:

  • client-id → unique identifier generated for this the service account
  • [Client Secret Basic] client-secret → a string representing the secret needed by the service account to authenticate
  • [Private Key JWT] client-key-id → unique identifier of the key employed for generating for this the service account
  • [Private Key JWT] private-key → private key in PEM format

These details then should be inserted in your Fabric BFF service configuration under the console property. In particular, the fields to be set are:

  • target → specifies the base URL of your Mia-Platform Console instance
  • auth → defines how the Fabric BFF service should authenticate on Mia-Platform Console APIs, that is the service account credentials
caution

It is responsibility of your Company Owner to ensure that service account credentials are properly processed according to your company security policies.

Furthermore, it is of extreme importance understanding that any Control Plane user will be able to list the project name and available environments of all the projects that can be accessed by the service account configured on Fabric BFF.

This is and example of console property configuration:

{
  "console": {
    "target": "https://<your-mia-platform-console-url>",
    "auth": {
      "type": "oauth2",
      "flow": "client_credentials",
      "tokenEndpoint": "/api/m2m/oauth/token",
      "credentials": {
        "clientId": "<service-account-client-id>",
        "clientSecret": "<service-account-client-secret>"
      }
    }
  }
  // ...other fabric bff configurations
}
tip

The following properties support secret resolution:

  • console.target
  • console.auth.credentials.clientId
  • console.auth.credentials.clientKeyId
  • console.auth.credentials.privateKey

A custom x509 certificate can be added to the default root keychain of certificates for any client/reversed-proxy reached by Fabric BFF. Custom certificate must be mounted on local file system of Fabric BFF and referenced in the configuration at 'settings.tls.certificate' as a secret.

Persistence Layer​

info

Currently only MongoDB is supported as persistence layer for storing relevant data, such as the one related to operations auditing.

caution

The MongoDB database selected for storing Data Catalog data must be configured to have replicaSet enabled, since Fabric BFF exploits features that can be used only when a replicaSet is available.

In order to carry out all its operations, Fabric BFF requires a persistence layer where relevant information, such as auditing details, are stored. This configuration can be set under the persistence.configuration key of the configuration file. The main properties are:

  • url → the connection string to your MongoDB instance;
  • database → the database name where to search for the collections relevant to Fabric BFF service. Please notice that setting this property will override the database name potentially set in the connection string;

An example of persistence configuration can be seen below:

{
  // ...other fabric bff configurations
  "persistence": {
    "type": "mongodb",
    "configuration": {
      "url": "mongodb://<server>:27017/<default-database>?replicaSet=local",
      "database": "<data-fabric-database-name>"
    }
  },
  // ...other fabric bff configurations
}
tip

The following properties support secret resolution:

  • persistence.configuration.url
  • persistence.configuration.database

Service Settings​

Additionally, the Fabric BFF service itself has a set of properties for changing its behavior. Here are listed the available ones within settings properties:

  • apiPrefix → the base path applied to all the exposed routes. It defaults to /;
  • auditUserHeader → specifies in which HTTP header can be found the user identifier set by the authentication system. The value of this header will be employed to correlate requests stored by the auditing system with the user that performed them. When using Mia-Platform Authentication and Authorization services this property can be set to miauserid.
    In case it is not set the auditing system does not correlate users with requests;

Here can be found a configuration example:

{
  // ...other fabric bff configurations
  "settings": {
    "apiPrefix": "/",
    "auditUserHeader": "miauserid"
  }
}

Endpoints​

In the table below is provided the list of endpoints that should be defined in Console and assigned to Fabric BFF service.

EndpointRewrite Base PathMicroserviceDescription
/api/data-catalog/api/data-catalogfabric-bffGroups all the requests related to
Data Catalog operations. For more details please refer to the corresponding documentation
/api/open-lineage/api/open-lineagefabric-bffGroups all the requests related to
Open Lineage operations. For more details please refer to the corresponding documentation
/api/connections/api/connectionsfabric-bffGroups all the requests related to
Connections operations
/api/job-runner/api/job-runnerfabric-bffGroups all the requests related to
Job Runner operations

Routes​

Here are described which routes Fabric BFF service serves.

Connection routes​

The following routes are exposed over the /api/connections endpoint.

RouteTypeMethodDescription
/connections/itemsRESTGETRoute for reading the list of connections configured for the data catalog application.
/connections/itemsRESTPOSTRoute for creating a connection for the data catalog application.
/connections/items/:idRESTGETRoute for reading a given connection configured for the data catalog application.
/connections/items/:idRESTGETRoute for updating a given connection configured for the data catalog application.
/connections/items/:idRESTDELETERoute for deleting a given connection configured for the data catalog application.

Job Runner routes​

The following routes are exposed over the /api/job-runner endpoint and are forwarded towards the corresponding gRPC method..

RouteTypeMethodDescription
/feedbackWebsocketGETRoute for receiving feedback messages from the Status method of the Job Runner gRPC service.
/job-runner/*REST*Routes prefixed with /job-runner are converted into gRPC requests towards Job Runner service.
For more details please read corresponding documentation
/agent/driversRESTGETRoute for invoking the ListDrivers method of the ODBC Client gRPC service.
/agent/dsnRESTGETRoute for invoking the ListDataSources method of the ODBC Client gRPC service.
/agent/syncRESTPOSTRoute for invoking the Run method of the Job Runner gRPC service.
/agent/abortRESTPOSTRoute for invoking the Abort method of the Job Runner gRPC service.
/agent/dsnRESTGETRoute for invoking the ListSecretNames method of the Configuration gRPC service.

Migration Guides​

Here are the migration steps to perform between each version of the service.

From 0.1.x to 0.2.0​

The following steps needs to be followed: