Database Preparation
Currently only MongoDB is supported as underlying database for storing the application data.
It is highly recommended to employ a dedicated database / namespace to store Fast Data Runtime Management data. This allows restricting which applications have access to the management collections and keeping the necessary collections all under the same location without additional clutter from other projects.
Fast Data Runtime Management solution relies on a set of collections to carry out most of its tasks, that is keeping track of runtime views or updating the desired Fast Data Pipelines state. In order to provide a performant and reliable system, the database must be configured accordingly to support application execution.
This means that required collections must be created, necessary indexes added and additional schema validation set, so that only valid data can be read and written to those collections.
To reach this objective in a straightforward manner we provide an administration command-line interface, which takes care of creating all the necessary resources on selected database. This CLI is offered within as a Docker image, which can be run within your namespace every time a new application version is released.
CLI Configuration
Running the CLI inside your project can be done leveraging a K8s cronjob. In this documentation page it is described how it can be configured, while here are highlighted which values within the K8s manifest need to be updated:
<URL_TO_CONTAINER_REGISTRY>
→ is the url to your Container Registry of reference;data-fabric/fabric-admin
→ image is employed, which should be found within your Container Registry;suspend
→ the cronjob must be released as suspended, since it should be launched manually only once;<NAME_CONTAINER_REGISTRY_SECRET>
→ name of the secret where are contained the credentials for connecting to your Container Registry;MONGODB_ADMIN_URL
→ connection string to your MongoDB database.
Please, beware that the credentials assigned to MongoDB connection string must have the admin permission
to modify collections (collMod
), so that the CLI can add a schema validation on the collection. This connection string
might be different from the one employed within Fast Data Runtime Management application, where only reading and writing is sufficient;
Here, is provided an example of cronjob definition, with values listed above as placeholders for the real ones:
apiVersion: batch/v1
kind: CronJob
metadata:
name: fabric-control-plane-admin
labels:
app: fabric-control-plane-admin
# ...
# ...
spec:
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 1
schedule: '* * * * *'
# this job MUST be deployed as suspended, since it just run once
suspend: true
jobTemplate:
spec:
backoffLimit: 1
template:
metadata:
name: fabric-control-plane-admin
labels:
app: fabric-control-plane-admin
# ...
# ...
spec:
imagePullSecrets:
- name: <NAME_CONTAINER_REGISTRY_SECRET>
containers:
- name: fabric-control-plane-admin
# current latest stable version of the image is 0.1.0
image: <URL_TO_CONTAINER_REGISTRY>/data-fabric/fabric-admin:0.1.0
args: [
"all",
"init",
"--url",
"{{MONGODB_ADMIN_URL}}",
]
restartPolicy: Never
This cronjob can then be manually triggered as described here
every time a new version of data-fabric/fabric-admin
image is available, which it should roughly correspond with a new release of
Fast Data Runtime Management application.
In case in your Console Project it is possible to enable Custom Resources, then instead of
creating a cronjob
manifest it would be possible to define a job
one, which does start as soon as released and does not need to be
manually launched.
Cronjob is an alternative better suited in case manually launching the job for administering your database is preferred or enforced by any internal policy (e.g. you need to be granted the permission to edit your database).