How to setup a Pipeline
Projects created with the Mia-Platform Console can be deployed with simple script and tooling for applying standard Kubernetes yaml files that describe the various resources. To aid the developer we have created a new open source tool called mlp.
In this chapter we will highlight how to use mlp to deploy the project and give you all the information if you want to use another tool (e.g. kubectl).
First Phase: Environment Preparation
We have to prepare the environment for the deploy of the resources. For preparing the environment we intend to setup
correctly various environment variables needed in the next phases.
These variables are the public ones generated by the console and some others needed for the metadata set inside the
various yaml file.
Here are the environment variables that we setup inside the pipelines that are used in our PaaS:
- AUTHOR_EMAIL: an email of the user that is running the deploy
- COMMIT_SHA: set as the commit sha of the commit to deploy
- STAGE_TO_DEPLOY: set to the current environment of the project to deploy
- RELEASE_DATE: current date generated with the unix command
date -I'seconds' -u
Additionally, to these variables, the Console will generate a .env
file for every environment of the project that can
be usually found in the path configuration/variables/<environment-name>.env
so we simply source it in the current
context with the command:
export VARIABLES_FILE="configuration/variables/<environment-name>.env"
test -f "${VARIABLES_FILE}" && set -a && source "${VARIABLES_FILE}"
For compatibility reasons we always check first if the file actually exists before sourcing it.
Additionally, we have inside the shell environment the variables set inside the Environment Variables table of the
Project Overview area of the Console.
After setting up the shell environment with all the variables, we can start to setup the files for being ready to be deployed inside the target environment.
Second Phase: Generation
This phase is targeted principally to the mlp
tool and is a function added to dynamically create
configmaps and secrets using variables and files set inside the pipeline and referenced by name inside the Design
section of the Console. mlp
can read its configuration from a file, typically created in the root of
the project and called mlp.yaml
for generating Kubernetes secrets and configmaps. For the syntax of the file
you can read the relative documentation page.
To generate the files we need to launch the following command:
export GENERATE_FILE="./mlp.yaml"
test -f "${GENERATE_FILE}" && mlp generate -c "${GENERATE_FILE}" -e "<environment-prefix>" -o "<target-folder>"
As you can see we check if the file is present before launching the command because this step can be skipped if not
necessary. The <environment-prefix>
is a prefix to the name of the environment variables that indicate if a more
specific environment variables can be used instead. You can read more on this subject in the relative page on this site.
For example we can have a variables named SERVICE_URL
referenced inside
the yaml files, and a variable named DEV_SERVICE_URL
that contains a variation of the values needed only for a specific
environment. In this case we can pass as an argument -e "DEV_"
to the cli and the values inside variables prefixed with
DEV_
will be considered as high priority. You can use the flag multiple times to add multiple prefixes with a scalar
priority, the first flag as an higher priority to the last one.
The generated files is outputted inside the <target-folder>
, that will be used in the last phase and can be
any empty folder you want.
Third Phase: Interpolation
The third phase will effectively substitute all the environment placeholders inside the generated yaml files with the environments present in the context.
mlp interpolate -e "<environment-prefix>" -f "configuration/<environment-name>" -f "configuration" -o "<target-folder>"
Even with this command we can pass one or multiple environment prefix that is used and follow the same logic explained
in the previous phase. We use two different path to get and interpolate all the files present inside the base folder,
usually called configuration
, and the folder specific to the environment target, usually placed as subfolder
of configuration
.
The interpolated files than will be outputted inside a <target-folder>
of your choice that is the same one used
in the previous command.
Fourth Phase: Deploy
The last phase of the pipeline is to apply the files to the target cluster with the following command:
mlp deploy --server "<k8s-api-server-url>" --certificate-authority "<path-to-cluster-ca>" \
--token "<token-for-deploying>" --deploy-type "${DEPLOY_TYPE}" \
--force-deploy-when-no-semver=${FORCE_DEPLOY_WHEN_NO_SEMVER} -f <target-folder> -n "${KUBE_NAMESPACE}"
As you can see there are three variables that are not defined yet, because are set by the console in the pipeline trigger. These variabiles contains:
- DEPLOY_TYPE: the type of deploy to use with
mlp
, the values supported aredeploy_all
andsmart_deploy
- FORCE_DEPLOY_WHEN_NO_SEMVER: boolean that will force the redeploy of deployments with tags that don't follow the SemVer convention
- KUBE_NAMESPACE: contains the namespace target where to deploy all the files
Additionally, mlp
supports the flag --ensure-namespace=false
to skip the namespace creation if not already present in the cluster. With this flag, if the namespace is not in already present mlp
will throw an error and the deploy step fails.
During this command we will also change the pods annotations with references to configmaps or secrets mounted to the
deployment for forcing a redeploy if the content has changed between deployment.
mlp
also has a strict ordering for deploying certain resources for ensuring the correct update has taken place
before deploying the main workloads. The order can be seen in this file.