Guidelines for logs
The objective of these guidelines is to guarantee uniformity between the logs from all services that run in a single project, regardless of whether they are custom or core.
Log Parsers
When configuring a custom service you can select a Log Parser. Each log parser supports a specific logging format, but they all share the same purpose: allowing log collection and processing from your Kubernetes cluster to the ELK Stack.
Mia-Platform Console provides a series of log parsers that can be selected from the Console and will be used to properly parse and process your service logs:
mia-json: processes all logs as JSON, parsing all the keys to store them;mia-nginx: a custom Nginx parser;mia-plain: a plain text parser;- Not Collected: prevents all logs from the Kubernetes Pod to be collected and processed.
JSON Logging format
When a service uses the label with value mia-json, all of its key/values are collected and stored in Elastic. However, if you want to perform queries on the logs using Kibana, you'll need to create indexes.
A set of preconfigured indexes come with each Console project. These indexes run on a specific set of keys. The following document provides a list of available log keys (following the Elastic Common Schema specifications) that can be used. Some of these log keys are indexed.
Each project has a set of preconfigured indexes. If you need custom indexes, please refer to your Mia-Platform representative.
Here you can find some guidelines, drawn from guidelines and best practices suggested by Elastic, which should be followed when application logs are generated:
- Field names must be in camelCase;
- Field names must not contain special characters;
- Field names should be singular or plural according to the content of the field;
- Whenever it makes sense, prefer nesting your fields in structured objects;
- Avoid using abbreviations in field names.
When creating a new server, consider offload the setup of the logging facilities to Mia-Platform service libraries:
- For Node.js services, you can use lc39 and custom-plugin-lib;
- For Go services, you can use glogger.
Use the appropriate logging level
It is important to always use the correct log level. Each level has a value expressed in tenths: the higher is the number, the higher the severity of the log. To choose which is the appropriate level, you should rely on the following criteria:
| Name | Severity | Description |
|---|---|---|
trace | 10 | Trace the operations flow of the application (do not use in production). |
debug | 20 | Report information that may be useful in troubleshooting (do not use in production). |
info | 30 | In cases where the service has to give information about the branch of code in execution. |
warning | 40 | In cases of recoverable error, the service can continue to process the request. |
error | 50 | In cases of not recoverable error and error for which the processing of the request (but not the service) must be interrupted. |
fatal | 60 | In cases of unexpected and not recoverable error, and as a result of which the service must stop its execution. |
It is important to put in logs the right information that can be used to track operations, highlight problems, and allow troubleshooting.
Avoid, under any circumstance, to insert any private or sensitive value that can lead to privacy-related issues or incidents.
Mandatory Fields
In each log, the following fields have to be always present:
| Name | Type | Indexed | Description |
|---|---|---|---|
msg | String | Indexed | The text message specific for each log. |
reqId | String | Indexed | Traces the flow of each request. It is taken from the platform headers |
time | Number | Indexed | Marks in Unix timestamp the moment when the log was generated. |
level | Number | Indexed | Identifies the severity level of the log. |
The reqId field is extremely useful to trace operations following a single request, and is considered mandatory since usually services expose APIs.
However, if the concept of request tracing is not applicable, you can ignore it.
Mandatory Logs
Each service exposing APIs must necessarily generate the logs shown in the table below, specifying the following fields in addition to the mandatory field specified above.
| Event | Level | Indexed | Field |
|---|---|---|---|
| Incoming request | trace | Indexed | host, url |
| Request completed | info | Indexed | host, http, responseTime, url |
If the service is not REST but takes its inputs from another source, the same criteria have to be applied.
Use a trace log when the event processing starts, and a info log when the event ends.
The final log should contain as much useful information as possible.
ECS Field Definition
The following are examples taken from ECS documentation. None of the fields are mandatory, however it is important to use standard keys, whenever applicable.
Errors
If you want to create error logs, we recommend the following structure.
When an error log has been generated, the passed object must necessarily have the key error,
whose value has to be an object of the form specified by ECS for the Error Fields,
which are shown in the table below:
| Field | Type | Level | Indexed | Description |
|---|---|---|---|---|
error.code | keyword | Core | Indexed | Error code describing the error. |
error.id | keyword | Core | Indexed | Unique identifier for the error. |
error.message | text | Core | Indexed | Error message. |
error.stackTrace | keyword | Extended | Indexed | The stack trace of the error in plain text. |
error.stackTrace.text | text | Extended | Indexed | Multi-fields stack trace of the error in plain text. |
error.type | keyword | Extended | Indexed | The type of the error, for example the class name of the exception. Example: java.lang.NullPointerException |
Host
| Field | Type | Level | Indexed | Description |
|---|---|---|---|---|
host.hostname | keyword | Core | Indexed | Hostname of the host. It normally contains what the hostname command returns on the host machine. |
host.name | keyword | Core | Indexed | Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
host.uptime | long | Extended | Indexed | Seconds the host has been up. Example: 1325 |
HTTP
| Field | Type | Level | Indexed | Description |
|---|---|---|---|---|
http.request.body.bytes | long | Extended | Indexed | Size in bytes of the request body. Example: 887 |
http.request.bytes | long | Extended | Indexed | Total size in bytes of the request (body and headers). Example: 1437 |
http.request.method | keyword | Extended | Indexed | HTTP request method. The field value must be normalized to lowercase for querying. See the documentation section "Implementing ECS". Example: get, post, put |
http.request.referrer | keyword | Extended | Indexed | Referrer for this HTTP request. Example: https://blog.example.com/ |
http.request.userAgent.device.name | keyword | Extended | Not indexed | Name of the device. Example: iPhone |
http.request.userAgent.name | keyword | Extended | Not indexed | Name of the user agent. Example: Safari |
http.request.userAgent.original | keyword | Extended | Indexed | Unparsed user_agent string. example: Mozilla/5.0 (iPhone; CPU iPhone OS 12_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1 |
http.request.userAgent.original.text | text | Extended | Indexed | Multi-fields unparsed user_agent string. |
http.request.userAgent.version | keyword | Extended | Not indexed | Version of the user agent. example: 12.0. |
http.response.body.bytes | long | Extended | Indexed | Size in bytes of the response body. example: 887 |
http.response.bytes | long | Extended | Indexed | Total size in bytes of the response (body and headers). example: 1437 |
http.response.status_code | long | Extended | Indexed | HTTP response status code. example: 404 |
http.version | keyword | Extended | Not indexed | HTTP version. example: 1 |
URL
| Field | Type | Level | Indexed | Description |
|---|---|---|---|---|
url.domain | keyword | Extended | Indexed | Domain of the URL, such as www.mia-platform.eu. In some cases a URL may refer to an IP and/or port directly, without a domain name. In this case, the IP address would go to the domain field. Example: www.mia-platform.eu |
url.fragment | keyword | Extended | Indexed | Portion of the URL after the #, such as "top". The # is not part of the fragment. |
url.full | keyword | Extended | Indexed | If full URLs are important to your use case, they should be stored in url.full, whether this field is reconstructed or present in the event source. |
url.full.text | text | Extended | Indexed | Multi-fields. Example: https://www.elastic.co:443/search?q=elasticsearch#top |
url.path | keyword | Extended | Indexed | Path of the request, such as /search. |
url.port | long | Extended | Indexed | Port of the request, such as 443. Example: 443 |
url.query | keyword | Extended | Indexed | The query field describes the query string of the request, such as "q=elasticsearch". The ? is excluded from the query string. If a URL contains no ?, there is no query field. If there is a ? but no query, the query field exists with an empty string. The exists query can be used to differentiate between the two cases. |
url.scheme | keyword | Extended | Indexed | Scheme of the request, such as "https". Note: The : is not part of the scheme. Example: https. |