Version: 6.5.x

Guidelines for logs

The objective of these guidelines is to guarantee uniformity in the logs generated by the platform and, thus, to ensure easy usage.

Logs are generated according to Elastic Common Schema specifications.

Guidelines#

The following are guidelines, drawn from guidelines and best practices suggested by Elastic, which have to be followed when application logs are generated:

  • Field names have to be in camelCase
  • Field names don't have to contain special characters
  • Field names have to properly be used in the singular or plural form according to the content of the field itself
  • For custom fields use, where it makes sense, nesting in objects
  • Avoid using abbreviations in field names as much as possible
note

Generally, the management of these logs has been entrusted to internal libraries:

  • for the services Node.js , they will be managed by lc39 and custom-plugin;
  • for the services Go , they will be managed by glogger;
  • for the services Kotlin and Java, instead, there are no internal libraries nowadays: custom implementations must be made.

Message#

The text message passed to the log is, by default, inserted in the key msg, if only one object has been passed to the logger. Make sure that the key used for the message is the same.

Mandatory Fields#

In each log, the following fields have to be always present:

nametypedescription
reqIdstringtaken from the platform headers and necessary to trace the flow of each request;
timenumberthat signals the moment when it was generated in Unix timestamp
levelnumberthis marks the importance level of a log

Use the appropriate logging level#

It is important to always use the correct level; Each level has a value expressed in tenths. To choose which is the appropriate level, one should rely on the following criteria:

nameleveldescription
trace10to trace the operations flow of the application (do not use in production)
debug20to report information that may be useful in troubleshooting (do not use in production)
info30in cases where the service has to give information about the branch of code in execution, generally this type of log should be strictly necessary
warning40in cases of recoverable error, the service can continue to process the request
error50in cases of not recoverable error and error for which the processing of the request (but not the service) must be interrupted
fatal60in cases of unexpected and not recoverable error, and as a result of which the service must stop its execution
note

It is important to enter in logs the right information that is used to track operations, highlight problems and allow troubleshooting. It is equally important to pay close attention to what is entered in logs, avoiding, in any case, to insert in the messages or in the custom properties of logs, any private value that can lead to privacy issues.

Mandatory Logs#

Each service must necessarily generate the logs of the table below specifying the following fields in addition to the mandatory field specify above.

EventLevelField
Incoming requesttracehost, url
Request completedinfohost, http, responseTime, url
note

If the service is not REST but takes its inputs from another source, the same criterion has been applied; Use a log trace when the processing event starts, and a log info when the event ends; The final log has to contain as much useful information as possible.

Field ECS Definition#

The following are examples taken ECS documentation. None of the fields is mandatory, however it is important to use standard keys, if applicable.

Errors#

If you want to create error logs we recommend this structure When an error log has been generated, the passed object must necessarily have the key error, whose value has to be an object of the form specified by ECS for the Error Fields, which are shown in the table below:

FieldDescriptionLevel
error.codeError code describing the error. type: keywordcore
error.idUnique identifier for the error. type: keywordcore
error.messageError message. type: textcore
error.stackTraceThe stack trace of this error in plain text. type: keyword Multi-fields: error.stackTrace.text (type: text)extended
error.typeThe type of the error, for example the class name of the exception. type: keyword example: java.lang.NullPointerExceptionextended

Host#

FieldDescriptionLevel
host.hostnameHostname of the host. It normally contains what the hostname command returns on the host machine. type: keywordcore
host.nameName of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. type: keywordcore
host.uptimeSeconds the host has been up. type: long example: 1325extended

HTTP#

FieldDescriptionLevel
http.request.body.bytesSize in bytes of the request body. type: long example: 887extended
http.request.bytesTotal size in bytes of the request (body and headers). type: long example: 1437extended
http.request.methodHTTP request method. The field value must be normalized to lowercase for querying. See the documentation section "Implementing ECS". type: keyword example: get, post, putextended
http.request.referrerReferrer for this HTTP request. type: keyword example: https://blog.example.com/extended
http.request.userAgent.device.nameName of the device. type: keyword example: iPhoneextended
http.request.userAgent.nameName of the user agent. type: keyword example: Safariextended
http.request.userAgent.originalUnparsed user_agent string. type: keyword Multi-fields: http.request.userAgent.original.text (type: text) example: Mozilla/5.0 (iPhone; CPU iPhone OS 12_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1extended
http.request.userAgent.versionVersion of the user agent. type: keyword example: 12.0.extended
http.response.body.bytesSize in bytes of the response body. type: long example: 887extended
http.response.bytesTotal size in bytes of the response (body and headers). type: long example: 1437extended
http.response.status_codeHTTP response status code. type: long example: 404extended
http.versionHTTP version. type: keyword example: 1extended

URL#

FieldDescriptionLevel
url.domainDomain of the url, such as www.elastic.com. In some cases a URL may refer to an IP and/or port directly, without a domain name. In this case, the IP address would go to the domain field. type: keyword example: www.elastic.coextended
url.fragmentPortion of the url after the #, such as "top". The # is not part of the fragment. type: keywordextended
url.fullIf full URLs are important to your use case, they should be stored in url.full, whether this field is reconstructed or present in the event source. type: keyword Multi-fields: url.full.text (type: text). example: https://www.elastic.co:443/search?q=elasticsearch#topextended
url.pathPath of the request, such as "/search". type: keywordextended
url.portPort of the request, such as 443. type: long example: 443extended
url.queryThe query field describes the query string of the request, such as "q=elasticsearch". The ? is excluded from the query string. If a URL contains no ?, there is no query field. If there is a ? but no query, the query field exists with an empty string. The exists query can be used to differentiate between the two cases. type: keywordextended
url.schemeScheme of the request, such as "https". Note: The : is not part of the scheme. type: keyword example: https.extended