Unmapped Fields

This dashboard will help you to monitor unmapped fields. With unmapped field we mean a field which is in a Kafka message, but it is not included in the projection definition or it can not be casted to the desired field type.

info

This feature is available since Realtime Updater v4.4.0

This situation could happen for two main reasons:

• Inside the console, the field was not registered
• The Cast Function launches an error

In order to use this dashboard, the following configuration steps are needed:

• Create a MongoDB collection named unmapped_fields
• Set the environment variable COLLECTION_NAME_UNMAPPED_FIELDS_STORAGE equal to unmapped_fields in any Real Time Updater service you want to keep track of the unmapped fields
• Register the collection on CMS
• Add the Fast Data Monitoring Application (if you don't have it already) to add to the project the MongoDB Query Exporter service and configure it accordingly
• Configure MongoDB Query Exporter
• Create a Grafana Dashboard to read the metrics

The collection must have the following fields:

[
    {
        "name": "type",
        "type": "string",
        "required": true,
        "nullable": false,
        "sensitivityValue": 0,
        "encryptionEnabled": false,
        "encryptionSearchable": false
    },
    {
        "name": "projection",
        "type": "string",
        "required": true,
        "nullable": false,
        "sensitivityValue": 0,
        "encryptionEnabled": false,
        "encryptionSearchable": false
    },
    {
        "name": "field",
        "type": "string",
        "required": true,
        "nullable": false,
        "sensitivityValue": 0,
        "encryptionEnabled": false,
        "encryptionSearchable": false
    },
    {
        "name": "lastError",
        "type": "RawObject",
        "required": false,
        "nullable": false,
        "sensitivityValue": 0,
        "encryptionEnabled": false,
        "encryptionSearchable": false
    },
    {
        "name": "__internal__kafkaInfo",
        "type": "RawObject",
        "required": true,
        "nullable": false,
        "sensitivityValue": 0,
        "encryptionEnabled": false,
        "encryptionSearchable": false
    }
]

Where:

• Type: could be UNCASTABLE_FIELD or UNKNOWN_FIELD
• lastError: it is the last casting error recorded. It exists only if type is equal to UNCASTABLE_FIELD. Moreover, the error contains, two more information:
  • value: The value of the field which causes the error
  • message: The error message
• projection: The name of the field's projection
• field: the name of the field
• _internal_kafkaInfo: the Kafka information generated by the document
  • offset: offset of the Kafka message
  • partition: the partition which contain the Kafka message
  • timestamp: the message timestamp
  • key: message's key

The configmap for MongoDB Query Exporter:

version: 2.0
bind: 0.0.0.0:3000
metricsPath: /-/metrics
log:
      encoding: json
      level: info
      development: false
      disableCaller: false
global:
      queryTimeout: 10
      maxConnection: 3
      defaultCache: 5
servers:
    - name: main
      uri:  {{MONGODB_SHORT_URL}}
metrics:
      - name: unknown_fields
        type: counter #Can also be empty, the default is gauge
        servers: [main] #Can also be empty, if empty the metric will be used for every server defined
        help: 'Count the number of Unknown fields'
        value: total
        labels: []
        mode: pull
        cache: 0
        constLabels:
            collection: unmapped-fields
        database: {{FAST_DATA_PROJECTIONS_DATABASE_NAME}}
        collection: unmapped-fields
        pipeline: |
          [
            {
                "$match": {
                    "type": "UNKNOWN_FIELD",
                    "__STATE__":  "PUBLIC"
                }
            }, {
                "$count": "total"
            }
          ]
      - name: uncastable_fields
        type: counter #Can also be empty, the default is gauge
        servers: [main] #Can also be empty, if empty the metric will be used for every server defined
        help: 'Count the number of Uncastable fields'
        value: total
        labels: []
        mode: pull
        cache: 0
        constLabels:
            collection: unmapped-fields
        database: {{FAST_DATA_PROJECTIONS_DATABASE_NAME}}
        collection: unmapped-fields
        pipeline: |
          [
            {
                "$match": {
                    "type": "UNCASTABLE_FIELD",
                    "__STATE__":  "PUBLIC"
                }
            }, {
                "$count": "total"
            }
          ]

Setting Alarms

An alarms system could be useful to tempestively recover these kinds of errors. The alarm system is developed to detect new unmapped fields. To define it you need to add inside the configurations folder of your project repository the following file:

fast-data.prometheusrule.yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: 'unmapped-fields'
spec:
  groups:
  - name: 'fast-data.unmapped-fields.error.rules'
    rules:
    - alert: UncastableFields
      annotations:
        summary: 'Unable to cast a field'
        message: 'In namespace {{ $labels.namespace }} an error occurs while performing a cast'
      expr: '
        delta(uncastable_fields{collection="unmapped-fields", namespace="YOUR_NAMESPACE"}[2m]) > 1'
      labels:
        severity: 'warn' # debug|info|warn|critical
        # here other tags can be applied to the alert and associated Prometheus ALERTS metric

This system will produce an alarm every time the following condition is true: delta(uncastable_fields{collection="unmapped-fields", namespace="YOUR_NAMESPACE"}[2m]) > 1. Where YOUR_NAMESPACE is the name of your namespace. The following condition will produce an alarm every time an error UNCASTABLE_FIELD is detected in an interval of two minutes.

Panels

The Dashboard contains only one panel, named Unmapped Unknown Fields and depicted in sample screenshot, that includes a single number with value equal or greater than 1 if there are new unmapped or unknown fields.

FAQ

How can I manage unmapped fields?

It is possible to manage the unmapped fields:

• If you would like to neglect the unmapped field, go inside the CMS and move the document in DRAFT/TRASH
• If you would register the unmapped field, go inside the console, and he has to register it

What happens if I register a field which previously was the source of an UNKNOWN_FIELD or UNCASTABLE_FIELD message?

The document related to an UNKNOWN_FIELD field will be removed from the collection, instead, document connected to an UNKNOWN_FIELD with type UNCASTABLE_FIELD will be retained.