Skip to main content
Version: 8.9.x

Unmapped Fields

This dashboard will help you to monitor unmapped fields. With unmapped field we mean a field which is in a Kafka message, but it is not mapped on a projection.

info

This feature is available since Realtime Updater v4.4.0

This situation could happen for two main reasons:

  • Inside the console, the field was not registered
  • The Cast Function launches an error

In order to use this dashboard, the following configuration steps are needed:

  • Create a collection unmapped_fields
  • Set the environment variable COLLECTION_NAME_UNMAPPED_FIELDS_STORAGE equal to unmapped_fields
  • Registering the collection on CMS
  • Add to the project the MongoDB Query Exporter service (already existing if Fast Data Monitoring application is used)
  • Configure MongoDB Query Exporter
  • Create a Grafana Dashboard to read the metrics

The collection must have the following fields:

[
{
"name": "type",
"type": "string",
"required": true,
"nullable": false,
"sensitivityValue": 0,
"encryptionEnabled": false,
"encryptionSearchable": false
},
{
"name": "projection",
"type": "string",
"required": true,
"nullable": false,
"sensitivityValue": 0,
"encryptionEnabled": false,
"encryptionSearchable": false
},
{
"name": "field",
"type": "string",
"required": true,
"nullable": false,
"sensitivityValue": 0,
"encryptionEnabled": false,
"encryptionSearchable": false
},
{
"name": "lastError",
"type": "RawObject",
"required": false,
"nullable": false,
"sensitivityValue": 0,
"encryptionEnabled": false,
"encryptionSearchable": false
},
{
"name": "__internal__kafkaInfo",
"type": "RawObject",
"required": true,
"nullable": false,
"sensitivityValue": 0,
"encryptionEnabled": false,
"encryptionSearchable": false
}
]

Where:

  • Type: could be UNCASTABLE_FIELD or UNKNOWN_FIELD
  • lastError: it is the last casting error recorded. It exist only if type is equal to UNCASTABLE_FIELD. Moreover, the error contains, two more information:
    • value: The value of the field which causes the error
    • message: The error message
  • projection: The name of the field's projection
  • field: the name of the field
  • _internal_kafkaInfo: the Kafka information generated by the document
    • offset: offset of the Kafka message
    • partition: the partition which contain the Kafka message
    • timestamp: the message timestamp
    • key: message's key

The configmap for MongoDB Query Exporter:

version: 2.0
bind: 0.0.0.0:3000
metricsPath: /-/metrics
log:
encoding: json
level: info
development: false
disableCaller: false
global:
queryTimeout: 10
maxConnection: 3
defaultCache: 5
servers:
- name: main
uri: {{MONGODB_SHORT_URL}}
metrics:
- name: unknown_fields
type: counter #Can also be empty, the default is gauge
servers: [main] #Can also be empty, if empty the metric will be used for every server defined
help: 'Count the number of Unknown fields'
value: total
labels: []
mode: pull
cache: 0
constLabels:
collection: unmapped-fields
database: {{FAST_DATA_PROJECTIONS_DATABASE_NAME}}
collection: unmapped-fields
pipeline: |
[
{
"$match": {
"type": "UNKNOWN_FIELD",
"__STATE__": "PUBLIC"
}
}, {
"$count": "total"
}
]
- name: uncastable_fields
type: counter #Can also be empty, the default is gauge
servers: [main] #Can also be empty, if empty the metric will be used for every server defined
help: 'Count the number of Uncastable fields'
value: total
labels: []
mode: pull
cache: 0
constLabels:
collection: unmapped-fields
database: {{FAST_DATA_PROJECTIONS_DATABASE_NAME}}
collection: unmapped-fields
pipeline: |
[
{
"$match": {
"type": "UNCASTABLE_FIELD",
"__STATE__": "PUBLIC"
}
}, {
"$count": "total"
}
]

Setting Alarms

An alarms system could be useful to tempestively recover these kind of errors. The alarms system is developed to detect new unmapped fields. To define it you need to add inside the configurations folder of your project repository the following file:

fast-data.prometheusrule.yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: 'unmapped-fields'
spec:
groups:
- name: 'fast-data.unmapped-fields.error.rules'
rules:
- alert: UncastableFields
annotations:
summary: 'Unable to cast a field'
message: 'In namespace {{ $labels.namespace }} an error occurs while performing a cast'
expr: '
delta(uncastable_fields{collection="unmapped-fields", namespace="YOUR_NAMESPACE"}[2m]) > 1'
labels:
severity: 'warn' # debug|info|warn|critical
# here other tags can be applied to the alert and associated Prometheus ALERTS metric

This system will produce an alarm every time the following condition is true: delta(uncastable_fields{collection="unmapped-fields", namespace="YOUR_NAMESPACE"}[2m]) > 1. Where YOUR_NAMESPACE is the name of your namespace. The following condition will produce an alarm every time an error UNCASTABLE_FIELD is detected in an interval of two minutes.

Panels

The Dashboard contains the following panels, first depicted in sample screenshots, and later described in detail in the table below.

unmapped field dashboard

Panel NameDescription
Unmapped / Unknown fieldIf the value is equal or greater than 1 new Unmapped / Unknown fields exist

FAQ

How can I manage unmapped fields?

It is possible to manage the unmapped fields:

  • If you would like to neglect the unmapped field, go inside the CMS and move the document in DRAFT/TRASH
  • If you would register the unmapped field, go inside the console, and he has to register it

What happens if I register a field which previously was the source of an UNKNOWN_FIELD or UNCASTABLE_FIELD message?

The document related to an UNKNOWN_FIELD field will be removed from the collection, instead, document connected to an UNKNOWN_FIELD with type UNCASTABLE_FIELD will be retained.