7.4. Edit Elasticsearch mappings

In this section we will see how to change a mapping for specific fields in Elasticsearch. Mismapping a field can cause Logstash to be unable to inject the record into Elasticsearch, so that record will be lost.

In this case, in the opennac_ud index we can see that there are three fields that are mapped as string but are an integer.

../_images/eem.png


For editing the mapping, we need to go to the following path at ON Analytics:

cd /usr/share/opennac/analytics/elasticsearch

On this path, we will find the mappings for all the index-patterns:

../_images/eem1.png


In this case, we want to change the opennac_ud index pattern template. So the file to edit will be:

index_template_opennac_ud.json

Inside the file, we need to search for the fields we want to change. In this case the followings:

../_images/eem2.png


Then, we will change the type of the fields to long:

../_images/eem3.png


To apply the changes, we need to import the template to Elasticsearch. To do that, we need to execute the following command:

curl -s --noproxy "*" -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/opennac_ud' -d@/usr/share/opennac/analytics/elasticsearch/index_template_opennac_ud.json

To import all the templates we should execute:

curl -s --noproxy "*" -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/identities' -d@/usr/share/opennac/analytics/elasticsearch/index_template_identities.json
curl -s --noproxy "*" -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/bro' -d@/usr/share/opennac/analytics/elasticsearch/index_template_bro.json
curl -s --noproxy "*" -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/radius' -d@/usr/share/opennac/analytics/elasticsearch/index_template_radius.json
curl -s --noproxy "*" -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/opennac_captive' -d@/usr/share/opennac/analytics/elasticsearch/index_template_opennac_captive.json
curl -s --noproxy "*" -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/opennac' -d@/usr/share/opennac/analytics/elasticsearch/index_template_opennac.json
curl -s --noproxy "*" -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/opennac_macport' -d@/usr/share/opennac/analytics/elasticsearch/index_template_opennac_macport.json
curl -s --noproxy "*" -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/opennac_net_dev' -d@/usr/share/opennac/analytics/elasticsearch/index_template_opennac_nd.json
curl -s --noproxy "*" -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/opennac_nd-' -d@/usr/share/opennac/analytics/elasticsearch/index_template_opennac_nd-.json
curl -s --noproxy "*" -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/opennac_ud' -d@/usr/share/opennac/analytics/elasticsearch/index_template_opennac_ud.json
curl -s --noproxy "*" -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/misc' -d@/usr/share/opennac/analytics/elasticsearch/index_template_misc.json
curl -s --noproxy "*" -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/external_syslog' -d@/usr/share/opennac/analytics/elasticsearch/index_template_external_syslog.json
curl -s --noproxy "*" -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/third_party_vpn' -d@/usr/share/opennac/analytics/elasticsearch/index_template_third_party_vpn.json

When the template is imported to Elasticsearch, we need to update the kibana index patterns with that template. To do that, we need to execute the following script.

bash /usr/share/opennac/analytics/scripts/elk8_update.sh -o update

This script will regenerate Kibana’s templates and reindex the not daily indexes. The not daily indexes are:

  • opennac_ud

  • opennac_nd

  • identitites

  • third_party_vpn

For the daily indexes, the correct template will not be applied until the next day index is created. If we want to apply immediately the changes, we should reindex the daily index.

In this case our index is opennac_ud, so the changes should be applied. If we go again to discover, we will see that the mappings have changed from text to long.

../_images/eem4.png