5.1.3.2.1.13. Dashboards: Indices in read-only
When we launch a dashboard and there is no data displayed. In addition, an error appears in the bottom right corner of the interface: Error in visualization [esaggs] Cannot read properly ‘4’ of undefined. This error is due to the Elasticsearch disk space occupation, in the analytics node, exceeding the security threshold that Elasticsearch has set. In the Kibana log file, we can observe the following lines repeatedly:
{"type":"response","@timestamp":"2020-01-30T13:59:01Z","tags":["api"],"pid":5261,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"user-agent":"curl/7.29.0","host":"localhost:5601","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":200,"responseTime":11,"contentLength":9},"message":"GET /api/status 200 11ms - 9.0B"}
{"type":"log","@timestamp":"2020-01-30T13:59:01Z","tags":["error","task_manager"],"pid":5261,"message":"Failed to poll for work: [cluster_block_exception] index [.kibana_task_manager] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {\"path\":\"/.kibana_task_manager/_update/Maps-maps_telemetry\",\"query\":{\"if_seq_no\":113,\"if_primary_term\":12,\"refresh\":\"true\"},\"body\":\"{\\\"doc\\\":{\\\"type\\\":\\\"task\\\",\\\"task\\\":{\\\"taskType\\\":\\\"maps_telemetry\\\",\\\"state\\\":\\\"{\\\\\\\"runs\\\\\\\":1,\\\\\\\"stats\\\\\\\":{\\\\\\\"mapsTotalCount\\\\\\\":0,\\\\\\\"timeCaptured\\\\\\\":\\\\\\\"2020-01-18T23:00:01.612Z\\\\\\\",\\\\\\\"attributesPerMap\\\\\\\":{\\\\\\\"dataSourcesCount\\\\\\\":{\\\\\\\"min\\\\\\\":0,\\\\\\\"max\\\\\\\":0,\\\\\\\"avg\\\\\\\":0},\\\\\\\"layersCount\\\\\\\":{\\\\\\\"min\\\\\\\":0,\\\\\\\"max\\\\\\\":0,\\\\\\\"avg\\\\\\\":0},\\\\\\\"layerTypesCount\\\\\\\":{},\\\\\\\"emsVectorLayersCount\\\\\\\":{}}}}\\\",\\\"params\\\":\\\"{}\\\",\\\"attempts\\\":0,\\\"scheduledAt\\\":\\\"2020-01-14T11:50:35.793Z\\\",\\\"runAt\\\":\\\"2020-01-30T14:00:01.762Z\\\",\\\"status\\\":\\\"running\\\"},\\\"kibana\\\":{\\\"uuid\\\":\\\"44a2c02e-e349-4951-b306-5551bc636eb5\\\",\\\"version\\\":7020099,\\\"apiVersion\\\":1}}}\",\"statusCode\":403,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}],\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"},\\\"status\\\":403}\"}"}
Check the high disk space occupancy with the following command:
[root@opennac-analytics ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 177M 7.6G 3% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/mapper/cl-root 50G 4.5G 46G 9% /
/dev/sda1 1014M 314M 701M 31% /boot
/dev/mapper/cl-home 20G 33M 20G 1% /home
/dev/mapper/cl-var 122G 90.2G 32.2G 74% /var
tmpfs 1.6G 0 1.6G 0% /run/user/0
As you can see, /var file system has high occupancy in disk space.
Check Elasticsearch’s indices:
[root@opennac-analytics scripts]# curl -XGET http://localhost:9200/_cat/indices
green open radius-2020.01.30 yPkjd_w6RKKEInEyKSxaZA 1 0 1440 0 396.7kb 396.7kb
green open opennac-2020.02.03 A1FSPD_tQxq2ej4Pj9xvIw 1 0 25509 0 44.5mb 44.5mb
green open opennac-2020.02.01 LxZ3SNDgTDCS2QEeoplzdg 1 0 16871 0 19.4mb 19.4mb
green open radius-2020.01.31 mcrewf3wR66r3kXOP2TmkQ 1 0 1440 0 385.6kb 385.6kb
green open opennac-2020.01.30 DMjGA941QKSV2SK9tsctaQ 1 0 11002 0 36mb 36mb
green open misc-2020.01.29 uSU5lSGgQU-zBkS3V0sZcA 1 0 1 0 13.5kb 13.5kb
green open radius-2020.02.02 7uDF9_LCQJaJpsQqFXHlYg 1 0 1440 0 462kb 462kb
green open bro-2020.02.04 4GpTfcnRQ9qzFqir4Vszbw 1 0 578099 0 2.4gb 2.4gb
green open misc-2020.01.30 JSiF98e6QyyFgqL47rUZ7Q 1 0 1 0 13.5kb 13.5kb
green open radius-2020.01.27 a8O3B7qhQECcMLwdonhZjw 1 0 1440 0 394.6kb 394.6kb
green open radius-2020.01.29 RHtfd_noQ0SXNXCdZvJ52A 1 0 1440 0 431kb 431kb
green open radius-2020.02.07 YMsT-gvsT-2J4_Wmsyahrw 1 0 1394 0 621.1kb 621.1kb
green open bro-2020.02.01 WPdAF_s1R6u4XW6wT3ZTag 1 0 6144969 0 20.6gb 20.6gb
green open opennac-2020.01.31 CX6cmm8KS4mPbsrmsCr9PQ 1 0 22458 0 63.3mb 63.3mb
green open opennac_ud dHB-XoF5TVG-Fw55h-dPow 1 0 10406 4342 33.3mb 33.3mb
green open bro-2020.02.06 DKI7O6nvTwiO45V_-92FUg 1 0 2 0 249.7kb 249.7kb
green open radius-2020.01.24 WneZcEaFRlGqZi5SH3WkGQ 1 0 1440 0 408.1kb 408.1kb
green open radius-2020.02.03 jHQqI2HqT7qImqtx8W8vDQ 1 0 1436 0 532.7kb 532.7kb
green open opennac-2020.02.07 nNN1NYzJRYO7PsDahwGkLg 1 0 3104 0 4.3mb 4.3mb
green open radius-2020.01.28 42A4FSwdQnCZFOqzOrHj2Q 1 0 1440 0 417.8kb 417.8kb
green open radius-2020.01.26 9xwTTDefQdaQ51lwgViJoA 1 0 1440 0 397.1kb 397.1kb
green open opennac-2020.01.25 zmlq1oCPTXWxtFeqRpzCow 1 0 54 0 68.6kb 68.6kb
green open bro-2020.02.02 1kMV8LuzTNOasjgO1RSiRQ 1 0 6019000 0 23gb 23gb
green open radius-2020.02.04 qG-CVEd2SA-d0UPaTzocRw 1 0 634 0 208.2kb 208.2kb
green open .kibana_1 wqKmCrM_QfWpQ4om8m4iwQ 1 0 1742 14 757.8kb 757.8kb
green open .kibana_task_manager DfGYkrrZTU2bEh4CT-x-Fw 1 0 2 0 31.8kb 31.8kb
green open opennac-2020.02.04 qGtiVwXUQ2ul6Yp4HDehlg 1 0 3463 0 4.3mb 4.3mb
green open bro-2020.02.05 hlZQfRL2RK2wvsZM9POFHA 1 0 1 0 37.9kb 37.9kb
green open bro-2020.02.03 BkYjo5e8QjKeuuPMnBASOw 1 0 10171817 0 37gb 37gb
green open radius-2020.02.01 D-4lehdqRHuSDG2vghp1Qg 1 0 1440 0 393.1kb 393.1kb
green open bro-2020.02.07 NOT37UrXTWGCCvWrMM3-Tw 1 0 244342 0 1.3gb 1.3gb
green open opennac-2020.02.02 teRp8YDXTRauDMoaFs_AvA 1 0 16278 0 20.8mb 20.8mb
green open opennac-2020.01.29 cooKP-tdSVKrAj2GtVB-0Q 1 0 268 0 195.1kb 195.1kb
We can observe most indices with high occupancy. Bro is very active and occupies too much disk space. To resolve it we need to change the ElasticCurator configuration and change the number of days of the Bro indices history to a lower value: Edit /etc/elastCurator/action.yaml
actions:
1:
action: delete_indices
description: Delete bro-* > 7 days.
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: bro-
exclude:
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 7
exclude:
We need to change the unit_count and description fields to a lower value. Save the configuration file and execute ElastCurator:
/usr/share/opennac/analytics/scripts/elasticsearch_purge_index_curator.sh
The next step is to set indices to ES executing the “read only.sh” script.
Execute the following command and check the result:
/usr/share/opennac/analytics/scripts/read_only.sh
You can now check the disk space occupancy again and the indices with commands shown during the troubleshooting.