5.1.3.2.4. ELK checks

In this section we can find automated checks that allow us to know the state of the stack ELK of an environment and to be able to identify the errors related to them.

These automated checks can be find in the all_checks.yml file of the following path:

cd /usr/share/opennac/ansible/

The following points detail the different checks that ansible performs in an automated way as well as the tags available when executing them.

5.1.3.2.4.1. Checks

• Ensure that the elasticsearch cluster for each analytics has the corresponding nodes and the number match with the number of analytics

• Ensure that the logstash output hosts are correctly configured

• Ensure that the filebeat output hosts are correctly configured

5.1.3.2.4.2. Tags

• elasticsearch

• logstash

• filebeat

5.1.3.2.4.3. Data Structure

• Elasticsearch:

"elasticsearch": {
    "{{ node_id }}": {
        "error": {
            "{{ node_id }}": "{{ node_ip }}"
        },
        "hostname": "{{ node_hostname }}",
        "ip": "{{ node_ip }}",
        "numberCheck": false,
        "ok": {
            "{{ node_id }}": "{{ node_ip }}"
        }
    }
}

The data structure shown is explained below:

• node_id: corresponds to the hostname of the node and contains the list of nodes with elasticsearch configured and their status.

• hostname: node hostname.

• ip: node IP address.

• ok: list of nodes with elasticsearch in OK status.

• error: list of nodes with elasticsearch in CRITICAL status.

• numberCheck: true if the number of analytics in the cluster matches the number of analytics that should be in the cluster, false if not.

• Logstash:

"logstash": {
    "{{ node_id }}": {
        "hostname": "{{ node_hostname }}",
        "ip": "{{ node_ip }}",
        "outputCheck": true
    }
}

The data structure shown is explained below:

• node_id: corresponds to the hostname of the node and contains the list of nodes with logstash configured and their status.

• hostname: node hostname.

• ip: node IP address.

• ok: list of nodes with logstash in OK status.

• error: list of nodes with logstash in CRITICAL status.

• outputCheck: true if the logstash output points to the elasticsearch touching nodes, false if not.

• Filebeat:

"filebeat": {
    "5000": {
        "{{ node_id }}": {
            "hostname": "{{ node_hostname }}",
            "ip": "{{ node_ip }}",
            "error": {},
            "ok": {
                "{{ node_id }}": "{{ node_ip }}"
            }
        }
    },
    "5001": {
        "{{ node_id }}": {
            "hostname": "{{ node_hostname }}",
            "ip": "{{ node_ip }}",
            "error": {},
            "ok": {
                "{{ node_id }}": "{{ node_ip }}"
            }
        }
    }
}

The data structure shown is explained below:

• node_id: corresponds to the hostname of the node and contains the node with filebeat configured and their status.

• hostname: node hostname.

• ip: node IP address.

• ok: list of nodes with filebeat in OK status.

• error: list of nodes with filebeat in CRITICAL status.

5.1.3.2.4.4. Example

Elasticsearch Command:

ansible-playbook -i inventory all_checks.yml --tags "elasticsearch"

Elasticsearch output:

"elasticsearch": {
    "02-analytics-04": {
        "error": {},
        "hostname": "02-analytics-04",
        "ip": "10.10.39.104",
        "numberCheck": false,
        "ok": {
            "02-analytics-04": "10.10.39.104",
            "02-analytics-05": "10.10.39.105"
        }
    }
}

Logstash Command:

ansible-playbook -i inventory all_checks.yml --tags "logstash"

Logstash output:

"logstash": {
    "02-aggregator-07": {
        "hostname": "02-aggregator-07",
        "ip": "10.10.39.107",
        "outputCheck": true
    }
}

Filebeat Command:

ansible-playbook -i inventory all_checks.yml --tags "filebeat"

Filebeat output:

"filebeat": {
    "5000": {
        "02-proxy-02": {
            "error": {},
            "hostname": "02-proxy-02",
            "ip": "10.10.39.102",
            "ok": {
                "02-worker-03": "10.10.39.103"
            }
        }
    },
    "5001": {
        "02-proxy-02": {
            "error": {},
            "hostname": "02-proxy-02",
            "ip": "10.10.39.102",
            "ok": {
                "02-worker-03": "10.10.39.103"
            }
        }
    }
}