openstack-ansible-ops/elk_metrics_6x

Install ELK with beats to gather metrics

tags

openstack, ansible

About this repository

This set of playbooks will deploy elk cluster (Elasticsearch, Logstash, Kibana) with topbeat to gather metrics from hosts metrics to the ELK cluster.

These playbooks require Ansible 2.5+.

OpenStack-Ansible Integration

These playbooks can be used as standalone inventory or as an integrated part of an OpenStack-Ansible deployment. For a simple example of standalone inventory, see inventory.example.yml.

Optional | Load balancer VIP address

In order to use multi-node elasticsearch a loadbalancer is required. Haproxy can provide the load balancer functionality needed. The option internal_lb_vip_address is used as the endpoint (virtual IP address) services like Kibana will use when connecting to elasticsearch. If this option is omitted, the first node in the elasticsearch cluster will be used.

Optional | configure haproxy endpoints

Edit the /etc/openstack_deploy/user_variables.yml file and add the following lines.

haproxy_extra_services:
 - service:
      haproxy_service_name: kibana
      haproxy_ssl: False
      haproxy_backend_nodes: "{{ groups['kibana'] | default([]) }}"
      haproxy_port: 81  # This is set using the "kibana_nginx_port" variable
      haproxy_balance_type: tcp
  - service:
      haproxy_service_name: elastic-logstash
      haproxy_ssl: False
      haproxy_backend_nodes: "{{ groups['elastic-logstash'] | default([]) }}"
      haproxy_port: 5044  # This is set using the "logstash_beat_input_port" variable
      haproxy_balance_type: tcp
  - service:
      haproxy_service_name: elastic-logstash
      haproxy_ssl: False
      haproxy_backend_nodes: "{{ groups['elastic-logstash'] | default([]) }}"
      haproxy_port: 9201  # This is set using the "elastic_hap_port" variable
      haproxy_check_port: 9200  # This is set using the "elastic_port" variable
      haproxy_backend_port: 9200  # This is set using the "elastic_port" variable
      haproxy_balance_type: tcp

Optional | run the haproxy-install playbook

cd /opt/openstack-ansible/playbooks/
openstack-ansible haproxy-install.yml --tags=haproxy-service-config

Setup | system configuration

Clone the elk-osa repo

cd /opt
git clone https://github.com/openstack/openstack-ansible-ops

Copy the env.d file into place

cd /opt/openstack-ansible-ops/elk_metrics_6x
cp env.d/elk.yml /etc/openstack_deploy/env.d/

Copy the conf.d file into place

cp conf.d/elk.yml /etc/openstack_deploy/conf.d/

In elk.yml, list your logging hosts under elastic-logstash_hosts to create the elasticsearch cluster in multiple containers and one logging host under kibana_hosts to create the kibana container

vi /etc/openstack_deploy/conf.d/elk.yml

Create the containers

cd /opt/openstack-ansible/playbooks
openstack-ansible lxc-containers-create.yml -e 'container_group=elastic-logstash:kibana:apm-server'

Deploying | Installing with embedded Ansible

If this is being executed on a system that already has Ansible installed but is incompatible with these playbooks the script bootstrap-embedded-ansible.sh can be sourced to grab an embedded version of Ansible prior to executing the playbooks.

source bootstrap-embedded-ansible.sh

Deploying | Manually resolving the dependencies

This playbook has external role dependencies. If Ansible is not installed with the bootstrap-ansible.sh script these dependencies can be resolved with the ansible-galaxy command and the ansible-role-requirements.yml file.

• Example galaxy execution

ansible-galaxy install -r ansible-role-requirements.yml

Once the dependencies are set make sure to set the action plugin path to the location of the config_template action directory. This can be done using the environment variable ANSIBLE_ACTION_PLUGINS or through the use of an ansible.cfg file.

Deploying | The environment

Install master/data elasticsearch nodes on the elastic-logstash containers, deploy logstash, deploy kibana, and then deploy all of the service beats.

cd /opt/openstack-ansible-ops/elk_metrics_6x
ansible-playbook site.yml $USER_VARS

• The openstack-ansible command can be used if the version of ansible on the system is greater than 2.5.

The individual playbooks found within this repository can be independently run at anytime.

Optional | add Grafana visualizations

See the grafana directory for more information on how to deploy grafana. Once When deploying grafana, source the variable file from ELK in order to automatically connect grafana to the Elasticsearch datastore and import dashboards. Including the variable file is as simple as adding -e @../elk_metrics_6x/vars/variables.yml to the grafana playbook run.

Included dashboards.

• https://grafana.com/dashboards/5569
• https://grafana.com/dashboards/5566

Example command using the embedded Ansible from within the grafana directory.

ansible-playbook ${USER_VARS} installGrafana.yml \
                              -e @../elk_metrics_6x/vars/variables.yml \
                              -e galera_root_user=root \
                              -e 'galera_address={{ internal_lb_vip_address }}'

Trouble shooting

If everything goes bad, you can clean up with the following command

openstack-ansible /opt/openstack-ansible-ops/elk_metrics_6x/site.yml -e "elk_package_state=absent" --tags package_install
openstack-ansible /opt/openstack-ansible/playbooks/lxc-containers-destroy.yml --limit=kibana:elastic-logstash_all