opendev/system-config
Code Issues Proposed changes

Update logstash doc for an elasticsearch cluster

Browse Source 
Change-Id: I57b84c87599f129d9c273e29ff6b1a48c56c115d
This commit is contained in:
Jeremy Stanley 2014-02-26 16:16:57 +00:00
parent ccc39c3bb8
commit ac13b41bed
1 changed files with 8 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
doc/source/logstash.rst
View File

@ -13,7 +13,7 @@ At a Glance
:Hosts: :Hosts:
* http://logstash.openstack.org * http://logstash.openstack.org
* logstash-worker\*.openstack.org * logstash-worker\*.openstack.org
* elasticsearch.openstack.org * elasticsearch\*.openstack.org
:Puppet: :Puppet:
* :file:`modules/logstash` * :file:`modules/logstash`
* :file:`modules/openstack_project/manifests/logstash.pp` * :file:`modules/openstack_project/manifests/logstash.pp`
@ -89,6 +89,7 @@ bottleneck very quickly. This looks something like:
\ | / \ | /
\ | / \ | /
elasticsearch elasticsearch
cluster
| |
| |
kibana kibana
@ -170,17 +171,12 @@ ElasticSearch
ElasticSearch is basically a REST API layer for Lucene. It provides ElasticSearch is basically a REST API layer for Lucene. It provides
the storage and search engine for Logstash. It scales horizontally and the storage and search engine for Logstash. It scales horizontally and
loves it when you give it more memory. Currently we run a single node loves it when you give it more memory. Currently we run a multi-node
cluster on a large VM to give ElasticSearch both memory and disk space. cluster on large VMs to give ElasticSearch both memory and disk space.
Per index (Logstash creates one index per day) we have one replica (on Per index (Logstash creates one index per day) we have N+1 replica
the same node, this does not provide HA, it speeds up searches) and redundancy to distribute disk utilization and provide high availability.
five shards (each shard is basically its own index, having multiple Each replica is broken into multiple shards providing inceased indexing
shards increases indexing throughput). and search throughput as each shard is essentially a valid mini index.
As this setup grows and handles more logs we may need to add more
ElasticSearch nodes and run a proper cluster. Haven't reached that point
yet, but will probably be necessary as disk and memory footprints
increase.
Kibana Kibana
------ ------