Remove configuration management for ELK stack
We indicated to the OpenStack TC that this service would be going away after the Yoga cycle if no one stepped up to start maintaining it. That help didn't arrive in the form of OpenDev assistance (there is effort to use OpenSearch external to OpenDev) and Yoga has released. This means we are now clear to retire and shutdown this service. This change attempts to remove our configuration management for these services so that we can shutdown the servers afterwards. It was a good run. Sad to see it go but it wasn't sustainable anymore. Note a follow-up will clean up elastic-recheck which runs on the status server. Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/837619 Change-Id: I5f7f73affe7b97c74680d182e68eb4bfebbe23e1
This commit is contained in:
Clark Boylan
parent
a5f06418b6
commit
4279e20293
@ -1,293 +0,0 @@
|
||||
:title: Logstash
|
||||
|
||||
.. _logstash:
|
||||
|
||||
Logstash
|
||||
########
|
||||
|
||||
Logstash is a high-performance indexing and search engine for logs.
|
||||
|
||||
At a Glance
|
||||
===========
|
||||
|
||||
:Hosts:
|
||||
* http://logstash.openstack.org
|
||||
* logstash-worker\*.openstack.org
|
||||
* elasticsearch\*.openstack.org
|
||||
:Puppet:
|
||||
* https://opendev.org/opendev/puppet-logstash
|
||||
* :git_file:`modules/openstack_project/manifests/logstash.pp`
|
||||
* :git_file:`modules/openstack_project/manifests/logstash_worker.pp`
|
||||
* :git_file:`modules/openstack_project/manifests/elasticsearch.pp`
|
||||
:Configuration:
|
||||
* :git_file:`modules/openstack_project/files/logstash`
|
||||
* :git_file:`modules/openstack_project/templates/logstash`
|
||||
* `submit-logstash-jobs defaults`_
|
||||
:Projects:
|
||||
* http://logstash.net/
|
||||
* http://kibana.org/
|
||||
* http://www.elasticsearch.org/
|
||||
* http://crm114.sourceforge.net/
|
||||
:Bugs:
|
||||
* https://storyboard.openstack.org/#!/project/748
|
||||
* https://logstash.jira.com/secure/Dashboard.jspa
|
||||
* https://github.com/rashidkpc/Kibana/issues
|
||||
* https://github.com/elasticsearch/elasticsearch/issues
|
||||
|
||||
Overview
|
||||
========
|
||||
|
||||
Logs from Zuul test runs are sent to logstash where they are
|
||||
indexed and stored. Logstash facilitates reviewing logs from multiple
|
||||
sources in a single test run, searching for errors or particular
|
||||
events within a test run, as well as searching for log event trends
|
||||
across test runs.
|
||||
|
||||
System Architecture
|
||||
===================
|
||||
|
||||
There are four major layers in our Logstash setup.
|
||||
|
||||
1. Submit Logstash Jobs.
|
||||
The `logs post-playbook`_ in the Zuul ``base`` job submit logs defined
|
||||
in the `submit-logstash-jobs defaults`_ to a Logstash Indexer.
|
||||
2. Logstash Indexer.
|
||||
Reads these log events from the log pusher, filters them to remove
|
||||
unwanted lines, collapses multiline events together, and parses
|
||||
useful information out of the events before shipping them to
|
||||
ElasticSearch for storage and indexing.
|
||||
3. ElasticSearch.
|
||||
Provides log storage, indexing, and search.
|
||||
4. Kibana.
|
||||
A Logstash oriented web client for ElasticSearch. You can perform
|
||||
queries on your Logstash logs in ElasticSearch through Kibana using
|
||||
the Lucene query language.
|
||||
|
||||
Each layer scales horizontally. As the number of logs grows we can add
|
||||
more log pushers, more Logstash indexers, and more ElasticSearch nodes.
|
||||
Currently we have multiple Logstash worker nodes that pair a log pusher
|
||||
with a Logstash indexer. We did this as each Logstash process can only
|
||||
dedicate a single thread to filtering log events which turns into a
|
||||
bottleneck very quickly. This looks something like:
|
||||
|
||||
::
|
||||
|
||||
zuul post-logs playbook
|
||||
|
|
||||
|
|
||||
gearman-client
|
||||
/ | \
|
||||
/ | \
|
||||
gearman gearman gearman
|
||||
worker1 worker2 worker3
|
||||
| | |
|
||||
logstash logstash logstash
|
||||
indexer1 indexer2 indexer3
|
||||
\ | /
|
||||
\ | /
|
||||
elasticsearch
|
||||
cluster
|
||||
|
|
||||
|
|
||||
kibana
|
||||
|
||||
Log Pusher
|
||||
----------
|
||||
|
||||
This is an ansible module in the `submit-log-processor-jobs role`_. It
|
||||
submits Gearman jobs to push log files into logstash.
|
||||
|
||||
Log pushing looks like this:
|
||||
|
||||
* Zuul runs post-playbook on job completion.
|
||||
* Using info in the Gearman job log files are retrieved.
|
||||
* Log files are processed then shipped to Logstash.
|
||||
|
||||
Using Gearman allows us to scale the number of log pushers
|
||||
horizontally. It is as simple as adding another process that talks to
|
||||
the Gearman server.
|
||||
|
||||
If you are interested in technical details the source of these scripts
|
||||
can be found at
|
||||
|
||||
* https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-client.py
|
||||
* https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-worker.py
|
||||
|
||||
Logstash
|
||||
--------
|
||||
|
||||
Logstash does the heavy lifting of squashing all of our log lines into
|
||||
events with a common format. It reads the JSON log events from the log
|
||||
pusher connected to it, deletes events we don't want, parses log lines
|
||||
to set the timestamp, message, and other fields for the event, then
|
||||
ships these processed events off to ElasticSearch where they are stored
|
||||
and made queryable.
|
||||
|
||||
At a high level Logstash takes:
|
||||
|
||||
::
|
||||
|
||||
{
|
||||
"fields" {
|
||||
"build_name": "gate-foo",
|
||||
"build_numer": "10",
|
||||
"event_message": "2013-05-31T17:31:39.113 DEBUG Something happened",
|
||||
},
|
||||
}
|
||||
|
||||
And turns that into:
|
||||
|
||||
::
|
||||
|
||||
{
|
||||
"fields" {
|
||||
"build_name": "gate-foo",
|
||||
"build_numer": "10",
|
||||
"loglevel": "DEBUG"
|
||||
},
|
||||
"@message": "Something happened",
|
||||
"@timestamp": "2013-05-31T17:31:39.113Z",
|
||||
}
|
||||
|
||||
It flattens each log line into something that looks very much like
|
||||
all of the other events regardless of the source log line format. This
|
||||
makes querying your logs for lines from a specific build that failed
|
||||
between two timestamps with specific message content very easy. You
|
||||
don't need to write complicated greps instead you query against a
|
||||
schema.
|
||||
|
||||
The config file that tells Logstash how to do this flattening can be
|
||||
found at
|
||||
https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf
|
||||
|
||||
This works via the tags that are associated with a given message.
|
||||
|
||||
The tags in
|
||||
https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf
|
||||
are used to tell logstash how to parse a given file's messages, based
|
||||
on the file's message format.
|
||||
|
||||
When adding a new file to be indexed to
|
||||
https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-logstash-jobs/defaults/main.yaml
|
||||
at least one tag from the openstack-filters.conf file should be associated
|
||||
with the new file. One can expect to see '{%logmessage%}' instead of
|
||||
actual message data if indexing is not working properly.
|
||||
|
||||
In the event a new file's format is not covered, a patch for
|
||||
https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf
|
||||
should be submitted with an appropriate parsing pattern.
|
||||
|
||||
ElasticSearch
|
||||
-------------
|
||||
|
||||
ElasticSearch is basically a REST API layer for Lucene. It provides
|
||||
the storage and search engine for Logstash. It scales horizontally and
|
||||
loves it when you give it more memory. Currently we run a multi-node
|
||||
cluster on large VMs to give ElasticSearch both memory and disk space.
|
||||
Per index (Logstash creates one index per day) we have N+1 replica
|
||||
redundancy to distribute disk utilization and provide high availability.
|
||||
Each replica is broken into multiple shards providing increased indexing
|
||||
and search throughput as each shard is essentially a valid mini index.
|
||||
|
||||
To check on the cluster health, run this command on any es.* node::
|
||||
|
||||
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
|
||||
|
||||
Kibana
|
||||
------
|
||||
|
||||
Kibana is a ruby app sitting behind Apache that provides a nice web UI
|
||||
for querying Logstash events stored in ElasticSearch. Our install can
|
||||
be reached at http://logstash.openstack.org. See
|
||||
:ref:`query-logstash` for more info on using Kibana to perform
|
||||
queries.
|
||||
|
||||
simpleproxy
|
||||
-----------
|
||||
Simpleproxy is a simple tcp proxy which allows forwarding tcp connections from
|
||||
one host to another. We use it to forward mysql traffic from a publicly
|
||||
accessible host to the trove instance running the subunit2sql MySQL DB. This
|
||||
allows for public access to the data on the database through the host
|
||||
logstash.openstack.org.
|
||||
|
||||
.. _query-logstash:
|
||||
|
||||
Querying Logstash
|
||||
=================
|
||||
|
||||
Hop on over to http://logstash.openstack.org and by default you get the
|
||||
last 15 minutes of everything Logstash knows about in chunks of 100.
|
||||
We run a lot of tests but it is possible no logs have come in over the
|
||||
last 15 minutes, change the dropdown in the top left from ``Last 15m``
|
||||
to ``Last 60m`` to get a better window on the logs. At this point you
|
||||
should see a list of logs, if you click on a log event it will expand
|
||||
and show you all of the fields associated with that event and their
|
||||
values (note Chromium and Kibana seem to have trouble with this at times
|
||||
and some fields end up without values, use Firefox if this happens).
|
||||
You can search based on all of these fields and if you click the
|
||||
magnifying glass next to a field in the expanded event view it will add
|
||||
that field and value to your search. This is a good way of refining
|
||||
searches without a lot of typing.
|
||||
|
||||
The above is good info for poking around in the Logstash logs, but
|
||||
one of your changes has a failing test and you want to know why. We
|
||||
can jumpstart the refining process with a simple query.
|
||||
|
||||
``@fields.build_change:"$FAILING_CHANGE" AND @fields.build_patchset:"$FAILING_PATCHSET" AND @fields.build_name:"$FAILING_BUILD_NAME" AND @fields.build_number:"$FAILING_BUILD_NUMBER"``
|
||||
|
||||
This will show you all logs available from the patchset and build pair
|
||||
that failed. Chances are that this is still a significant number of
|
||||
logs and you will want to do more filtering. You can add more filters
|
||||
to the query using ``AND`` and ``OR`` and parentheses can be used to
|
||||
group sections of the query. Potential additions to the above query
|
||||
might be
|
||||
|
||||
* ``AND @fields.filename:"logs/syslog.txt"`` to get syslog events.
|
||||
* ``AND @fields.filename:"logs/screen-n-api.txt"`` to get Nova API events.
|
||||
* ``AND @fields.loglevel:"ERROR"`` to get ERROR level events.
|
||||
* ``AND @message"error"`` to get events with error in their message.
|
||||
and so on.
|
||||
|
||||
General query tips:
|
||||
|
||||
* Don't search ``All time``. ElasticSearch is bad at trying to find all
|
||||
the things it ever knew about. Give it a window of time to look
|
||||
through. You can use the presets in the dropdown to select a window or
|
||||
use the ``foo`` to ``bar`` boxes above the frequency graph.
|
||||
* Only the @message field can have fuzzy searches performed on it. Other
|
||||
fields require specific information.
|
||||
* This system is growing fast and may not always keep up with the load.
|
||||
Be patient. If expected logs do not show up immediately after the
|
||||
Zuul job completes wait a few minutes.
|
||||
|
||||
crm114
|
||||
=======
|
||||
|
||||
In an effort to assist with automated failure detection, the infra team
|
||||
has started leveraging crm114 to classify and analyze the messages stored
|
||||
by logstash.
|
||||
|
||||
The tool utilizes a statistical approach for classifying data, and is
|
||||
frequently used as an email spam detector. For logstash data, the idea
|
||||
is to flag those log entries that are not in passing runs and only in
|
||||
failing ones, which should be useful in pinpointing what caused the
|
||||
failures.
|
||||
|
||||
In the OpenStack logstash system, crm114 attaches an error_pr attribute
|
||||
to all indexed entries. Values from -1000.00 to -10.00 should be considered
|
||||
sufficient to get all potential errors as identified by the program.
|
||||
Used in a kibana query, it would be structured like this:
|
||||
|
||||
::
|
||||
|
||||
error_pr:["-1000.0" TO "-10.0"]
|
||||
|
||||
|
||||
This is still an early effort and additional tuning and refinement should
|
||||
be expected. Should the crm114 settings need to be tuned or expanded,
|
||||
a patch may be submitted for this file, which controls the process:
|
||||
https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/classify-log.crm
|
||||
|
||||
.. _logs post-playbook: https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/post-logs.yaml
|
||||
.. _submit-logstash-jobs defaults: https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-logstash-jobs/defaults/main.yaml
|
||||
.. _submit-log-processor-jobs role: https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-log-processor-jobs
|
||||
@ -16,7 +16,6 @@ Major Systems
|
||||
grafyaml
|
||||
keycloak
|
||||
zuul
|
||||
logstash
|
||||
elastic-recheck
|
||||
devstack-gate
|
||||
nodepool
|
||||
|
||||
@ -14,12 +14,6 @@ cacti_hosts:
|
||||
- bridge.openstack.org
|
||||
- cacti.openstack.org
|
||||
- eavesdrop01.opendev.org
|
||||
- elasticsearch02.openstack.org
|
||||
- elasticsearch03.openstack.org
|
||||
- elasticsearch04.openstack.org
|
||||
- elasticsearch05.openstack.org
|
||||
- elasticsearch06.openstack.org
|
||||
- elasticsearch07.openstack.org
|
||||
- ethercalc02.openstack.org
|
||||
- etherpad01.opendev.org
|
||||
- gitea-lb01.opendev.org
|
||||
@ -39,27 +33,6 @@ cacti_hosts:
|
||||
- kdc04.openstack.org
|
||||
- keycloak01.opendev.org
|
||||
- lists.openstack.org
|
||||
- logstash-worker01.openstack.org
|
||||
- logstash-worker02.openstack.org
|
||||
- logstash-worker03.openstack.org
|
||||
- logstash-worker04.openstack.org
|
||||
- logstash-worker05.openstack.org
|
||||
- logstash-worker06.openstack.org
|
||||
- logstash-worker07.openstack.org
|
||||
- logstash-worker08.openstack.org
|
||||
- logstash-worker09.openstack.org
|
||||
- logstash-worker10.openstack.org
|
||||
- logstash-worker11.openstack.org
|
||||
- logstash-worker12.openstack.org
|
||||
- logstash-worker13.openstack.org
|
||||
- logstash-worker14.openstack.org
|
||||
- logstash-worker15.openstack.org
|
||||
- logstash-worker16.openstack.org
|
||||
- logstash-worker17.openstack.org
|
||||
- logstash-worker18.openstack.org
|
||||
- logstash-worker19.openstack.org
|
||||
- logstash-worker20.openstack.org
|
||||
- logstash.openstack.org
|
||||
- nb01.opendev.org
|
||||
- nb02.opendev.org
|
||||
- nb03.opendev.org
|
||||
|
||||
@ -91,48 +91,6 @@ all:
|
||||
region_name: DFW
|
||||
public_v4: 104.239.144.232
|
||||
public_v6: 2001:4800:7818:104:be76:4eff:fe04:46c8
|
||||
elasticsearch02.openstack.org:
|
||||
ansible_host: 104.130.159.19
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 104.130.159.19
|
||||
public_v6: 2001:4800:7818:102:be76:4eff:fe05:40b
|
||||
elasticsearch03.openstack.org:
|
||||
ansible_host: 104.130.246.141
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 104.130.246.141
|
||||
public_v6: 2001:4800:7819:103:be76:4eff:fe05:f26
|
||||
elasticsearch04.openstack.org:
|
||||
ansible_host: 104.130.246.138
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 104.130.246.138
|
||||
public_v6: 2001:4800:7819:103:be76:4eff:fe02:3042
|
||||
elasticsearch05.openstack.org:
|
||||
ansible_host: 104.130.159.15
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 104.130.159.15
|
||||
public_v6: 2001:4800:7818:102:be76:4eff:fe04:55b5
|
||||
elasticsearch06.openstack.org:
|
||||
ansible_host: 104.130.246.111
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 104.130.246.111
|
||||
public_v6: 2001:4800:7819:103:be76:4eff:fe04:b9d7
|
||||
elasticsearch07.openstack.org:
|
||||
ansible_host: 104.130.246.44
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 104.130.246.44
|
||||
public_v6: 2001:4800:7819:103:be76:4eff:fe04:35bb
|
||||
ethercalc02.openstack.org:
|
||||
ansible_host: 162.242.144.125
|
||||
location:
|
||||
@ -279,153 +237,6 @@ all:
|
||||
region_name: DFW
|
||||
public_v4: 50.56.173.222
|
||||
public_v6: 2001:4800:780e:510:3bc3:d7f6:ff04:b736
|
||||
logstash-worker01.openstack.org:
|
||||
ansible_host: 172.99.116.137
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 172.99.116.137
|
||||
public_v6: 2001:4800:7821:105:be76:4eff:fe04:b9e7
|
||||
logstash-worker02.openstack.org:
|
||||
ansible_host: 23.253.253.66
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.253.66
|
||||
public_v6: 2001:4800:7817:104:be76:4eff:fe04:aee1
|
||||
logstash-worker03.openstack.org:
|
||||
ansible_host: 23.253.242.246
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.242.246
|
||||
public_v6: 2001:4800:7817:104:be76:4eff:fe04:81e6
|
||||
logstash-worker04.openstack.org:
|
||||
ansible_host: 166.78.47.27
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 166.78.47.27
|
||||
public_v6: 2001:4800:7817:101:be76:4eff:fe04:46ec
|
||||
logstash-worker05.openstack.org:
|
||||
ansible_host: 23.253.100.127
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.100.127
|
||||
public_v6: 2001:4800:7817:101:be76:4eff:fe04:3b27
|
||||
logstash-worker06.openstack.org:
|
||||
ansible_host: 23.253.100.168
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.100.168
|
||||
public_v6: 2001:4800:7817:101:be76:4eff:fe04:5e21
|
||||
logstash-worker07.openstack.org:
|
||||
ansible_host: 172.99.116.187
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 172.99.116.187
|
||||
public_v6: 2001:4800:7821:105:be76:4eff:fe04:bca2
|
||||
logstash-worker08.openstack.org:
|
||||
ansible_host: 166.78.47.61
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 166.78.47.61
|
||||
public_v6: 2001:4800:7817:101:be76:4eff:fe04:432c
|
||||
logstash-worker09.openstack.org:
|
||||
ansible_host: 23.253.119.179
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.119.179
|
||||
public_v6: 2001:4800:7817:103:be76:4eff:fe04:38d1
|
||||
logstash-worker10.openstack.org:
|
||||
ansible_host: 172.99.116.111
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 172.99.116.111
|
||||
public_v6: 2001:4800:7821:105:be76:4eff:fe04:3a04
|
||||
logstash-worker11.openstack.org:
|
||||
ansible_host: 166.78.47.86
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 166.78.47.86
|
||||
public_v6: 2001:4800:7817:101:be76:4eff:fe05:3e1
|
||||
logstash-worker12.openstack.org:
|
||||
ansible_host: 166.78.174.249
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 166.78.174.249
|
||||
public_v6: 2001:4800:7815:105:be76:4eff:fe01:61e4
|
||||
logstash-worker13.openstack.org:
|
||||
ansible_host: 23.253.230.28
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.230.28
|
||||
public_v6: 2001:4800:7817:103:be76:4eff:fe04:6ff1
|
||||
logstash-worker14.openstack.org:
|
||||
ansible_host: 172.99.116.170
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 172.99.116.170
|
||||
public_v6: 2001:4800:7821:105:be76:4eff:fe04:4a4a
|
||||
logstash-worker15.openstack.org:
|
||||
ansible_host: 23.253.242.14
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.242.14
|
||||
public_v6: 2001:4800:7817:104:be76:4eff:fe04:829b
|
||||
logstash-worker16.openstack.org:
|
||||
ansible_host: 23.253.230.58
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.230.58
|
||||
public_v6: 2001:4800:7817:103:be76:4eff:fe03:5f45
|
||||
logstash-worker17.openstack.org:
|
||||
ansible_host: 172.99.117.120
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 172.99.117.120
|
||||
public_v6: 2001:4800:7821:105:be76:4eff:fe04:6173
|
||||
logstash-worker18.openstack.org:
|
||||
ansible_host: 172.99.116.203
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 172.99.116.203
|
||||
public_v6: 2001:4800:7821:105:be76:4eff:fe05:435
|
||||
logstash-worker19.openstack.org:
|
||||
ansible_host: 166.78.47.173
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 166.78.47.173
|
||||
public_v6: 2001:4800:7817:101:be76:4eff:fe03:a5ec
|
||||
logstash-worker20.openstack.org:
|
||||
ansible_host: 23.253.76.170
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.76.170
|
||||
public_v6: 2001:4800:7815:105:be76:4eff:fe04:59a6
|
||||
logstash01.openstack.org:
|
||||
ansible_host: 104.239.141.90
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 104.239.141.90
|
||||
public_v6: 2001:4800:7819:104:be76:4eff:fe04:935b
|
||||
meetpad01.opendev.org:
|
||||
ansible_host: 104.239.240.194
|
||||
location:
|
||||
|
||||
@ -1,4 +0,0 @@
|
||||
iptables_extra_allowed_groups:
|
||||
- {'protocol': 'tcp', 'port': '9200:9400', 'group': 'elasticsearch'}
|
||||
- {'protocol': 'tcp', 'port': '9200:9400', 'group': 'logstash'}
|
||||
- {'protocol': 'tcp', 'port': '9200:9400', 'group': 'logstash-worker'}
|
||||
@ -8,7 +8,6 @@ iptables_extra_allowed_hosts:
|
||||
|
||||
iptables_extra_allowed_groups:
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'logstash'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'zookeeper'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'zuul'}
|
||||
|
||||
@ -8,7 +8,6 @@ iptables_extra_allowed_hosts:
|
||||
|
||||
iptables_extra_allowed_groups:
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'logstash'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'zookeeper'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'zuul'}
|
||||
|
||||
@ -1,6 +0,0 @@
|
||||
iptables_extra_public_tcp_ports:
|
||||
- 80
|
||||
- 3306
|
||||
iptables_extra_allowed_groups:
|
||||
- {'protocol': 'tcp', 'port': '4730', 'group': 'logstash-worker'}
|
||||
- {'protocol': 'tcp', 'port': '4730', 'group': 'zuul-executor'}
|
||||
@ -52,7 +52,6 @@ groups:
|
||||
- adns*.opendev.org
|
||||
- ns*.opendev.org
|
||||
eavesdrop: eavesdrop[0-9]*.opendev.org
|
||||
elasticsearch: elasticsearch[0-9]*.open*.org
|
||||
ethercalc: ethercalc*.open*.org
|
||||
etherpad: etherpad[0-9]*.open*.org
|
||||
gitea:
|
||||
@ -105,10 +104,6 @@ groups:
|
||||
- storyboard[0-9]*.opendev.org
|
||||
- translate[0-9]*.open*.org
|
||||
- zuul[0-9]*.opendev.org
|
||||
logstash:
|
||||
- logstash[0-9]*.open*.org
|
||||
logstash-worker:
|
||||
- logstash-worker[0-9]*.open*.org
|
||||
mailman:
|
||||
- lists*.katacontainers.io
|
||||
- lists*.open*.org
|
||||
@ -131,11 +126,8 @@ groups:
|
||||
- paste[0-9]*.opendev.org
|
||||
puppet:
|
||||
- cacti[0-9]*.open*.org
|
||||
- elasticsearch[0-9]*.open*.org
|
||||
- ethercalc[0-9]*.open*.org
|
||||
- health[0-9]*.openstack.org
|
||||
- logstash-worker[0-9]*.open*.org
|
||||
- logstash[0-9]*.open*.org
|
||||
- status*.open*.org
|
||||
- storyboard-dev[0-9]*.opendev.org
|
||||
- storyboard[0-9]*.opendev.org
|
||||
@ -143,11 +135,8 @@ groups:
|
||||
- translate[0-9]*.open*.org
|
||||
puppet4:
|
||||
- cacti[0-9]*.open*.org
|
||||
- elasticsearch[0-9]*.open*.org
|
||||
- ethercalc[0-9]*.open*.org
|
||||
- health[0-9]*.openstack.org
|
||||
- logstash-worker[0-9]*.open*.org
|
||||
- logstash[0-9]*.open*.org
|
||||
- status*.open*.org
|
||||
- storyboard[0-9]*.opendev.org
|
||||
- storyboard-dev[0-9]*.opendev.org
|
||||
|
||||
@ -20,66 +20,6 @@ node /^ethercalc\d+\.open.*\.org$/ {
|
||||
}
|
||||
}
|
||||
|
||||
# Node-OS: xenial
|
||||
node /^logstash\d*\.open.*\.org$/ {
|
||||
class { 'openstack_project::server': }
|
||||
|
||||
class { 'openstack_project::logstash':
|
||||
discover_nodes => [
|
||||
'elasticsearch03.openstack.org:9200',
|
||||
'elasticsearch04.openstack.org:9200',
|
||||
'elasticsearch05.openstack.org:9200',
|
||||
'elasticsearch06.openstack.org:9200',
|
||||
'elasticsearch07.openstack.org:9200',
|
||||
'elasticsearch02.openstack.org:9200',
|
||||
],
|
||||
subunit2sql_db_host => hiera('subunit2sql_db_host', ''),
|
||||
subunit2sql_db_pass => hiera('subunit2sql_db_password', ''),
|
||||
}
|
||||
}
|
||||
|
||||
# Node-OS: xenial
|
||||
node /^logstash-worker\d+\.open.*\.org$/ {
|
||||
$group = 'logstash-worker'
|
||||
|
||||
$elasticsearch_nodes = [
|
||||
'elasticsearch02.openstack.org',
|
||||
'elasticsearch03.openstack.org',
|
||||
'elasticsearch04.openstack.org',
|
||||
'elasticsearch05.openstack.org',
|
||||
'elasticsearch06.openstack.org',
|
||||
'elasticsearch07.openstack.org',
|
||||
]
|
||||
|
||||
class { 'openstack_project::server': }
|
||||
|
||||
class { 'openstack_project::logstash_worker':
|
||||
discover_node => 'elasticsearch03.openstack.org',
|
||||
enable_mqtt => false,
|
||||
mqtt_password => hiera('mqtt_service_user_password'),
|
||||
mqtt_ca_cert_contents => hiera('mosquitto_tls_ca_file'),
|
||||
}
|
||||
}
|
||||
|
||||
# Node-OS: xenial
|
||||
node /^elasticsearch\d+\.open.*\.org$/ {
|
||||
$group = "elasticsearch"
|
||||
|
||||
$elasticsearch_nodes = [
|
||||
'elasticsearch02.openstack.org',
|
||||
'elasticsearch03.openstack.org',
|
||||
'elasticsearch04.openstack.org',
|
||||
'elasticsearch05.openstack.org',
|
||||
'elasticsearch06.openstack.org',
|
||||
'elasticsearch07.openstack.org',
|
||||
]
|
||||
|
||||
class { 'openstack_project::server': }
|
||||
class { 'openstack_project::elasticsearch_node':
|
||||
discover_nodes => $elasticsearch_nodes,
|
||||
}
|
||||
}
|
||||
|
||||
# A machine to run Storyboard
|
||||
# Node-OS: xenial
|
||||
node /^storyboard\d+\.opendev\.org$/ {
|
||||
|
||||
@ -58,15 +58,12 @@ SOURCE_MODULES["https://github.com/voxpupuli/puppet-nodejs"]="v2.3.0"
|
||||
# Please keep sorted
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-bup"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-elastic_recheck"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-elasticsearch"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-ethercalc"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-httpd"]="origin/master"
|
||||
# Storyboard and translate use the jeepyb module
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-jeepyb"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-kibana"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-log_processor"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-logrotate"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-logstash"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-mysql_backup"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-pip"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-project_config"]="origin/master"
|
||||
@ -76,7 +73,6 @@ INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-reviewday"]="origin/mast
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-simpleproxy"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-ssh"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-storyboard"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-subunit2sql"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-tmpreaper"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-ulimit"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-user"]="origin/master"
|
||||
|
||||
@ -1,3 +0,0 @@
|
||||
# Increase the max heap size to twice the default.
|
||||
# Default is 25% of memory or 1g whichever is less.
|
||||
JAVA_ARGS='-Xmx2g'
|
||||
@ -1,46 +0,0 @@
|
||||
# Copyright 2013 Hewlett-Packard Development Company, L.P.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
# Elasticsearch server glue class.
|
||||
#
|
||||
class openstack_project::elasticsearch_node (
|
||||
$discover_nodes = ['localhost'],
|
||||
$heap_size = '30g',
|
||||
) {
|
||||
class { 'logstash::elasticsearch': }
|
||||
|
||||
class { '::elasticsearch':
|
||||
es_template_config => {
|
||||
'index.store.compress.stored' => true,
|
||||
'index.store.compress.tv' => true,
|
||||
'indices.memory.index_buffer_size' => '33%',
|
||||
'indices.breaker.fielddata.limit' => '70%',
|
||||
'bootstrap.mlockall' => true,
|
||||
'gateway.recover_after_nodes' => '5',
|
||||
'gateway.recover_after_time' => '5m',
|
||||
'gateway.expected_nodes' => '6',
|
||||
'discovery.zen.minimum_master_nodes' => '4',
|
||||
'discovery.zen.ping.multicast.enabled' => false,
|
||||
'discovery.zen.ping.unicast.hosts' => $discover_nodes,
|
||||
'http.cors.enabled' => true,
|
||||
'http.cors.allow-origin' => "'*'", # lint:ignore:double_quoted_strings
|
||||
},
|
||||
heap_size => $heap_size,
|
||||
version => '1.7.6',
|
||||
}
|
||||
|
||||
class { 'logstash::curator':
|
||||
keep_for_days => '7',
|
||||
}
|
||||
}
|
||||
@ -1,47 +0,0 @@
|
||||
# Copyright 2013 Hewlett-Packard Development Company, L.P.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
# Logstash web frontend glue class.
|
||||
#
|
||||
class openstack_project::logstash (
|
||||
$discover_nodes = ['elasticsearch02.openstack.org:9200'],
|
||||
$statsd_host = 'graphite.opendev.org',
|
||||
$subunit2sql_db_host,
|
||||
$subunit2sql_db_pass,
|
||||
) {
|
||||
class { 'logstash::web':
|
||||
frontend => 'kibana',
|
||||
discover_nodes => $discover_nodes,
|
||||
proxy_elasticsearch => true,
|
||||
}
|
||||
|
||||
class { 'log_processor': }
|
||||
|
||||
class { 'log_processor::geard':
|
||||
statsd_host => $statsd_host,
|
||||
}
|
||||
|
||||
include 'subunit2sql'
|
||||
|
||||
class { 'subunit2sql::server':
|
||||
db_host => $subunit2sql_db_host,
|
||||
db_pass => $subunit2sql_db_pass,
|
||||
}
|
||||
|
||||
include 'simpleproxy'
|
||||
|
||||
class { 'simpleproxy::server':
|
||||
db_host => $subunit2sql_db_host,
|
||||
}
|
||||
}
|
||||
@ -1,100 +0,0 @@
|
||||
# Copyright 2013 Hewlett-Packard Development Company, L.P.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
# Logstash indexer worker glue class.
|
||||
#
|
||||
class openstack_project::logstash_worker (
|
||||
$discover_node = 'elasticsearch02.openstack.org',
|
||||
$filter_rev = 'master',
|
||||
$filter_source = 'https://opendev.org/openstack/logstash-filters',
|
||||
$enable_mqtt = true,
|
||||
$mqtt_hostname = 'firehose.openstack.org',
|
||||
$mqtt_port = 8883,
|
||||
$mqtt_topic = "logstash/${::hostname}",
|
||||
$mqtt_username = 'infra',
|
||||
$mqtt_password = undef,
|
||||
$mqtt_ca_cert_contents = undef,
|
||||
) {
|
||||
|
||||
file { '/etc/logprocessor/worker.yaml':
|
||||
ensure => present,
|
||||
owner => 'root',
|
||||
group => 'root',
|
||||
mode => '0644',
|
||||
content => template('openstack_project/logstash/jenkins-log-worker.yaml.erb'),
|
||||
require => Class['::log_processor'],
|
||||
}
|
||||
|
||||
file { '/etc/default/logstash-indexer':
|
||||
ensure => present,
|
||||
owner => 'root',
|
||||
group => 'root',
|
||||
mode => '0644',
|
||||
source => 'puppet:///modules/openstack_project/logstash/logstash-indexer.default',
|
||||
}
|
||||
|
||||
vcsrepo { '/opt/logstash-filters':
|
||||
ensure => latest,
|
||||
provider => git,
|
||||
revision => $filter_rev,
|
||||
source => $filter_source,
|
||||
}
|
||||
|
||||
include ::logstash
|
||||
|
||||
logstash::filter { 'openstack-logstash-filters':
|
||||
level => '50',
|
||||
target => '/opt/logstash-filters/filters/openstack-filters.conf',
|
||||
require => [
|
||||
Class['::logstash'],
|
||||
Vcsrepo['/opt/logstash-filters'],
|
||||
],
|
||||
notify => Service['logstash'],
|
||||
}
|
||||
|
||||
file { '/etc/logstash/mqtt-root-CA.pem.crt':
|
||||
ensure => present,
|
||||
content => $mqtt_ca_cert_contents,
|
||||
replace => true,
|
||||
owner => 'root',
|
||||
group => 'root',
|
||||
mode => '0555',
|
||||
require => Class['::logstash'],
|
||||
}
|
||||
|
||||
validate_array($elasticsearch_nodes) # needed by output.conf.erb
|
||||
class { '::logstash::indexer':
|
||||
input_template => 'openstack_project/logstash/input.conf.erb',
|
||||
output_template => 'openstack_project/logstash/output.conf.erb',
|
||||
require => Logstash::Filter['openstack-logstash-filters'],
|
||||
}
|
||||
|
||||
include ::log_processor
|
||||
log_processor::worker { 'A':
|
||||
config_file => '/etc/logprocessor/worker.yaml',
|
||||
require => File['/etc/logprocessor/worker.yaml'],
|
||||
}
|
||||
log_processor::worker { 'B':
|
||||
config_file => '/etc/logprocessor/worker.yaml',
|
||||
require => File['/etc/logprocessor/worker.yaml'],
|
||||
}
|
||||
log_processor::worker { 'C':
|
||||
config_file => '/etc/logprocessor/worker.yaml',
|
||||
require => File['/etc/logprocessor/worker.yaml'],
|
||||
}
|
||||
log_processor::worker { 'D':
|
||||
config_file => '/etc/logprocessor/worker.yaml',
|
||||
require => File['/etc/logprocessor/worker.yaml'],
|
||||
}
|
||||
}
|
||||
@ -1,8 +0,0 @@
|
||||
input {
|
||||
tcp {
|
||||
host => "localhost"
|
||||
port => 9999
|
||||
codec => json_lines {}
|
||||
type => "jenkins"
|
||||
}
|
||||
}
|
||||
@ -1,13 +0,0 @@
|
||||
gearman-host: logstash.openstack.org
|
||||
gearman-port: 4730
|
||||
output-host: localhost
|
||||
output-port: 9999
|
||||
output-mode: tcp
|
||||
crm114-script: /usr/local/bin/classify-log.crm
|
||||
crm114-data: /var/lib/crm114
|
||||
mqtt-host: <%= @mqtt_hostname %>
|
||||
mqtt-port: <%= @mqtt_port %>
|
||||
mqtt-topic: gearman-logstash/<%= @hostname %>
|
||||
mqtt-user: <%= @mqtt_username %>
|
||||
mqtt-pass: <%= @mqtt_password %>
|
||||
mqtt-ca-certs: /etc/logstash/mqtt-root-CA.pem.crt
|
||||
@ -1,9 +0,0 @@
|
||||
output {
|
||||
elasticsearch {
|
||||
hosts => <%= @elasticsearch_nodes.map { |node| node + ":9200" }.inspect %>
|
||||
manage_template => false
|
||||
flush_size => 1024
|
||||
timeout => 300
|
||||
}
|
||||
|
||||
}
|
||||
@ -1,3 +0,0 @@
|
||||
output {
|
||||
redis { host => "127.0.0.1" data_type => "list" key => "logstash" }
|
||||
}
|
||||
@ -22,11 +22,6 @@ results:
|
||||
- letsencrypt
|
||||
- mailman
|
||||
|
||||
logstash-worker02.openstack.org:
|
||||
- logstash-worker
|
||||
- puppet
|
||||
- puppet4
|
||||
|
||||
mirror02.regionone.linaro-us.opendev.org:
|
||||
- afs-client
|
||||
- kerberos-client
|
||||
|
||||
@ -11,14 +11,11 @@
|
||||
- opendev/ansible-role-puppet
|
||||
- opendev/puppet-bup
|
||||
- opendev/puppet-elastic_recheck
|
||||
- opendev/puppet-elasticsearch
|
||||
- opendev/puppet-ethercalc
|
||||
- opendev/puppet-httpd
|
||||
- opendev/puppet-jeepyb
|
||||
- opendev/puppet-kibana
|
||||
- opendev/puppet-log_processor
|
||||
- opendev/puppet-logrotate
|
||||
- opendev/puppet-logstash
|
||||
- opendev/puppet-mysql_backup
|
||||
- opendev/puppet-openstack_infra_spec_helper
|
||||
- opendev/puppet-pip
|
||||
@ -28,7 +25,6 @@
|
||||
- opendev/puppet-simpleproxy
|
||||
- opendev/puppet-ssh
|
||||
- opendev/puppet-storyboard
|
||||
- opendev/puppet-subunit2sql
|
||||
- opendev/puppet-tmpreaper
|
||||
- opendev/puppet-ulimit
|
||||
- opendev/puppet-user
|
||||
@ -87,19 +83,15 @@
|
||||
- opendev/puppet-project_config
|
||||
- opendev/puppet-ethercalc
|
||||
- opendev/puppet-httpd
|
||||
- opendev/puppet-subunit2sql
|
||||
- opendev/puppet-reviewday
|
||||
- opendev/puppet-kibana
|
||||
- opendev/puppet-redis
|
||||
- opendev/puppet-zanata
|
||||
- opendev/puppet-logstash
|
||||
- opendev/puppet-tmpreaper
|
||||
- opendev/puppet-elastic_recheck
|
||||
- opendev/puppet-ulimit
|
||||
- opendev/puppet-logrotate
|
||||
- opendev/puppet-elasticsearch
|
||||
- opendev/puppet-storyboard
|
||||
- opendev/puppet-log_processor
|
||||
- opendev/puppet-simpleproxy
|
||||
- opendev/puppet-bup
|
||||
- opendev/puppet-ssh
|
||||
|
||||
Loading…
Reference in New Issue
Block a user