Merge "Remove firehose.openstack.org"
This commit is contained in:
Zuul
Gerrit Code Review
commit
cb5898ae0a
@ -1,366 +0,0 @@
|
||||
:title: Firehose
|
||||
|
||||
.. _firehose:
|
||||
|
||||
Firehose
|
||||
########
|
||||
|
||||
The unified message bus for Infra services.
|
||||
|
||||
At a Glance
|
||||
===========
|
||||
|
||||
:Hosts:
|
||||
* firehose*.openstack.org
|
||||
:Puppet:
|
||||
* https://opendev.org/opendev/puppet-mosquitto
|
||||
* https://opendev.org/opendev/puppet-germqtt
|
||||
* https://opendev.org/opendev/puppet-lpmqtt
|
||||
* :git_file:`modules/openstack_project/manifests/firehose.pp`
|
||||
:Projects:
|
||||
* https://mosquitto.org/
|
||||
* http://opendev.org/opendev/germqtt/
|
||||
* http://opendev.org/opendev/lpmqtt/
|
||||
|
||||
Overview
|
||||
========
|
||||
|
||||
The firehose is an infra run MQTT broker that is a place for any infra run
|
||||
service to publish events to. The concept behind it is that if anything needs
|
||||
to consume an event from an infra run service we should have a single place
|
||||
to go for consuming them.
|
||||
|
||||
firehose.openstack.org hosts an instance of Mosquitto to be the MQTT broker
|
||||
and also locally runs an instance of germqtt to publish the gerrit event
|
||||
stream over MQTT and lpmqtt to publish a launchpad event stream over MQTT.
|
||||
|
||||
Connection Info
|
||||
---------------
|
||||
|
||||
firehose.openstack.org has 2 open ports for MQTT traffic:
|
||||
|
||||
* **1883** - The default MQTT port
|
||||
* **80** - Uses websockets for the MQTT communication
|
||||
* **8883** - The default SSL/TLS MQTT port
|
||||
* **443** - The SSL/TLS websockets port
|
||||
|
||||
|
||||
Topics
|
||||
------
|
||||
|
||||
Topics at a top level are set based on the name of the service publishing the
|
||||
messages. The higher levels are specified by the publisher. For example::
|
||||
|
||||
gerrit/openstack-infra/germqtt/comment-added
|
||||
|
||||
is a typical message topic on firehose. The top level 'gerrit' specifies the
|
||||
service the message is from, and the rest of the message comes from germqtt
|
||||
(the daemon used for publishing the gerrit events)
|
||||
|
||||
MQTT topics are hierarchical and you can filter your subscription on part of the
|
||||
hierarchy. `[1]`_
|
||||
|
||||
.. _[1]: https://mosquitto.org/man/mqtt-7.html
|
||||
|
||||
Services Publishing to firehose
|
||||
-------------------------------
|
||||
|
||||
As of right now the following services publish messages to the firehose:
|
||||
|
||||
+-----------------+------------------+----------------------------+
|
||||
| Service | Base Topic | Source of Messages |
|
||||
+=================+==================+============================+
|
||||
| ansible | ansible | `ansible_mqtt_plugin`_ |
|
||||
+-----------------+------------------+----------------------------+
|
||||
| gerrit | gerrit | `germqtt`_ |
|
||||
+-----------------+------------------+----------------------------+
|
||||
| launchpad | launchpad | `lpmqtt`_ |
|
||||
+-----------------+------------------+----------------------------+
|
||||
| subunit worker | gearman-subunit | `subunit-gearman-worker`_ |
|
||||
+-----------------+------------------+----------------------------+
|
||||
| logstash worker | gearman-logstash | `logstash-gearman-worker`_ |
|
||||
+-----------------+------------------+----------------------------+
|
||||
|
||||
.. _germqtt: http://opendev.org/opendev/germqtt/
|
||||
.. _lpmqtt: http://opendev.org/opendev/lpmqtt/
|
||||
.. _subunit-gearman-worker: https://opendev.org/opendev/puppet-subunit2sql/src/branch/master/files/subunit-gearman-worker.py
|
||||
.. _ansible_mqtt_plugin: https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/puppetmaster/mqtt.py
|
||||
.. _logstash-gearman-worker: https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-worker.py
|
||||
|
||||
For a full schema description see :ref:`firehose_schema`
|
||||
|
||||
Client Usage
|
||||
============
|
||||
There is no outside access to publishing messages to the firehose available,
|
||||
however anyone is able to subscribe to any topic services publish to. To
|
||||
interact with the firehose you need to use the MQTT protocol. The specific
|
||||
contents of the payload are dictated by the service publishing the
|
||||
messages. So this section only covers how to subscribe and receive the messages
|
||||
not how to consume the content received.
|
||||
|
||||
Available Clients
|
||||
-----------------
|
||||
The MQTT community wiki maintains a page that lists available client bindings
|
||||
for many languages here: https://github.com/mqtt/mqtt.github.io/wiki/libraries
|
||||
For python using the `paho-mqtt`_ library is recommended
|
||||
|
||||
.. _paho-mqtt: https://pypi.org/project/paho-mqtt/
|
||||
|
||||
CLI Example
|
||||
-----------
|
||||
The mosquitto project also provides both a CLI publisher and subscriber client
|
||||
that can be used to easily subscribe to any topic and receive the messages. On
|
||||
debian based distributions these are included in the mosquitto-clients package.
|
||||
For example, to subscribe to every topic on the firehose you would run::
|
||||
|
||||
mosquitto_sub -h firehose.openstack.org --topic '#'
|
||||
|
||||
You can adjust the value of the topic parameter to make what you're subscribing
|
||||
to more specific.
|
||||
|
||||
MQTT Protocol Example
|
||||
---------------------
|
||||
Interacting with firehose on the unencrypted MQTT port is normally pretty easy in
|
||||
most language bindings. Here are some examples that will have the same behavior
|
||||
as the CLI example above and will subscribe to all topics on the firehose and
|
||||
print it to STDOUT.
|
||||
|
||||
|
||||
Python
|
||||
''''''
|
||||
.. code-block:: python
|
||||
|
||||
import paho.mqtt.client as mqtt
|
||||
|
||||
|
||||
def on_connect(client, userdata, flags, rc):
|
||||
print("Connected with result code " + str(rc))
|
||||
client.subscribe('#')
|
||||
|
||||
def on_message(client, userdata, msg):
|
||||
print(msg.topic+" "+str(msg.payload))
|
||||
|
||||
# Create a websockets client
|
||||
client = mqtt.Client()
|
||||
client.on_connect = on_connect
|
||||
client.on_message = on_message
|
||||
|
||||
# Connect to the firehose
|
||||
client.connect('firehose.openstack.org')
|
||||
# Listen forever
|
||||
client.loop_forever()
|
||||
|
||||
Haskell
|
||||
'''''''
|
||||
This requires the `mqtt-hs`_ library to be installed.
|
||||
|
||||
.. _mqtt-hs: https://hackage.haskell.org/package/mqtt-hs
|
||||
|
||||
.. code-block:: haskell
|
||||
|
||||
{-# Language DataKinds, OverloadedStrings #-}
|
||||
|
||||
module Subscribe where
|
||||
|
||||
import Control.Concurrent
|
||||
import Control.Concurrent.STM
|
||||
import Control.Monad (unless, forever)
|
||||
import System.Exit (exitFailure)
|
||||
import System.IO (hPutStrLn, stderr)
|
||||
|
||||
import qualified Network.MQTT as MQTT
|
||||
|
||||
topic :: MQTT.Topic
|
||||
topic = "#"
|
||||
|
||||
handleMsg :: MQTT.Message MQTT.PUBLISH -> IO ()
|
||||
handleMsg msg = do
|
||||
let t = MQTT.topic $ MQTT.body msg
|
||||
p = MQTT.payload $ MQTT.body msg
|
||||
print t
|
||||
print p
|
||||
|
||||
main :: IO ()
|
||||
main = do
|
||||
cmds <- MQTT.mkCommands
|
||||
pubChan <- newTChanIO
|
||||
let conf = (MQTT.defaultConfig cmds pubChan)
|
||||
{ MQTT.cHost = "firehose.openstack.org" }
|
||||
_ <- forkIO $ do
|
||||
qosGranted <- MQTT.subscribe conf [(topic, MQTT.Handshake)]
|
||||
forever $ atomically (readTChan pubChan) >>= handleMsg
|
||||
terminated <- MQTT.run conf
|
||||
print terminated
|
||||
|
||||
Go
|
||||
''
|
||||
.. code-block:: go
|
||||
|
||||
package main
|
||||
import (
|
||||
"fmt"
|
||||
MQTT "github.com/eclipse/paho.mqtt.golang"
|
||||
"os"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
func onMessageReceived(client MQTT.Client, msg MQTT.Message) {
|
||||
fmt.Printf("TOPIC: %s\n", msg.Topic())
|
||||
fmt.Printf("MSG: %s\n", msg.Payload())
|
||||
}
|
||||
func main() {
|
||||
hostname, _ := os.Hostname()
|
||||
opts := &MQTT.ClientOptions{
|
||||
ClientID: hostname+strconv.Itoa(time.Now().Second()),
|
||||
}
|
||||
opts.AddBroker("tcp://firehose.openstack.org:1883")
|
||||
opts.OnConnect = func(c MQTT.Client) {
|
||||
if token := c.Subscribe("#", 0, onMessageReceived); token.Wait() && token.Error() != nil {
|
||||
fmt.Println(token.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
client := MQTT.NewClient(opts)
|
||||
if token := client.Connect(); token.Wait() && token.Error() != nil {
|
||||
panic(token.Error())
|
||||
}
|
||||
for {
|
||||
time.Sleep(1 * time.Second)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Websocket Example
|
||||
-----------------
|
||||
In addition to using the raw MQTT protocol firehose.o.o provides a websocket
|
||||
interface on port 80 that MQTT traffic can go through. This is especially useful
|
||||
for web applications that intend to consume any events from MQTT. To see an
|
||||
example of this in action you can try: http://mitsuruog.github.io/what-mqtt/
|
||||
(the source is available here: https://github.com/mitsuruog/what-mqtt) and use
|
||||
that to subscribe to any topics on firehose.openstack.org.
|
||||
|
||||
Another advantage of using websockets over port 80 is that it's much more
|
||||
firewall friendly, especially in environments that are more locked down. If you
|
||||
would like to consume events from the firehose and are concerned about a
|
||||
firewall blocking your access, the websocket interface is a good choice.
|
||||
|
||||
You can also use the paho-mqtt python library to subscribe to mqtt over
|
||||
websockets fairly easily. For example this script will subscribe to all topics
|
||||
on the firehose and print it to STDOUT
|
||||
|
||||
.. code-block:: python
|
||||
:emphasize-lines: 12,17
|
||||
|
||||
import paho.mqtt.client as mqtt
|
||||
|
||||
|
||||
def on_connect(client, userdata, flags, rc):
|
||||
print("Connected with result code " + str(rc))
|
||||
client.subscribe('#')
|
||||
|
||||
def on_message(client, userdata, msg):
|
||||
print(msg.topic+" "+str(msg.payload))
|
||||
|
||||
# Create a websockets client
|
||||
client = mqtt.Client(transport="websockets")
|
||||
client.on_connect = on_connect
|
||||
client.on_message = on_message
|
||||
|
||||
# Connect to the firehose
|
||||
client.connect('firehose.openstack.org', port=80)
|
||||
# Listen forever
|
||||
client.loop_forever()
|
||||
|
||||
Using SSL/TLS
|
||||
-------------
|
||||
If you would like to connect to the firehose using ssl to encrypt the events you
|
||||
receive from MQTT you just need to connect with ssl enabled via either of the
|
||||
encrypted ports. If you'd like to verify the server ssl certificate when
|
||||
connecting you'll need to provide a CA bundle to use as most MQTT clients do
|
||||
not know how to use the system trusted CA bundle like most http clients.
|
||||
|
||||
To connect to the firehose and subscribe to all topics you can use the
|
||||
mosquitto CLI client::
|
||||
|
||||
mosquitto_sub --topic '#' -h firehose.openstack.org --cafile /etc/ca-certificates/extracted/tls-ca-bundle.pem -p 8883
|
||||
|
||||
You can use python:
|
||||
|
||||
.. code-block:: python
|
||||
:emphasize-lines: 15,20
|
||||
|
||||
import paho.mqtt.client as mqtt
|
||||
|
||||
|
||||
def on_connect(client, userdata, flags, rc):
|
||||
print("Connected with result code " + str(rc))
|
||||
client.subscribe('#')
|
||||
|
||||
|
||||
def on_message(client, userdata, msg):
|
||||
print(msg.topic+" "+str(msg.payload))
|
||||
|
||||
|
||||
# Create an SSL encrypted websockets client
|
||||
client = mqtt.Client()
|
||||
client.tls_set(ca_certs='/etc/ca-certificates/extracted/tls-ca-bundle.pem')
|
||||
client.on_connect = on_connect
|
||||
client.on_message = on_message
|
||||
|
||||
# Connect to the firehose
|
||||
client.connect('firehose.openstack.org', port=8883)
|
||||
client.loop_forever()
|
||||
|
||||
|
||||
Or with ruby:
|
||||
|
||||
.. code-block:: ruby
|
||||
:emphasize-lines: 6,7,8
|
||||
|
||||
require 'rubygems'
|
||||
require 'mqtt'
|
||||
|
||||
client = MQTT::Client.new
|
||||
client.host = 'firehose.openstack.org'
|
||||
client.ssl = true
|
||||
client.cert_file = '/etc/ca-certificates/extracted/tls-ca-bundle.pem'
|
||||
client.port = 8883
|
||||
client.connect()
|
||||
client.subscribe('#')
|
||||
|
||||
client.get do |topic,message|
|
||||
puts message
|
||||
end
|
||||
|
||||
Example Use Cases
|
||||
=================
|
||||
|
||||
Event Notifications
|
||||
-------------------
|
||||
|
||||
A common use case for the event bus is to get a notification when an event
|
||||
occurs. There is an open source tool, `mqttwarn`_ that makes setting this up
|
||||
off the firehose (or any other mqtt broker) very straightforward.
|
||||
|
||||
.. _mqttwarn: https://github.com/jpmens/mqttwarn
|
||||
|
||||
You can use mqttwarn to setup custom notifications to a large number of tools
|
||||
and services. (both local and remote). You can read the full docs on how to
|
||||
configure and use mqttwarn at https://github.com/jpmens/mqttwarn/wiki and
|
||||
https://github.com/jpmens/mqttwarn/blob/master/README.md
|
||||
|
||||
|
||||
IMAP and MX
|
||||
===========
|
||||
|
||||
We're using Cyrus as an IMAP server in order to consume launchpad bug
|
||||
events via email. The configuration of the admin password account and
|
||||
creation of the lpmqtt user for Cyrus were completed using the
|
||||
following::
|
||||
|
||||
$ sudo saslpasswd2 cyrus
|
||||
$ cyradm --user=cyrus --server=localhost
|
||||
Password:
|
||||
localhost> create user.lpmqtt
|
||||
|
||||
An MX record has also been set up to point to the firehose server.
|
||||
@ -1,434 +0,0 @@
|
||||
:title: Firehose Schema
|
||||
|
||||
.. _firehose_schema:
|
||||
|
||||
Firehose Schema
|
||||
###############
|
||||
|
||||
This attempts to document the topic and payload schema for all the services
|
||||
reporting to the firehose. However since much of what is reported to firehose
|
||||
is dynamically generated it is possible this document misses a case.
|
||||
|
||||
Gerrit
|
||||
======
|
||||
|
||||
Messages on firehose for gerrit are generated using the `germqtt`_ project. For
|
||||
the most part these are basically identical to what gerrit returns on it's
|
||||
native event stream except over MQTT.
|
||||
|
||||
.. _germqtt: http://opendev.org/opendev/germqtt/
|
||||
|
||||
Topics
|
||||
------
|
||||
|
||||
The topics for gerrit are generated dynamically. However, they follow a fairly
|
||||
straightforward pattern. The basic formula for this is::
|
||||
|
||||
gerrit/<git namespace>/<repo name>/<gerrit event>
|
||||
|
||||
So for example a typical topic would be::
|
||||
|
||||
gerrit/openstack/nova/comment-added
|
||||
|
||||
The ``git namespace`` and ``repo name`` are pretty self explanatory and are just
|
||||
from the git repository the change in gerrit is for. The event is defined in the gerrit event stream. You can see the full reference for topics in the Gerrit
|
||||
docs for `Gerrit events`_. However, for simplicity the possible values are:
|
||||
|
||||
* change-abandoned
|
||||
* change-merged
|
||||
* change-restored
|
||||
* comment-added
|
||||
* draft-published
|
||||
* hashtags-changed
|
||||
* merge-failed
|
||||
* patchset-created
|
||||
* ref-updated
|
||||
* reviewer-added
|
||||
* topic-changed
|
||||
|
||||
Payload
|
||||
-------
|
||||
The payload for gerrit messages are basically the same JSON that gets returned
|
||||
by gerrit's event stream command. Instead of repeating the entire gerrit schema
|
||||
doc here just refer to gerrit's docs on the `JSON payload`_ which documents the
|
||||
contents of each JSON object and refer to the doc on `Gerrit events`_ for which
|
||||
JSON objects are included with which event type.
|
||||
|
||||
.. _JSON payload: https://review.opendev.org/Documentation/json.html
|
||||
.. _Gerrit events: https://review.opendev.org/Documentation/cmd-stream-events.html#events
|
||||
|
||||
Launchpad
|
||||
=========
|
||||
The messages sent to firehose for launchpad are generated using `lpmqtt`_
|
||||
|
||||
.. _lpmqtt: http://opendev.org/opendev/lpmqtt/
|
||||
|
||||
Topics
|
||||
------
|
||||
|
||||
The topics for lpmqtt follow a pretty simple formula::
|
||||
|
||||
launchpad/<project>/<event type>/<bug number>
|
||||
|
||||
the ``project`` is the launchpad project name, ``event type`` will always be
|
||||
"bug" (or not present). The intent of this was to be "bug" or "blueprint", but
|
||||
due to limitations in launchpad getting notifications from blueprints is not
|
||||
possible. The flexibility was left in the schema just in case this ever changes.
|
||||
The ``bug number`` is obviously the bug number from launchpad.
|
||||
|
||||
It's also worth noting that only the base topic is a guaranteed field. Depending
|
||||
on the notification email from launchpad some of the other fields may not be
|
||||
present. In those cases the topic will be populated left to right until a
|
||||
missing field is encountered.
|
||||
|
||||
Payload
|
||||
-------
|
||||
|
||||
The payload of messages is dynamically generated and dependent on the
|
||||
notification received from launchpad, and launchpad isn't always consistent in
|
||||
what fields are present in those notifications.
|
||||
|
||||
However, for bug event types there is a standard format. The fields which
|
||||
are always present for bugs (which should normally be the only message for
|
||||
firehose) are:
|
||||
|
||||
* commenters
|
||||
* bug-reporter
|
||||
* bug-modifier
|
||||
* bug-number
|
||||
* event-type
|
||||
|
||||
The remaining fields are dynamic and depend on launchpad. An example message
|
||||
payload (with the body trimmed) for a bug is::
|
||||
|
||||
{
|
||||
"status": "Triaged",
|
||||
"project": "octavia",
|
||||
"assignee": "email@fakedomain.com",
|
||||
"bug-reporter": "Full name (username)",
|
||||
"event-type": "bug",
|
||||
"bug-number": "1680938",
|
||||
"commenters": ["username"]
|
||||
"tags": ["rfe"],
|
||||
"importance": "Medium",
|
||||
"bug-modifier": "Full Name (username)",
|
||||
"body": "notification body, often is just bug comment or summary",
|
||||
}
|
||||
|
||||
Subunit Workers
|
||||
===============
|
||||
|
||||
The messages for the subunit workers are generated directly in the
|
||||
`subunit gearman worker scripts`_.
|
||||
|
||||
.. _subunit gearman worker scripts: https://opendev.org/opendev/puppet-subunit2sql/src/branch/master/files/subunit-gearman-worker.py
|
||||
|
||||
Topics
|
||||
------
|
||||
|
||||
The topics for the subunit workers follow a simple pattern::
|
||||
|
||||
gearman-subunit/<worker hostname>/<git namespace/<repo name>/<change number>
|
||||
|
||||
Where ``worker hostname`` is the host which processed the subunit file, as
|
||||
of right now there are 2, subunit-worker01 and subunit-worker02, but there may
|
||||
be more (or fewer) in the future. The ``git namespace`` and ``repo name`` are
|
||||
pretty self explanatory, and are just for the git repo under test that the
|
||||
subunit was emitted from. ``change number`` is the gerrit change number for the
|
||||
job that launched the tests the subunit is for.
|
||||
|
||||
Payload
|
||||
-------
|
||||
The payload for the messages from the subunit workers is pretty straightforward
|
||||
json that contains 3 fields: ``status``, ``build_uuid``, and ``source_url``.
|
||||
|
||||
An example is::
|
||||
|
||||
{
|
||||
'status': 'success',
|
||||
'build_uuid': '45f7c1ddbfd74c6aba94662623bd61b8'
|
||||
'source_url': 'A url',
|
||||
}
|
||||
|
||||
Ansible
|
||||
=======
|
||||
|
||||
We have mqtt events emitted from ansible being run on :ref:`bridge`.
|
||||
These events are generated using a `MQTT Ansible Callback Plugin`_.
|
||||
|
||||
.. _MQTT Ansible Callback Plugin: https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/puppetmaster/mqtt.py
|
||||
|
||||
Topics
|
||||
------
|
||||
|
||||
The topics for ansible are a bit more involved than some of the other services
|
||||
publishing to firehose. It depends on the type of event that ansible just
|
||||
finished. There are 3 categories of events which have slightly different topic
|
||||
formulas (and payloads).
|
||||
|
||||
Playbook Events
|
||||
'''''''''''''''
|
||||
Whenever a playbook action occurs the callback plugin will emit an event for
|
||||
it. The topics for playbook events fall into this pattern::
|
||||
|
||||
ansible/playbook/<playbook uuid>/action/<playbook action>/<status>
|
||||
|
||||
``playbook uuid`` is pretty self explanatory here, it's the uuid ansible uses
|
||||
to uniquely identify the playbook being run. ``playbook action`` is the action
|
||||
that the event is for, this is either going to be ``start`` or ``finish``.
|
||||
``status`` is only set on ``finish`` and will be one of the following:
|
||||
|
||||
* ``OK``
|
||||
* ``FAILED``
|
||||
|
||||
to indicate whether the playbook successfully executed or not.
|
||||
|
||||
Playbook Stats Events
|
||||
'''''''''''''''''''''
|
||||
|
||||
At the end of a playbook these events are emitted for each host that tasks were
|
||||
run on. The topics for these events fall into the following pattern::
|
||||
|
||||
ansible/playbook/<playbook uuid>/stats/<hostname>
|
||||
|
||||
In this case ``playbook uuid`` is the same as above and the internal ansible
|
||||
unique playbook identifier. ``hostname`` here is the host that ansible was
|
||||
running tasks on as part of the playbook.
|
||||
|
||||
Task Events
|
||||
'''''''''''
|
||||
|
||||
At the end of each individual task the callback plugin will emit an event. Those
|
||||
events' topics fall into the following pattern::
|
||||
|
||||
ansible/playbook/<playbook uuid>/task/<hostname>/<status>
|
||||
|
||||
``playbook uuid`` is the same as in the previous 2 event types. ``hostname`` is
|
||||
the hostname the task was executed on. ``status`` is the result of the task
|
||||
and will be one of the following:
|
||||
|
||||
* ``OK``
|
||||
* ``FAILED``
|
||||
* ``UNREACHABLE``
|
||||
|
||||
Payload
|
||||
-------
|
||||
|
||||
Just as with the topics the message payloads depend on the event type. Each
|
||||
event uses a JSON payload with slightly different fields.
|
||||
|
||||
Playbook Events
|
||||
'''''''''''''''
|
||||
|
||||
For playbook events the payload falls into this schema on playbook starts::
|
||||
|
||||
{
|
||||
"status": "OK",
|
||||
"host": <hostname>
|
||||
"session": <session id>,
|
||||
"playbook_name": <playbook name>,
|
||||
"playbook_id": <playbook uuid>,
|
||||
"ansible_type": "start",
|
||||
}
|
||||
|
||||
When a playbook finishes the payload is slightly smaller and the schema is::
|
||||
|
||||
{
|
||||
"playbook_id": <playbook uuid>,
|
||||
"playbook_name": <playbook name>,
|
||||
"status": <status>,
|
||||
}
|
||||
|
||||
In both cases ``playbook uuid`` is the same field from the topic.
|
||||
``playbook name`` is the human readable name for the playbook. If one is
|
||||
set in the playbook this will be that. ``status`` will be whether the
|
||||
playbook was successfully executed or not. It will always be ``OK`` on starts
|
||||
(otherwise the event isn't emitted) but on failures, just like in the topic,
|
||||
this will be one of the following:
|
||||
|
||||
* ``OK``
|
||||
* ``FAILED``
|
||||
|
||||
``session id`` is a UUID generated by the callback plugin to uniquely identify
|
||||
the execution of the playbook. ``hostname`` is the hostname where the ansible
|
||||
playbook was launched. (which is not necessarily where tasks are being run)
|
||||
|
||||
|
||||
An example of this from the system is for a start event::
|
||||
|
||||
{
|
||||
"status": "OK",
|
||||
"playbook_name": "localhost:!disabled",
|
||||
"ansible_type": "start",
|
||||
"host": "puppetmaster.openstack.org",
|
||||
"session": "14d6e568-2c75-11e7-bd24-bc764e048db9",
|
||||
"playbook_id": "5a95e9da-8d33-4dbb-a8b3-a77affc065d0"
|
||||
}
|
||||
|
||||
and for a finish::
|
||||
|
||||
{
|
||||
"status": "FAILED",
|
||||
"playbook_name": "compute*.ic.openstack.org:!disabled",
|
||||
"playbook_id": "b259ac6d-6cb5-4403-bb8d-0ff2131c3d7a"
|
||||
}
|
||||
|
||||
|
||||
Playbook Stats Events
|
||||
'''''''''''''''''''''
|
||||
|
||||
The schema for stats events is::
|
||||
|
||||
{
|
||||
"host": <hostname>,
|
||||
"ansible_host": <execute hostname>,
|
||||
"playbook_id": <playbook uuid>,
|
||||
"playbook_name": <playbook name>,
|
||||
"stats": {
|
||||
"unreachable": int,
|
||||
"skipped": int,
|
||||
"ok": int,
|
||||
"changed": int,
|
||||
"failures": int,
|
||||
}
|
||||
}
|
||||
|
||||
``playbook uuid`` is the same field from the topic. ``playbook name`` is the
|
||||
human readable name for the playbook. If one is set in the playbook this will be
|
||||
that. ``execute hostname`` is the hostname where the tasks were being executed,
|
||||
while ``hostname`` is the hostname where ansible launched the playbook. The
|
||||
``stats`` subdict contains the task status counts where the key is the tasks
|
||||
statuses.
|
||||
|
||||
An example from the running system is::
|
||||
|
||||
{
|
||||
"playbook_name": "compute*.ic.openstack.org:!disabled",
|
||||
"host": "puppetmaster.openstack.org",
|
||||
"stats": {
|
||||
"unreachable": 0,
|
||||
"skipped": 5,
|
||||
"ok": 13,
|
||||
"changed": 1,
|
||||
"failures": 0
|
||||
},
|
||||
"playbook_id": "b259ac6d-6cb5-4403-bb8d-0ff2131c3d7a",
|
||||
"ansible_host": "controller00.vanilla.ic.openstack.org"
|
||||
}
|
||||
|
||||
|
||||
Task Events
|
||||
'''''''''''
|
||||
|
||||
The schema for tasks events is::
|
||||
|
||||
{
|
||||
"status": <status>,
|
||||
"host": <hostname>,
|
||||
"ansible_host": <execute hostname>,
|
||||
"session": <session id>,
|
||||
"ansible_type": "task",
|
||||
"playbook_name": <playbook name>,
|
||||
"playbook_id": <playbook uuid>,
|
||||
"ansible_task": <task name>,
|
||||
"ansible_result": <ansible result>
|
||||
}
|
||||
|
||||
``playbook uuid`` is the same field from the topic. ``playbook name`` is the
|
||||
human readable name for the playbook. If one is set in the playbook this will be
|
||||
that. ``execute hostname`` is the hostname where the tasks were being executed,
|
||||
while ``hostname`` is the hostname where ansible launched the playbook. ``task
|
||||
name``, like the name implies, is the human readable name of the task executed.
|
||||
If one was specified in the playbook that will be the value. ``status`` is the
|
||||
result of the task and will be one of the following:
|
||||
|
||||
* ``OK``
|
||||
* ``FAILED``
|
||||
* ``UNREACHABLE``
|
||||
|
||||
``session id`` is a UUID generated by the callback plugin to uniquely identify
|
||||
the execution of the playbook.
|
||||
|
||||
``ansible result`` is a free form subdict that comes directly from ansible to
|
||||
completely describe the task that just finished. The structure here is fully
|
||||
dependent on ansible internals and the way that the task was invoked in the
|
||||
playbook. Note, that sometimes this can be quite large in size depending on the
|
||||
task and whether facts were enabled or not.
|
||||
|
||||
An example of a task event from the running system is::
|
||||
|
||||
{
|
||||
"status": "OK",
|
||||
"host": "puppetmaster.openstack.org",
|
||||
"session": "092aa3fa-2c73-11e7-bd24-bc764e048db9",
|
||||
"playbook_name": "compute*.ic.openstack.org:!disabled",
|
||||
"ansible_result": {
|
||||
"_ansible_parsed": true,
|
||||
"_ansible_no_log": false,
|
||||
"stdout": "",
|
||||
"changed": false,
|
||||
"stderr": "",
|
||||
"rc": 0,
|
||||
"invocation": {
|
||||
"module_name": "puppet",
|
||||
"module_args": {
|
||||
"logdest": "syslog",
|
||||
"execute": null,
|
||||
"facter_basename": "ansible",
|
||||
"tags": null,
|
||||
"puppetmaster": null,
|
||||
"show_diff": false,
|
||||
"certname": null,
|
||||
"manifest": "/opt/system-config/manifests/site.pp",
|
||||
"environment": "production",
|
||||
"debug": false,
|
||||
"noop": false,
|
||||
"timeout": "30m",
|
||||
"facts": null
|
||||
}
|
||||
},
|
||||
"stdout_lines": []
|
||||
},
|
||||
"ansible_type": "task",
|
||||
"ansible_task": "TASK: puppet : run puppet",
|
||||
"playbook_id": "b259ac6d-6cb5-4403-bb8d-0ff2131c3d7a",
|
||||
"ansible_host": "compute014.chocolate.ic.openstack.org"
|
||||
}
|
||||
|
||||
|
||||
Logstash Workers
|
||||
================
|
||||
|
||||
The messages for the subunit workers are generated directly in the
|
||||
`logstash gearman worker scripts`_.
|
||||
|
||||
.. _logstash gearman worker scripts: https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-worker.py
|
||||
|
||||
Topics
|
||||
------
|
||||
|
||||
The topics for the subunit workers follow a simple pattern::
|
||||
|
||||
gearman-logstash/<worker hostname>/<git namespace>/<repo name>/<change number>/<action>
|
||||
|
||||
Where ``worker hostname`` is the host which processed the log file. The
|
||||
``git namespace`` and ``repo name`` are pretty self explanatory, and are just
|
||||
for the git repo under test that the log was generated. ``change number`` is
|
||||
the gerrit change number for the job that launched the tests the subunit is for.
|
||||
In the case of periodic or post queue jobs, this will either say ``periodic`` or
|
||||
``post`` because there isn't an associated change number. The action field is
|
||||
the phase of the log processing that just completed. Right now the only
|
||||
possible value is ``retrieve_logs`` but there may be others in the future.
|
||||
|
||||
Payload
|
||||
-------
|
||||
The payload for the messages from the logstash workers is pretty straightforward
|
||||
json that contains 3 fields: ``status``, ``build_uuid``, and ``source_url``.
|
||||
|
||||
An example is::
|
||||
|
||||
{
|
||||
'status': 'success',
|
||||
'build_uuid': '45f7c1ddbfd74c6aba94662623bd61b8'
|
||||
'source_url': 'A url',
|
||||
}
|
||||
@ -37,8 +37,6 @@ Major Systems
|
||||
refstack
|
||||
codesearch
|
||||
signing
|
||||
firehose
|
||||
firehose_schema
|
||||
github
|
||||
activity
|
||||
asterisk
|
||||
|
||||
@ -222,7 +222,6 @@ cacti_hosts:
|
||||
- elasticsearch07.openstack.org
|
||||
- ethercalc02.openstack.org
|
||||
- etherpad01.opendev.org
|
||||
- firehose01.openstack.org
|
||||
- gitea-lb01.opendev.org
|
||||
- gitea01.opendev.org
|
||||
- gitea02.opendev.org
|
||||
|
||||
@ -154,13 +154,6 @@ all:
|
||||
region_name: DFW
|
||||
public_v4: 104.130.124.120
|
||||
public_v6: 2001:4800:7818:104:be76:4eff:fe02:b0ff
|
||||
firehose01.openstack.org:
|
||||
ansible_host: 104.130.155.115
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 104.130.155.115
|
||||
public_v6: 2001:4800:7818:103:be76:4eff:fe04:40aa
|
||||
gitea-lb01.opendev.org:
|
||||
ansible_host: 38.108.68.124
|
||||
location:
|
||||
|
||||
@ -1,25 +0,0 @@
|
||||
exim_local_domains: "@:firehose.openstack.org"
|
||||
# TODO(jeblair): have the cyrus router check to see if there is a
|
||||
# cyrus account.
|
||||
exim_routers:
|
||||
- dnslookup: '{{ exim_dnslookup_router }}'
|
||||
- system_aliases: '{{ exim_system_aliases_router }}'
|
||||
- cyrus: |
|
||||
driver = accept
|
||||
domains = +local_domains
|
||||
local_part_suffix = +*
|
||||
local_part_suffix_optional
|
||||
transport = cyrus
|
||||
- localuser: '{{ exim_localuser_router }}'
|
||||
exim_transports:
|
||||
- cyrus: |
|
||||
driver = lmtp
|
||||
socket = /var/run/cyrus/socket/lmtp
|
||||
user = cyrus
|
||||
batch_max = 35
|
||||
iptables_extra_public_tcp_ports:
|
||||
- 25
|
||||
- 80
|
||||
- 443
|
||||
- 1883
|
||||
- 8883
|
||||
@ -7,7 +7,6 @@ iptables_extra_allowed_hosts:
|
||||
protocol: udp
|
||||
|
||||
iptables_extra_allowed_groups:
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'firehose'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'logstash'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'}
|
||||
|
||||
@ -7,7 +7,6 @@ iptables_extra_allowed_hosts:
|
||||
protocol: udp
|
||||
|
||||
iptables_extra_allowed_groups:
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'firehose'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'logstash'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'}
|
||||
|
||||
@ -55,7 +55,6 @@ groups:
|
||||
elasticsearch: elasticsearch[0-9]*.open*.org
|
||||
ethercalc: ethercalc*.open*.org
|
||||
etherpad: etherpad[0-9]*.open*.org
|
||||
firehose: firehose[0-9]*.open*.org
|
||||
gerrit:
|
||||
- review[0-9]*.open*.org
|
||||
gitea:
|
||||
@ -138,7 +137,6 @@ groups:
|
||||
- eavesdrop[0-9]*.open*.org
|
||||
- elasticsearch[0-9]*.open*.org
|
||||
- ethercalc[0-9]*.open*.org
|
||||
- firehose[0-9]*.open*.org
|
||||
- health[0-9]*.openstack.org
|
||||
- lists*.katacontainers.io
|
||||
- lists*.open*.org
|
||||
@ -166,7 +164,6 @@ groups:
|
||||
- eavesdrop[0-9]*.open*.org
|
||||
- elasticsearch[0-9]*.open*.org
|
||||
- ethercalc[0-9]*.open*.org
|
||||
- firehose[0-9]*.open*.org
|
||||
- health[0-9]*.openstack.org
|
||||
- lists*.katacontainers.io
|
||||
- lists*.open*.org
|
||||
|
||||
@ -175,24 +175,6 @@ node /^elasticsearch\d+\.open.*\.org$/ {
|
||||
}
|
||||
}
|
||||
|
||||
# Node-OS: xenial
|
||||
node /^firehose\d+\.open.*\.org$/ {
|
||||
class { 'openstack_project::server': }
|
||||
class { 'openstack_project::firehose':
|
||||
gerrit_ssh_host_key => hiera('gerrit_ssh_rsa_pubkey_contents'),
|
||||
gerrit_public_key => hiera('germqtt_gerrit_ssh_public_key'),
|
||||
gerrit_private_key => hiera('germqtt_gerrit_ssh_private_key'),
|
||||
mqtt_password => hiera('mqtt_service_user_password'),
|
||||
ca_file => hiera('mosquitto_tls_ca_file'),
|
||||
cert_file => hiera('mosquitto_tls_server_cert_file'),
|
||||
key_file => hiera('mosquitto_tls_server_key_file'),
|
||||
imap_hostname => hiera('lpmqtt_imap_server'),
|
||||
imap_username => hiera('lpmqtt_imap_username'),
|
||||
imap_password => hiera('lpmqtt_imap_password'),
|
||||
statsd_host => 'graphite.opendev.org',
|
||||
}
|
||||
}
|
||||
|
||||
# A machine to run Storyboard
|
||||
# Node-OS: xenial
|
||||
node /^storyboard\d+\.opendev\.org$/ {
|
||||
|
||||
@ -9,7 +9,6 @@ docs.starlingx.io 443
|
||||
ethercalc.openstack.org 443
|
||||
etherpad.openstack.org 443
|
||||
files.openstack.org 443
|
||||
firehose.openstack.org 8883
|
||||
git.airshipit.org 443
|
||||
git.openstack.org 443
|
||||
git.starlingx.io 443
|
||||
|
||||
@ -1,87 +0,0 @@
|
||||
# Copyright 2016 Hewlett-Packard Development Company, L.P.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
# firehose glue class.
|
||||
#
|
||||
class openstack_project::firehose (
|
||||
$gerrit_username = 'germqtt',
|
||||
$gerrit_public_key,
|
||||
$gerrit_private_key,
|
||||
$gerrit_ssh_host_key,
|
||||
$imap_username,
|
||||
$imap_hostname,
|
||||
$imap_password,
|
||||
$mqtt_hostname = 'firehose.openstack.org',
|
||||
$mqtt_password,
|
||||
$mqtt_username = 'infra',
|
||||
$statsd_host,
|
||||
$ca_file,
|
||||
$cert_file,
|
||||
$key_file,
|
||||
) {
|
||||
include mosquitto
|
||||
class {'mosquitto::server':
|
||||
infra_service_username => $mqtt_username,
|
||||
infra_service_password => $mqtt_password,
|
||||
enable_tls => true,
|
||||
enable_tls_websocket => true,
|
||||
ca_file => $ca_file,
|
||||
cert_file => $cert_file,
|
||||
key_file => $key_file,
|
||||
websocket_tls_port => 443,
|
||||
}
|
||||
|
||||
include germqtt
|
||||
class {'germqtt::server':
|
||||
gerrit_username => $gerrit_username,
|
||||
gerrit_public_key => $gerrit_public_key,
|
||||
gerrit_private_key => $gerrit_private_key,
|
||||
gerrit_ssh_host_key => $gerrit_ssh_host_key,
|
||||
mqtt_username => $mqtt_username,
|
||||
mqtt_password => $mqtt_password,
|
||||
}
|
||||
|
||||
package {'cyrus-imapd':
|
||||
ensure => latest,
|
||||
}
|
||||
|
||||
package {'sasl2-bin':
|
||||
ensure => latest,
|
||||
}
|
||||
|
||||
package {'cyrus-admin':
|
||||
ensure => latest,
|
||||
}
|
||||
|
||||
service {'cyrus-imapd':
|
||||
ensure => running,
|
||||
}
|
||||
|
||||
include lpmqtt
|
||||
class {'lpmqtt::server':
|
||||
mqtt_username => $mqtt_username,
|
||||
mqtt_password => $mqtt_password,
|
||||
imap_hostname => $imap_hostname,
|
||||
imap_username => $imap_username,
|
||||
imap_password => $imap_password,
|
||||
imap_use_ssl => false,
|
||||
imap_delete_old => true,
|
||||
}
|
||||
|
||||
include mqtt_statsd
|
||||
class {'mqtt_statsd::server':
|
||||
mqtt_hostname => $mqtt_hostname,
|
||||
statsd_hostname => $statsd_host,
|
||||
}
|
||||
}
|
||||
@ -2,7 +2,6 @@ letsencrypt_certcheck_additional_domains:
|
||||
- ask.openstack.org 443
|
||||
- ethercalc.openstack.org 443
|
||||
- etherpad.openstack.org 443
|
||||
- firehose.openstack.org 8883
|
||||
- git.openstack.org 443
|
||||
- openstackid-dev.openstack.org 443
|
||||
- openstackid.org 443
|
||||
|
||||
@ -13,11 +13,6 @@ results:
|
||||
- afs-client
|
||||
- kerberos-client
|
||||
|
||||
firehose01.openstack.org:
|
||||
- firehose
|
||||
- puppet
|
||||
- puppet4
|
||||
|
||||
graphite02.opendev.org:
|
||||
- graphite
|
||||
- letsencrypt
|
||||
|
||||
Loading…
Reference in New Issue
Block a user