Firehose: A unified message bus for Infra services

This commit adds a spec for implementing a unified Infra message
bus, the "firehose."

Change-Id: I6295aec6025600a9ef02de5249a3a5c834dcba8c
Story: #2000643

doc/source/index.rst

@@ -42,6 +42,7 @@ permits.
    specs/deploy-ci-dashboard
    specs/deploy-stackviz
    specs/doc-publishing
+   specs/firehose
    specs/jenkins-job-builder_2.0.0-api-changes
    specs/nodepool-launch-workers
    specs/nodepool-workers

specs/firehose.rst

@@ -0,0 +1,165 @@
+::
+  This work is licensed under a Creative Commons Attribution 3.0
+  Unported License.
+  http://creativecommons.org/licenses/by/3.0/legalcode
+
+==================================================
+Firehose: A unified message bus for Infra services
+==================================================
+
+https://storyboard.openstack.org/#!/story/2000643
+
+Using MQTT start running a unified message bus that community infrastructure
+services can emit events on. This will enable efficient external consumption of
+infrastructure events for experimentation and rapid innovation of tooling
+within the community.
+
+Problem Description
+===================
+
+There are a lot of services running in openstack-infra doing lots of different
+things all the time. Following events generated by these services can be
+tricky, and we don't have a consistent solution to making arbitrary state
+change details externally visible (so we fall back to relying on those with
+access to the machines letting us know what operations are being performed).
+What event data does exist is inconsistently presented in a variety of places
+over different protocols and requires significant effort to correlate
+cross-service.
+
+
+Proposed Change
+===============
+
+The goal of this spec is to create a unified message bus that Infra services
+can publish event streams to. This will enable external public consumption of
+Infra events in a consistent manner.
+
+To accomplish this we will deploy a new server and run the mosquitto_ MQTT_
+broker on it. MQTT protocol was designed in the late '90s to be lightweight
+pub-sub protocol that would work with connections to remote locations that
+needed a "small code footprint" and where there was limited network bandwith.
+More recently this has made it quite popular in the IoT space. However, it is
+also a good fit for the distributed cloud model that a lot of Infra services
+use. We will also deploy mosquitto using websockets_ this should also provide
+a benefit to international community members where things like the Gerrit event
+stream are problematic to access over Gerrit's SSH port.
+
+.. _mosquitto: http://mosquitto.org/
+.. _MQTT: http://mqtt.org/
+.. _websockets: http://jpmens.net/2014/07/03/the-mosquitto-mqtt-broker-gets-websockets-support/
+
+That will provide the base infrastructure for publishing and subscribing to
+events but nothing will be using MQTT after that. To start the event reporting
+we'll leverage germqtt_ (and make it an Infra project). This will be the
+initial usage for the unified message bus and make the Gerrit event stream
+available over MQTT. While the scope of this spec ends with getting Gerrit into
+MQTT, once we have an example we can more easily add more sources reporting
+directly into MQTT.
+
+.. _germqtt: https://github.com/mtreinish/germqtt
+
+Alternatives
+------------
+
+Use fedmsg
+^^^^^^^^^^
+The fedora infrastructure team had a similar need/desire to have a unified
+message bus and they started the fedmsg_ project to address this need.
+The primary issue with fedmsg is that it depends on AMQP or ZMQ both of which
+we have had previous issues with. The choice to go with MQTT was partially a
+reaction to that.
+
+.. _fedmsg: http://www.fedmsg.com/en/latest/
+
+
+Use a different mqtt broker
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+There are several open source MQTT brokers besides Mosquitto out there
+including ones like RabbitMQ_, VerneMQ_, and mosca_.
+
+.. _RabbitMQ: http://www.rabbitmq.com/blog/2012/09/12/mqtt-adapter/
+.. _VerneMQ: https://github.com/erlio/vernemq
+.. _mosca: https://github.com/mcollina/mosca
+
+Mosquitto was chosen because it's a lightweight C program which will need less
+resources to run it. It also should be simpler to deploy since we don't have to
+worry about deploying a JavaVM, nodejs, or erlang (which seems to be quite
+popular for writing message brokers).
+
+
+Implementation
+==============
+
+Assignee(s)
+-----------
+
+Primary assignee:
+
+ * mtreinish
+ * fungi
+
+
+Gerrit Topic
+------------
+
+Use Gerrit topic "firehose" for all patches related to this spec.
+
+.. code-block:: bash
+
+    git-review -t firehose
+
+Work Items
+----------
+
+ #. Write puppet-mosquitto module
+ #. Deploy firehose01 server with puppet-mosquitto module and a
+     firehose.openstack.org CNAME to it
+ #. Add germqtt to openstack-infra
+ #. Write puppet-germqtt
+ #. Deploy puppet-germqtt on firehose01
+
+Repositories
+------------
+
+2 New repos will be created in the initial implemention, puppet-mosquitto and
+puppet-germqtt.
+
+Servers
+-------
+
+1 new server will need to be created for running mosquitto, firehose01. Since
+mosquitto and MQTT is fairly lightweight we probably can get away with a
+smaller server (2GB flavor should be plenty).
+
+DNS Entries
+-----------
+
+Forward/reverse DNS entries will be added for firehose01.openstack.org, as well
+as a CNAME of firehose.openstack.org.
+
+Documentation
+-------------
+
+Initial service documentation will be added to the
+openstack-infra/system-config repo as usual. An availability announcement will
+be sent to the openstack-dev mailing list.
+
+Security
+--------
+
+Sensitive data should not be published over the message bus. It is public and
+anonymously accessible.
+
+Testing
+-------
+
+The new Puppet modules will run our usual Puppet test and style jobs, and the
+germqtt repo will have typical Python unit testing and style checks as well.
+
+Dependencies
+============
+
+- This will be the first use of germqtt in infra (which will be added as an
+  infra project)
+- 2 new puppet modules will be created, 1 to deploy mosquitto and the other to
+  run germqtt