From 05b0c5313340e054e8786439704282e999a8a053 Mon Sep 17 00:00:00 2001 From: Matthew Treinish Date: Mon, 13 Jun 2016 12:58:16 -0400 Subject: [PATCH] Firehose: A unified message bus for Infra services This commit adds a spec for implementing a unified Infra message bus, the "firehose." Change-Id: I6295aec6025600a9ef02de5249a3a5c834dcba8c Story: #2000643 --- doc/source/index.rst | 1 + specs/firehose.rst | 165 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 166 insertions(+) create mode 100644 specs/firehose.rst diff --git a/doc/source/index.rst b/doc/source/index.rst index 3115e56..2a05315 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -42,6 +42,7 @@ permits. specs/deploy-ci-dashboard specs/deploy-stackviz specs/doc-publishing + specs/firehose specs/jenkins-job-builder_2.0.0-api-changes specs/nodepool-launch-workers specs/nodepool-workers diff --git a/specs/firehose.rst b/specs/firehose.rst new file mode 100644 index 0000000..f3800bc --- /dev/null +++ b/specs/firehose.rst @@ -0,0 +1,165 @@ +:: + This work is licensed under a Creative Commons Attribution 3.0 + Unported License. + http://creativecommons.org/licenses/by/3.0/legalcode + +================================================== +Firehose: A unified message bus for Infra services +================================================== + +https://storyboard.openstack.org/#!/story/2000643 + +Using MQTT start running a unified message bus that community infrastructure +services can emit events on. This will enable efficient external consumption of +infrastructure events for experimentation and rapid innovation of tooling +within the community. + +Problem Description +=================== + +There are a lot of services running in openstack-infra doing lots of different +things all the time. Following events generated by these services can be +tricky, and we don't have a consistent solution to making arbitrary state +change details externally visible (so we fall back to relying on those with +access to the machines letting us know what operations are being performed). +What event data does exist is inconsistently presented in a variety of places +over different protocols and requires significant effort to correlate +cross-service. + + +Proposed Change +=============== + +The goal of this spec is to create a unified message bus that Infra services +can publish event streams to. This will enable external public consumption of +Infra events in a consistent manner. + +To accomplish this we will deploy a new server and run the mosquitto_ MQTT_ +broker on it. MQTT protocol was designed in the late '90s to be lightweight +pub-sub protocol that would work with connections to remote locations that +needed a "small code footprint" and where there was limited network bandwith. +More recently this has made it quite popular in the IoT space. However, it is +also a good fit for the distributed cloud model that a lot of Infra services +use. We will also deploy mosquitto using websockets_ this should also provide +a benefit to international community members where things like the Gerrit event +stream are problematic to access over Gerrit's SSH port. + +.. _mosquitto: http://mosquitto.org/ +.. _MQTT: http://mqtt.org/ +.. _websockets: http://jpmens.net/2014/07/03/the-mosquitto-mqtt-broker-gets-websockets-support/ + +That will provide the base infrastructure for publishing and subscribing to +events but nothing will be using MQTT after that. To start the event reporting +we'll leverage germqtt_ (and make it an Infra project). This will be the +initial usage for the unified message bus and make the Gerrit event stream +available over MQTT. While the scope of this spec ends with getting Gerrit into +MQTT, once we have an example we can more easily add more sources reporting +directly into MQTT. + +.. _germqtt: https://github.com/mtreinish/germqtt + +Alternatives +------------ + +Use fedmsg +^^^^^^^^^^ +The fedora infrastructure team had a similar need/desire to have a unified +message bus and they started the fedmsg_ project to address this need. +The primary issue with fedmsg is that it depends on AMQP or ZMQ both of which +we have had previous issues with. The choice to go with MQTT was partially a +reaction to that. + +.. _fedmsg: http://www.fedmsg.com/en/latest/ + + +Use a different mqtt broker +^^^^^^^^^^^^^^^^^^^^^^^^^^^ +There are several open source MQTT brokers besides Mosquitto out there +including ones like RabbitMQ_, VerneMQ_, and mosca_. + +.. _RabbitMQ: http://www.rabbitmq.com/blog/2012/09/12/mqtt-adapter/ +.. _VerneMQ: https://github.com/erlio/vernemq +.. _mosca: https://github.com/mcollina/mosca + +Mosquitto was chosen because it's a lightweight C program which will need less +resources to run it. It also should be simpler to deploy since we don't have to +worry about deploying a JavaVM, nodejs, or erlang (which seems to be quite +popular for writing message brokers). + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + + * mtreinish + * fungi + + +Gerrit Topic +------------ + +Use Gerrit topic "firehose" for all patches related to this spec. + +.. code-block:: bash + + git-review -t firehose + +Work Items +---------- + + #. Write puppet-mosquitto module + #. Deploy firehose01 server with puppet-mosquitto module and a + firehose.openstack.org CNAME to it + #. Add germqtt to openstack-infra + #. Write puppet-germqtt + #. Deploy puppet-germqtt on firehose01 + +Repositories +------------ + +2 New repos will be created in the initial implemention, puppet-mosquitto and +puppet-germqtt. + +Servers +------- + +1 new server will need to be created for running mosquitto, firehose01. Since +mosquitto and MQTT is fairly lightweight we probably can get away with a +smaller server (2GB flavor should be plenty). + +DNS Entries +----------- + +Forward/reverse DNS entries will be added for firehose01.openstack.org, as well +as a CNAME of firehose.openstack.org. + +Documentation +------------- + +Initial service documentation will be added to the +openstack-infra/system-config repo as usual. An availability announcement will +be sent to the openstack-dev mailing list. + +Security +-------- + +Sensitive data should not be published over the message bus. It is public and +anonymously accessible. + +Testing +------- + +The new Puppet modules will run our usual Puppet test and style jobs, and the +germqtt repo will have typical Python unit testing and style checks as well. + +Dependencies +============ + +- This will be the first use of germqtt in infra (which will be added as an + infra project) +- 2 new puppet modules will be created, 1 to deploy mosquitto and the other to + run germqtt