From bf09f09d0da0548021065ebaf31686a53aa5b489 Mon Sep 17 00:00:00 2001 From: Ozgur Akan Date: Wed, 5 Mar 2014 11:29:43 -0500 Subject: [PATCH] Marconi Operations Document Adding developer/operation documents including: 1. Glossary 2. Install and configure 3. Deploy Marconi in a minimal HA env 4. Access API with Marconi client Co-Authored-By: Fei Long Wang Implements: blueprint docs Change-Id: I2549995fb0754c7f3c8ce718639e1299e2805795 --- doc/source/api.rst | 64 ++++++++++ doc/source/glossary.rst | 77 ++++++++++++ doc/source/ha.rst | 30 +++++ doc/source/index.rst | 11 +- doc/source/installing.rst | 252 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 431 insertions(+), 3 deletions(-) create mode 100644 doc/source/api.rst create mode 100644 doc/source/glossary.rst create mode 100644 doc/source/ha.rst create mode 100644 doc/source/installing.rst diff --git a/doc/source/api.rst b/doc/source/api.rst new file mode 100644 index 000000000..144699c72 --- /dev/null +++ b/doc/source/api.rst @@ -0,0 +1,64 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Using Marconi's Public APIs +=========================== + +Marconi fully implements version 1.0 of the OpenStack Messaging API by now. +Generally, you can use any HTTP client to talk with Marconi public REST API, +though Marconi client is the recommended approach. + + +Marconi Client +############################################ +We can easily access the Marconi REST API via Marconi client. Below is an example +to create a queue, post messages to it and finally delete it:: + + from marconiclient.queues.v1 import client + + URL = 'http://localhost:8888' + messages = [{'body': {'id': idx}, 'ttl': 360} for idx in range(20)] + + cli = client.Client(URL) + queue = cli.queue('myqueue') + queue.post(messages) + + for msg in queue.messages(echo=True): + print(msg.body) + msg.delete() + + queue.delete() + + +curl +#### + +Define these variables:: + + # USERNAME=my identity username + # APIKEY=my-long-api-key + # ENDPOINT=test-queue.mydomain.com < keystone endpoint > + # QUEUE=test-queue + # CLIENTID=c5a6114a-523c-4085-84fb-533c5ac40789 + # HTTP=http + # PORT=80 + # TOKEN=9abb6d47de3143bf80c9208d37db58cf < your token here > + +Create the queue:: + + # curl -i -X PUT $HTTP://$ENDPOINT:$PORT/v1/queues/$QUEUE -H "X-Auth-Token: $TOKEN" -H "Client-ID: $CLIENTID" + HTTP/1.1 201 Created + content-length: 0 + location: /v1/queues/test-queue + +```HTTP/1.1 201 Created``` response proves that service is functioning properly. \ No newline at end of file diff --git a/doc/source/glossary.rst b/doc/source/glossary.rst new file mode 100644 index 000000000..6680fa19d --- /dev/null +++ b/doc/source/glossary.rst @@ -0,0 +1,77 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +========== + Glossary +========== + +Messaging Service Concepts +========================== +The Messaging Service is a multi-tenant, message queue implementation that +utilizes a RESTful HTTP interface to provide an asynchronous communications +protocol, which is one of the main requirements in today’s scalable applications. + +.. glossary:: + + Queue + Queue is a logical entity that groups messages. Ideally a queue is created + per work type. For example, if you want to compress files, you would create + a queue dedicated for this job. Any application that reads from this queue + would only compress files. + + Message + Message is sent through a queue and exists until it is deleted by a recipient + or automatically by the system based on a TTL (time-to-live) value. + + Claim + Claim is a mechanism to mark messages so that other workers will not process the same message. + + Worker + Worker is an application that reads one or multiple messages from the queue. + + Producer + Producer is an application that creates messages in one or multiple queues. + + Publish - Subscribe + Publish - Subscribe is a pattern where all worker applications have access + to all messages in the queue. Workers can not delete or update messages. + + Producer - Consumer + Producer - Consumer is a pattern where each worker application that reads + the queue has to claim the message in order to prevent duplicate processing. + Later, when the work is done, the worker is responsible from deleting the + message. If message is not deleted in a predefined time (claim TTL), it can + be claimed by other workers. + + Message TTL + Message TTL is time-to-live value and defines how long a message will be accessible. + + Claim TTL + Claim TTL is time-to-live value and defines how long a message will be in + claimed state. A message can be claimed by one worker at a time. + + Queues Database + Queues database stores the information about the queues and the messages + within these queues. Storage layer has to guarantee durability and availability of the data. + + Sharding + If sharding enabled, queuing service uses multiple queues databases in order + to scale horizontally. A shard (queues database) can be added anytime without + stopping the service. Each shard has a weight that is assigned during the + creation time but can be changed later. Sharding is done by queue which + indicates that all messages for a particular queue can be found in the same shard (queues database). + + Catalog Database + If sharding is enabled, catalog database has to be created. Catalog database + maintains ```queues``` to ```queues database``` mapping. Storage layer has + to guarantee durability and availability of data. diff --git a/doc/source/ha.rst b/doc/source/ha.rst new file mode 100644 index 000000000..d68604fb4 --- /dev/null +++ b/doc/source/ha.rst @@ -0,0 +1,30 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Minimum Scalable HA Setup +========================= + +OpenStack Queuing Service has two main layers. First one is the transport +(queuing application) layer which provides the RESTful interface, second one +is the storage layer which keeps all the data and meta-data about queues and messages. + +For a HA setup, a load balancer has to be placed in front of the web servers. +Load balancer setup is out of scope in this document. + +For storage we will use ```mongoDB``` in order to provide high availability with +minimum administration overhead. For transport, we will use ```wsgi```. + +To have a small footprint while providing HA, we will use 2 web servers which +will host the application and 3 mongoDB servers (configured as replica-sets) +which will host the catalog and queues databases. At larger scale, catalog +database and the queues database are advised to be hosted on different mongoDB replica sets. diff --git a/doc/source/index.rst b/doc/source/index.rst index 3611ac004..c1c766b69 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -1,7 +1,4 @@ .. - Copyright 2010-2014 OpenStack Foundation - All Rights Reserved. - Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at @@ -41,20 +38,28 @@ Concepts .. toctree:: :maxdepth: 1 + glossary + Installing/Configuring Marconi ============================== .. toctree:: :maxdepth: 1 + installing + Operating Marconi ================= .. toctree:: :maxdepth: 1 + ha + Using Marconi ============= .. toctree:: :maxdepth: 1 + + api \ No newline at end of file diff --git a/doc/source/installing.rst b/doc/source/installing.rst new file mode 100644 index 000000000..edb28a5dd --- /dev/null +++ b/doc/source/installing.rst @@ -0,0 +1,252 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Installing and Configuring +============================ + +System Requirements +~~~~~~~~~~~~~~~~~~~ + +Before you install OpenStack Queuing Service, you must meet the following system requirements.lst:: + +- OpenStack Compute Installation. +- Enable the Identity Service for user and project management. +- Python 2.6 or 2.7 + +Installing from packages +~~~~~~~~~~~~~~~~~~~~~~~~ + +Before you install and configure queuing service, meet the requirements in the +section called "System Requirements". Below samples are installing the packages on +RedHat operating system. + +Install mongoDB on Database Servers +################################### + +Install mongoDB on three servers and setup the replica-set. + +Configure Package Management System (YUM) +######################################### + +Create a ```/etc/yum.repos.d/mongodb.repo``` file to hold the following +configuration information for the MongoDB repository: + +If you are running a 64-bit system, use the following configuration:: + + [mongodb] + name=MongoDB Repository + baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/ + gpgcheck=0 + enabled=1 + +If you are running a 32-bit system, which is not recommended for production +deployments, use the following configuration:: + + [mongodb] + name=MongoDB Repository + baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/i686/ + gpgcheck=0 + enabled=1 + +Install Packages +################ + +Issue the following command (as root or with sudo) to install the latest stable +version of MongoDB and the associated tools:: + + #yum install mongo-10gen mongo-10gen-server + +Edit ```/etc/sysconfig/mongod```:: + + logpath=/var/log/mongo/mongod.log + logappend=true + fork = true + dbpath=/var/lib/mongo + pidfilepath = /var/run/mongodb/mongod.pid + replSet = catalog + nojournal = true + profile = 1 + slowms = 200 + oplogSize = 2048 + +Start mongoDB on all database servers:: + + #service mongodb start + +Configure Replica Set +##################### + +Assuming that primary mongodb servers hostname is ```mydb0.example-queues.net```, +once you install mongoDB on three servers go to ```mydb0``` and run the commands below;:: + + mydb0# mongo local --eval "printjson(rs.initiate())" + mydb0# rs.add("mydb1.example-queues.net") + mydb0# rs.add("mydb2.example-queues.net") + +To check if replica-set is established run this command;:: + + mydb0:~# mongo local --eval "printjson(rs.status())" + +Install memcached on Web Servers +################################ + +Install memcached on web servers in order to cache identity tokens and catalog mappings.:: + + web# yum install memcached + +Start memcached service:: + + web# service memcached start + +Install uwsgi on Web Servers:: + + web# yum -y install python-pip + web# pip install uwsgi + +Configure OpenStack Marconi +########################### + +On web servers run these commands:: + + web# git clone https://github.com/openstack/marconi.git . + web# pip install . -r ./requirements.txt --upgrade --log /tmp/marconi-pip.log + +Create ```/srv/marconi``` folder to store related configurations files. + +Create ```/srv/marconi/marconi_uwsgi.py``` with the following content:: + + from keystoneclient.middleware import auth_token + from marconi.transport.wsgi import app + + app = auth_token.AuthProtocol(app.app, {}) + +Create ```/srv/marconi/uwsgi.ini``` file with the following content:: + + [uwsgi] + http = 192.168.192.168:80 + daemonize = /var/log/marconi.log + pidfile = /var/run/marconi.pid + gevent = 2000 + gevent-monkey-patch = true + listen = 1024 + enable-threads = true + module = marconi_uwsgi:app + workers = 4 + +The uwsgi configuration options above can be modified for different performance requirements. + +Create marconi configuration file ```/etc/marconi.conf```:: + + [DEFAULT] + # Show more verbose log output (sets INFO log level output) + #verbose = False + + # Show debugging output in logs (sets DEBUG log level output) + #debug = False + + # Sharding and admin mode configs + sharding = True + admin_mode = True + + # Log to this file! + log_file = /var/log/marconi-queues.log + debug = False + verbose = False + + # This is taken care of in our custom app.py, so disable here + ;auth_strategy = keystone + + [keystone_authtoken] + admin_password = < admin password > + admin_tenant_name = < admin tenant name > + admin_user = < admin user > + auth_host = < identity service host > + auth_port = '443' + auth_protocol = 'https' + auth_uri = < identity service uri > + auth_version = < auth version > + token_cache_time = < token cache time > + memcache_servers = 'localhost:11211' + + [oslo_cache] + cache_backend = memcached + memcache_servers = 'localhost:11211' + + [drivers] + # Transport driver module (e.g., wsgi, zmq) + transport = wsgi + # Storage driver module (e.g., mongodb, sqlite) + storage = mongodb + + [drivers:storage:mongodb] + uri = mongodb://mydb0,mydb1,mydb2:27017/?replicaSet=catalog&w=2&readPreference=secondaryPreferred + database = marconi + partitions = 8 + + # Maximum number of times to retry a failed operation. Currently + # only used for retrying a message post. + ;max_attempts = 1000 + + # Maximum sleep interval between retries (actual sleep time + # increases linearly according to number of attempts performed). + ;max_retry_sleep = 0.1 + + # Maximum jitter interval, to be added to the sleep interval, in + # order to decrease probability that parallel requests will retry + # at the same instant. + ;max_retry_jitter = 0.005 + + # Frequency of message garbage collections, in seconds + ;gc_interval = 5 * 60 + + # Threshold of number of expired messages to reach in a given + # queue, before performing the GC. Useful for reducing frequent + # locks on the DB for non-busy queues, or for worker queues + # which process jobs quickly enough to keep the number of in- + # flight messages low. + # + # Note: The higher this number, the larger the memory-mapped DB + # files will be. + ;gc_threshold = 1000 + + [limits:transport] + queue_paging_uplimit = 1000 + metadata_size_uplimit = 262144 + message_paging_uplimit = 10 + message_size_uplimit = 262144 + message_ttl_max = 1209600 + claim_ttl_max = 43200 + claim_grace_max = 43200 + + [limits:storage] + default_queue_paging = 10 + default_message_paging = 10 + +Start queuing service:: + + #/usr/bin/uwsgi --ini /srv/marconi/uwsgi.ini + + +Configure Shards +~~~~~~~~~~~~~~~~ + +To have a functional queuing service, we need to define a shard. On one of the +web servers run this command:: + + curl -i -X PUT -H 'X-Auth-Token: $TOKEN' -d '{"weight": 100, "uri": "mongodb://mydb0,mydb1,mydb2:27017/?replicaSet=catalog&w=2&readPreference=secondaryPreferred", "options": {"partitions": 8}}' http://localhost:8888/v1/shards/shard1 + +Above ```$TOKEN``` is the authentication token retrieved from identity service. +If you choose not to enable keystone authentication you won't have to pass a token. + +Reminder: In larger deployments, catalog database and queues databases (shards) +are going to be on different mongoDB replica-sets.