diff --git a/README.rst b/README.rst index 0a10f1330..a960253f1 100644 --- a/README.rst +++ b/README.rst @@ -2,179 +2,63 @@ Zaqar ===== -Message queuing service for `OpenStack`_. -To find more information read our `wiki`_. +Zaqar is a multi-tenant cloud messaging and notification service for web +and mobile developers. +It combines the ideas pioneered by Amazon's SQS product with additional +semantics to support event broadcasting. -Running a local Zaqar server with MongoDB ------------------------------------------ +The service features a fully RESTful API, which developers can use to send +messages between various components of their SaaS and mobile applications, by +using a variety of communication patterns. Underlying this API is an efficient +messaging engine designed with scalability and security in mind. -**Note:** These instructions are for running a local instance of Zaqar and not -all of these steps are required. It is assumed you have `MongoDB`_ and `tox` -(see "Running tests" section below) installed and running. +Other OpenStack components can integrate with Zaqar to surface events to end +users and to communicate with guest agents that run in the "over-cloud" layer. +Cloud operators can leverage Zaqar to provide equivalents of SQS and SNS to +their customers. -1. Install prerequisites: +General information is available in wiki: -# Ubuntu/Debian: -sudo apt-get install gcc python-pip libxml2-dev libxslt1-dev python-dev zlib1g-dev + https://wiki.openstack.org/wiki/Zaqar -# Fedora/RHEL: -sudo yum install gcc python-pip libxml2-devel libxslt-devel python-devel +The API v1.1 (stable) specification and documentation are available at: -2. From your home folder create the ``~/.zaqar`` folder and clone the repo:: + https://wiki.openstack.org/wiki/Zaqar/specs/api/v1.1 - $ cd - $ mkdir ~/.zaqar - $ git clone https://git.openstack.org/openstack/zaqar.git +Zaqar Contributor Documentation, the source of which is in ``doc/source/``, is +available at: -3. Generate and copy the Zaqar config files to the directory ``~/.zaqar``:: + http://docs.openstack.org/developer/zaqar/ - $ pip install tox - $ cd zaqar - $ tox -e genconfig - $ cp etc/zaqar.conf.sample ~/.zaqar/zaqar.conf - $ cp etc/logging.conf.sample ~/.zaqar/logging.conf +Contributors are encouraged to join IRC (``#openstack-zaqar`` channel on +``irc.freenode.net``): -4. Find ``[drivers]`` section in ``~/.zaqar/zaqar.conf`` - and specify to use mongodb storage:: + https://wiki.openstack.org/wiki/IRC - message_store = mongodb - management_store = mongodb +Information on how to run unit and functional tests is available at: - Then find the ``[drivers:message_store:mongodb]`` and - ``[drivers:management_store:mongodb]`` sections and - specify the URI to point to your local - mongod instance by adding this line to both the - sections:: + http://docs.openstack.org/developer/zaqar/running_tests.html - uri = mongodb://$MONGODB_HOST:$MONGODB_PORT +Information on how to run benchmarking tool is available at: - By default, you will have:: + http://docs.openstack.org/developer/zaqar/running_benchmark.html - uri = mongodb://127.0.0.1:27017 +Using Zaqar +----------- - NOTE: If your local dev/test mongodb doesn't enable the replica set, then - you have to set below in [default] section:: +If you are new to Zaqar and just want to try it, you can set up Zaqar in +the development environment. - unreliable = True +Using Zaqar in production environment: -5. For logging, find the ``[handler_file]`` section in - ``~/.zaqar/logging.conf`` and modify as desired:: + Coming soon! - args=('zaqar.log', 'w') +Using Zaqar in development environment: -6. Change directories back to your local copy of the repo:: + The instruction is available at: - $ cd ~/zaqar + http://docs.openstack.org/developer/zaqar/devref/development.environment.html -7. Run the following so you can see the results of any changes you - make to the code without having to reinstall the package each time:: - - $ pip install -e . - -8. Start the Zaqar server with logging level set to INFO so you can see - the port on which the server is listening:: - - $ zaqar-server -v - -9. Test out that Zaqar is working by creating a queue:: - - $ ZQ_CLIENT_ID=`uuidgen` - $ curl -i -X PUT http://127.0.0.1:8888/v1.1/queues/samplequeue \ - -H "Content-type: application/json" \ - -H "Client-ID: $ZQ_CLIENT_ID" \ - -H "X-PROJECT-ID: default" - -You should get an **HTTP 201** along with some headers that will look -similar to this:: - - HTTP/1.0 201 Created - Date: Fri, 25 Oct 2013 15:34:37 GMT - Server: WSGIServer/0.1 Python/2.7.3 - Content-Length: 0 - Location: /v1.1/queues/samplequeue - -Running tests -------------- - -Run tests using the following command:: - - $ tox -e py27 - -You can read more about running functional tests in separate `TESTS_README`_. - -Running the benchmarking tool ------------------------------ - -First install and run zaqar-server (see above). - -Then install additional requirements:: - - $ pip install -r bench-requirements.txt - -Copy the configuration file to ``~/.zaqar``:: - - $ cp etc/zaqar-benchmark.conf.sample ~/.zaqar/zaqar-benchmark.conf - -In the configuration file specify where zaqar-server can be found:: - - server_url = http://localhost:8888 - -The benchmarking tool needs a set of messages to work with. Specify the path -to the file with messages in the configuration file. Alternatively, put it in -the directory with the configuration file and name it ``zaqar-benchmark- -messages.json``. As a starting point, you can use the sample file from the -``etc`` directory:: - - $ cp etc/zaqar-benchmark-messages.json ~/.zaqar/ - -If the file is not found or no file is specified, a single hard-coded message -is used for all requests. - -Run the benchmarking tool using the following command:: - - $ zaqar-bench - -By default, the command will run a performance test for 5 seconds, using one -producer process with 10 greenlet workers, and one observer process with -5 workers. The consumer role is disabled by default. - -You can override these defaults in the config file or on the command line -using a variety of options. For example, the following command runs a -performance test for 30 seconds using 4 producer processes with -20 workers each, plus 4 consumer processes with 20 workers each. Note that -the observer role is also disabled in this example by setting its number of -workers to zero:: - - $ zaqar-bench -pp 4 -pw 10 -cp 4 -cw 20 -ow 0 -t 30 - -By default, the results are in JSON. For more human-readable output add -the ``--verbose`` flag. Verbose output looks similar to the following:: - - $ zaqar-bench --verbose - - Starting producer (pp=1 , pw=10)... - - Starting observer (op=1 , ow=5)... - - Producer - ======== - duration_sec: 5.1 - ms_per_req: 2.9 - reqs_per_sec: 344.5 - successful_reqs: 1742.0 - total_reqs: 1742.0 - - Observer - ======== - duration_sec: 5.0 - ms_per_req: 2.9 - reqs_per_sec: 339.3 - successful_reqs: 1706.0 - total_reqs: 1706.0 - - -.. _`OpenStack` : http://openstack.org/ -.. _`MongoDB` : http://docs.mongodb.org/manual/installation/ -.. _`wiki` : https://wiki.openstack.org/wiki/Zaqar -.. _`TESTS_README` : https://github.com/openstack/zaqar/blob/master/zaqar/tests/functional/README.rst + This will allow you to run local Zaqar server with MongoDB as database. + This way is the easiest, quickest and most suitable for beginners. \ No newline at end of file diff --git a/doc/source/devref/development.environment.rst b/doc/source/devref/development.environment.rst index e95b77c7f..fd9e49787 100644 --- a/doc/source/devref/development.environment.rst +++ b/doc/source/devref/development.environment.rst @@ -15,10 +15,10 @@ Setting up a development environment ==================================== -This section describes how to setup a working Python development -environment that you can use in developing Zaqar on Ubuntu or Fedora. -These instructions assume that you are familiar with -Git. Refer to GettingTheCode_ for additional information. +This section describes how to setup a working Python development environment +that you can use in developing Zaqar on Ubuntu or Fedora. These instructions +assume that you are familiar with Git. Refer to GettingTheCode_ for +additional information. .. _GettingTheCode: http://wiki.openstack.org/GettingTheCode @@ -26,20 +26,18 @@ Git. Refer to GettingTheCode_ for additional information. Virtual environments -------------------- -Use virtualenv_ to track and manage Python dependencies -for developing and testing Zaqar. -Using virtualenv_ enables you to install Python dependencies -in an isolated virtual environment, instead of installing the -packages at the system level. +Use virtualenv_ to track and manage Python dependencies for developing and +testing Zaqar. +Using virtualenv_ enables you to install Python dependencies in an isolated +virtual environment, instead of installing the packages at the system level. .. _virtualenv: http://pypi.python.org/pypi/virtualenv .. note:: - Virtualenv is useful for development purposes, but is not - typically used for full integration testing or production usage. - If you want to learn about production best practices, check out - the `OpenStack Operations Guide`_. + Virtualenv is useful for development purposes, but is not typically used for + full integration testing or production usage. If you want to learn about + production best practices, check out the `OpenStack Operations Guide`_. .. _`OpenStack Operations Guide`: http://docs.openstack.org/ops/ @@ -48,21 +46,22 @@ Install GNU/Linux system dependencies .. note:: - This section is tested for Zaqar on Ubuntu 14.04 (Trusty) and - Fedora-based (RHEL 6.1) distributions. Feel free to add notes - and change according to your experiences or operating system. - Learn more about contributing to Zaqar documentation in the - `Write the Docs!`_ wiki. - - .. _`Write the Docs!`: https://wiki.openstack.org/wiki/Write_the_Docs!_(Zaqar) + This section is tested for Zaqar on Ubuntu 14.04 (Trusty) and Fedora-based + (RHEL 6.1) distributions. Feel free to add notes and change according to your + experiences or operating system. Learn more about contributing to Zaqar + documentation in the :doc:`../welcome` manual. Install the prerequisite packages. -On Ubuntu:: +On Ubuntu: + +.. code-block:: console $ sudo apt-get install gcc python-pip libxml2-dev libxslt1-dev python-dev zlib1g-dev -On Fedora-based distributions (e.g., Fedora/RHEL/CentOS):: +On Fedora-based distributions (e.g., Fedora/RHEL/CentOS): + +.. code-block:: console $ sudo yum install gcc python-pip libxml2-devel libxslt-devel python-devel @@ -73,7 +72,8 @@ You also need to have MongoDB_ installed and running. .. _MongoDB: http://www.mongodb.org -On Ubuntu, follow the instructions in the `MongoDB on Ubuntu Installation Guide`_. +On Ubuntu, follow the instructions in the +`MongoDB on Ubuntu Installation Guide`_. .. _`MongoDB on Ubuntu installation guide`: http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/ @@ -82,115 +82,217 @@ On Fedora-based distributions, follow the instructions in the .. _`MongoDB on Red Hat Enterprise, CentOS, Fedora, or Amazon Linux installation guide`: http://docs.mongodb.org/manual/tutorial/install-mongodb-on-red-hat-centos-or-fedora-linux/ +.. note:: + + If you are Contributor and plan to run Unit tests on Zaqar, you may want to + add this line to mongodb configuration file (``etc/mongod.conf`` or + ``etc/mongodb.conf`` depending on distribution): + + .. code-block:: ini + + smallfiles = true + + Many Zaqar's Unit tests do not clean up their testing databases after + executing. And database files consume much disk space even if they do not + contain any records. This behavior will be fixed soon. + Getting the code ################ -Get the code from GitHub:: +Get the code from GitHub to create a local repository with Zaqar: - $ git clone https://github.com/openstack/zaqar.git +.. code-block:: console + + $ git clone https://github.com/openstack/zaqar.git Configuration ############# -1. From your home folder create the ~/.zaqar folder. This directory holds the configuration files for Zaqar:: +#. From your home folder create the ``~/.zaqar`` folder. This directory holds + the configuration files for Zaqar: - $ mkdir ~/.zaqar + .. code-block:: console -2. Generate the sample configuration file zaqar/etc/zaqar.conf.sample:: + $ mkdir ~/.zaqar - $ pip install tox - $ cd zaqar - $ tox -e genconfig +#. Generate the sample configuration file ``zaqar/etc/zaqar.conf.sample``: -3. Copy the Zaqar configuration samples to the directory ~/.zaqar/:: + .. code-block:: console - $ cp etc/zaqar.conf.sample ~/.zaqar/zaqar.conf - $ cp etc/logging.conf.sample ~/.zaqar/logging.conf + $ pip install tox + $ cd zaqar + $ tox -e genconfig -4. Find the [drivers] section in ~/.zaqar/zaqar.conf and specify mongodb as the message store:: +#. Copy the Zaqar configuration samples to the directory ``~/.zaqar/``: - message_store = mongodb - management_store = mongodb + .. code-block:: console -5. Find the [drivers:message_store:mongodb] section and modify the URI to point to your local mongod instance:: + $ cp etc/zaqar.conf.sample ~/.zaqar/zaqar.conf + $ cp etc/logging.conf.sample ~/.zaqar/logging.conf - uri = mongodb://$MONGODB_HOST:$MONGODB_PORT # default = mongodb://localhost:27017 +#. Find the ``[drivers]`` section in ``~/.zaqar/zaqar.conf`` and specify + ``mongodb`` as the message store: -6. For logging, find the [handler_file] section in ~/.zaqar/logging.conf and modify as desired:: + .. code-block:: ini - args=('zaqar.log', 'w') + message_store = mongodb + management_store = mongodb + +#. Then find ``[drivers:message_store:mongodb]`` and + ``[drivers:management_store:mongodb]`` sections and specify the + :samp:`{URI}` to point to your local mongodb instance by adding this line + to both the sections: + + .. code-block:: ini + + uri = mongodb://$MONGODB_HOST:$MONGODB_PORT + + By default you will have: + + .. code-block:: ini + + uri = mongodb://127.0.0.1:27017 + + This :samp:`{URI}` points to single mongodb node which of course is not + reliable, so you need to set in the ``[default]`` section of configuration + file: + + .. code-block:: ini + + unreliable = True + + For your reference, you can omit this parameter or set it to False only + if the provided :samp:`{URI}` to your mongodb is actually the URI to mongodb + Replica Set or Mongos. Also it must have "Write concern" parameter set to + ``majority`` or to a number more than ``1``. + + For example, :samp:`{URI}` to reliable mongodb can look like this: + + .. code-block:: ini + + uri = mongodb://mydb0,mydb1,mydb2:27017/?replicaSet=foo&w=2 + + Where ``mydb0``, ``mydb1``, ``mydb2`` are addresses of the configured + mongodb Replica Set nodes, ``replicaSet`` (Replica Set name) parameter is + set to ``foo``, ``w`` (Write concern) parameter is set to ``2``. + +#. For logging, find the ``[handler_file]`` section in + ``~/.zaqar/logging.conf`` and modify as desired: + + .. code-block:: ini + + args=('zaqar.log', 'w') Installing and using virtualenv ############################### -1. Install virtualenv by running:: +#. Install virtualenv by running: - $ pip install virtualenv + .. code-block:: console -2. Create and activate a virtual environment:: + $ pip install virtualenv - $ virtualenv zaqarenv - $ source zaqarenv/bin/activate +#. Create and activate a virtual environment: -3. Install Zaqar:: + .. code-block:: console - $ pip install -e . + $ virtualenv zaqarenv + $ source zaqarenv/bin/activate -4. Install the required Python binding for MongoDB:: +#. Install Zaqar: - $ pip install pymongo + .. code-block:: console -5. Start the Zaqar server:: + $ pip install -e . - $ zaqar-server -v +#. Install the required Python binding for MongoDB: -6. Verify Zaqar is running by creating a queue:: + .. code-block:: console - $ curl -i -X PUT http://localhost:8888/v1/queues/samplequeue -H "Content-type: application/json" + $ pip install pymongo -7. Get ready to code! +#. Start Zaqar server in ``info`` logging mode: + + .. code-block:: console + + $ zaqar-server -v + + Or you can start Zaqar server in ``debug`` logging mode: + + .. code-block:: console + + $ zaqar-server -d + +#. Verify Zaqar is running by creating a queue via curl. In a separate + terminal run: + + .. code-block:: console + + $ curl -i -X PUT http://localhost:8888/v1/queues/samplequeue -H "Content-type: application/json" + +#. Get ready to code! .. note:: - You can run the Zaqar server in the foreground by passing the - --nodaemon flag:: + You can run the Zaqar server in the background by passing the + :option:`--daemon` flag: - $ zaqar-server -v --nodaemon + .. code-block:: console - With this method you get immediate visual feedback and it is - easier to kill and restart the process. + $ zaqar-server -v --daemon - If you do so, you have to run the cURL test (step 5) in a - separate terminal. + But with this method you will not get immediate visual feedback and it will + be harder to kill and restart the process. + +Troubleshooting +^^^^^^^^^^^^^^^ + +No handlers found for zaqar.client (...) +"""""""""""""""""""""""""""""""""""""""" + +This happens because the current user cannot create the log file (for the +default configuration in ``/var/log/zaqar/server.log``). To solve it, create +the folder: + +.. code-block:: console + + $ sudo mkdir /var/log/zaqar + +Create the file: + +.. code-block:: console + + $ sudo touch /var/log/zaqar/server.log + +And try running the server again. DevStack -------- -If you want to use Zaqar in an integrated OpenStack developing -environment, you can add it to your DevStack_ deployment. +If you want to use Zaqar in an integrated OpenStack developing environment, you +can add it to your DevStack_ deployment. -To do this, you first need to add the following setting -to your local.conf:: +To do this, you first need to add the following setting to your ``local.conf``: - enable_plugin zaqar https://github.com/openstack/zaqar +.. code-block:: ini -Then run the stack.sh script as usual. + enable_plugin zaqar https://github.com/openstack/zaqar -After running the DevStack_ script, you can start the Zaqar server -and test it by following steps 5 and 6 from the previous section. +Then run the ``stack.sh`` script as usual. .. _DevStack: http://docs.openstack.org/developer/devstack/ -Running unit tests ------------------- +Running tests +------------- See :doc:`../running_tests` for details. +Running the benchmarking tool +----------------------------- + +See :doc:`../running_benchmark` for details. + Contributing your work ---------------------- -Once your work is complete, you may wish to contribute it to the project. -Zaqar uses the Gerrit code review system. For information on how to submit -your branch to Gerrit, see GerritWorkflow_. - -.. _GerritWorkflow: http://docs.openstack.org/infra/manual/developers.html#development-workflow +See :doc:`../welcome` and :doc:`../first_patch` for details. \ No newline at end of file diff --git a/doc/source/first_patch.rst b/doc/source/first_patch.rst index bac351667..3f3c65d51 100644 --- a/doc/source/first_patch.rst +++ b/doc/source/first_patch.rst @@ -25,13 +25,16 @@ Create your contributor accounts and set up your code environment Accounts setup ############## -You will need to create a Launchpad_ account to login to the Gerrit_ review system dashboard. -This is also useful for automatically crediting bug fixes to you when you address them -with your code commits. You will also have to sign the `Contributors License Agreement`_ and `join the OpenStack Foundation`_. It is a good idea to use the same email all of these accounts to +You will need to create a Launchpad_ account to login to the Gerrit_ review +system dashboard. +This is also useful for automatically crediting bug fixes to you when you +address them with your code commits. You will also have to sign the +`Contributors License Agreement`_ and `join the OpenStack Foundation`_. +It is a good idea to use the same email all of these accounts to avoid hooks errors. -Visit the the `Gerrit Workflow's account setup`_ section in the wiki -to get more information on setting up your accounts. +Visit the the `Gerrit Workflow's account setup`_ section in the wiki to get +more information on setting up your accounts. .. _Launchpad: http://launchpad.net/ .. _Gerrit: http://review.openstack.org/ @@ -42,14 +45,18 @@ to get more information on setting up your accounts. SSH setup ######### -You are going to need to create and upload an SSH key to Gerrit to be -able to commit changes for review. To create an SSH key:: +You are going to need to create and upload an SSH key to Gerrit to be able to +commit changes for review. To create an SSH key: + +.. code-block:: console $ ssh-keygen –t rsa -You can optionally enter a password to enchance security. +You can optionally enter a password to enhance security. -View and copy your SSH key:: +View and copy your SSH key: + +.. code-block:: console $ less ~/.ssh/id_rsa.pub @@ -60,23 +67,31 @@ Now you can `upload the SSH key to Gerrit`_. Git Review installation ####################### -Before you start working, make sure you have git-review installed on your system. -You can install it with the following command:: +Before you start working, make sure you have ``git-review`` installed on your +system. + +You can install it with the following command: + +.. code-block:: console $ pip install git-review -Git-review checks if you can authenticate to Gerrit with your SSH key. -It will ask you for your username. You can configure your Gerrit username so you don't have to -keep re-entering it every time you want to use git-review:: +``Git-review`` checks if you can authenticate to Gerrit with your SSH key. +It will ask you for your username. You can configure your Gerrit username so +you don't have to keep re-entering it every time you want to use +``git-review``: + +.. code-block:: console $ git config --global gitreview.username yourgerritusername -You can also save some time by entering your email and your name:: +You can also save some time by entering your email and your name: + +.. code-block:: console $ git config --global gitreview.email "yourgerritemail" $ git config --global gitreview.name "Firstname Lastname" - You can view your Gerrit user name in the `settings page`_. .. _`settings page`: https://review.openstack.org/#/settings/ @@ -84,29 +99,39 @@ You can view your Gerrit user name in the `settings page`_. Project setup ############# -Clone the Zaqar repository with the following git command:: +Clone the Zaqar repository with the following git command: + +.. code-block:: console $ git clone git://git.openstack.org/openstack/zaqar.git For information on how to set up the Zaqar development environment see :doc:`devref/development.environment`. -Before writing code, you will have to do some configurations -to connect your local repository with Gerrit. You will only need to do this -your first time setting up the development environment. +Before writing code, you will have to do some configurations to connect your +local repository with Gerrit. You will only need to do this your first time +setting up the development environment. + +You can set ``git-review`` to configure the project and install the Gerrit +change-id commit hook with the following command: + +.. code-block:: console -You can set git-review to configure the project and install the gerrit change-id commit hook with the following command:: $ cd zaqar $ git review -s -If you get the error "We don't know where your Gerrit is", you will need to -add a new git remote. The URL should be in the error message. Copy that and -create the new remote. It looks something like:: +If you get the error "We don't know where your Gerrit is", you will need to add +a new git remote. The URL should be in the error message. Copy that and create +the new remote. It looks something like: + +.. code-block:: console $ git remote add gerrit ssh://@review.openstack.org:29418/openstack/zaqar.git -In the project directory you have a hidden .git directory and a -.gitreview file. You can view them with the following command:: +In the project directory you have a hidden ``.git`` directory and a +``.gitreview`` file. You can view them with the following command: + +.. code-block:: console $ ls -la @@ -117,15 +142,18 @@ Pick or report a bug #################### You can start tackling some bugs from the `bugs list in Launchpad`_. -If you find a bug you want to work on, assign yourself. Make sure to read -the bug report. If you need more information, ask the reporter to provide -more details through a comment on Launchpad or through IRC or email. +If you find a bug you want to work on, assign yourself. Make sure to read the +bug report. If you need more information, ask the reporter to provide more +details through a comment on Launchpad or through IRC or email. -If you find a bug, look through Launchpad to see if it has been reported. If it hasn't, report the bug, and -ask for another developer to confirm it. You can start working on it if another developer confirms the bug. +If you find a bug, look through Launchpad to see if it has been reported. If it +hasn't, report the bug, and ask for another developer to confirm it. You can +start working on it if another developer confirms the bug. Here are some details you might want to include when filling out a bug report: -* The release, or milestone, or commit ID corresponding to the software that you are running + +* The release, or milestone, or commit ID corresponding to the software that + you are running * The operating system and version where you've identified the bug * Steps to reproduce the bug, including what went wrong * Description of the expected results instead of what you saw @@ -142,25 +170,68 @@ You can read more about `Launchpad bugs`_ in the wiki. Workflow ######## -Make sure your repo is up to date. You can update it with the following git commands:: +Make sure your repo is up to date. You can update it with the following git +commands: + +.. code-block:: console $ git remote update $ git checkout master $ git pull --ff-only origin master -Create a topic branch. You can create one with the following git command:: +Create a topic branch. You can create one with the following git command: + +.. code-block:: console $ git checkout -b TOPIC-BRANCH -If you are working on a blueprint, name your topic branch bp/BLUEPRINT where BLUEPRINT -is the name of a blueprint in Launchpad (for example, "bp/authentication"). The general -convention when working on bugs is to name the branch bug/BUG-NUMBER -(for example, "bug/1234567"). +If you are working on a blueprint, name your :samp:`{TOPIC-BRANCH}` +``bp/BLUEPRINT`` where :samp:`{BLUEPRINT}` is the name of a blueprint in +Launchpad (for example, "bp/authentication"). The general convention when +working on bugs is to name the branch ``bug/BUG-NUMBER`` (for example, +"bug/1234567"). Read more about the commit syntax in the `Gerrit workflow`_ wiki. .. _`Gerrit workflow`: http://docs.openstack.org/infra/manual/developers.html#development-workflow +Common problems +^^^^^^^^^^^^^^^ + +#. You realized that you were working in master and you haven't made any + commits. Solution: + + .. code-block:: console + + $ git checkout -b newbranch + $ git commit -a -m "Edited" + + If you already created the branch, omit the :option:`-b`. + + You put all your changes to :samp:`{newbranch}`. Problem solved. + +#. You realized that you were working in master and you have made commits to + master. Solution: + + .. code-block:: console + + $ git branch newbranch + $ git reset --hard HEAD~x + $ git checkout newbranch + + Where ``x`` is the number of commits you have made to master. + And remember, you will lose any uncommitted work. + + You put your commits in :samp:`{newbranch}`. Problem solved. + +#. You made multiple commits and realized that Gerrit requires one commit per + patch. Solution: + + * You need to squash your previous commits. Make sure you are in your + branch and follow `squashing guide`_. Then fill commit message properly. + + You squashed your commits. Problem solved. + Design principles ################# @@ -192,37 +263,58 @@ Your commit message should: * Insert a single blank line after the first line. * Provide a detailed description of the change in the following lines, breaking paragraphs where needed. -* The first line should be limited to 50 characters and should not end with a period. +* The first line should be limited to 50 characters and should not end with a + period. * Subsequent lines should be wrapped at 72 characters. * Put the 'Change-id', 'Closes-Bug #NNNNN' and 'blueprint NNNNNNNNNNN' lines at the very end. Read more about `making a good commit message`_. -To submit it for review use the following git command:: +To submit it for review use the following git command: + +.. code-block:: console $ git review -You will see the URL of your review page once it is successfully sent. You can also see your -reviews in "My Changes" in Gerrit. The first thing to watch for is a +1 in the "Verified" -column next to your patch in the server and/or client list of pending patches. -If the "Jenkins" user gives you a -1, you'll need to check the log it posts to -find out what gate test failed, update your patch, and resubmit. +You will see the URL of your review page once it is successfully sent. -You can set your patch as a "work in progress" if your patch is not ready to be merged, -but you would still like some feedback from other developers. To do this -leave a review on your patch setting "Workflow" to -1. +You can also see your reviews in :guilabel:`My Changes` in Gerrit. The first +thing to watch for is a ``+1`` in the :guilabel:`Verified` column next to your +patch in the server and/or client list of pending patches. -Once the gate has verified your patch, other Zaqar devs will take a look and submit -their comments. When you get two or more +2's from core reviewers, -the patch will be approved and merged. +If the "Jenkins" user gives you a ``-1``, you'll need to check the log it posts +to find out what gate test failed, update your patch, and resubmit. -Don't be discouraged if a reviewer submits their comments with a "-1". -Patches iterate through several updates and reviews before they are ready for merging. -To reply to feedback save all your comments as draft, then click on the "Review" button. -When replying to feedback, you as the patch author can use the score of "0". -The only exception to using the score of "0" is when you discover a blocking issue -and you don't want your patch to be merged. In which case, you can review your own -patch with a "-2", while you decide whether to keep, refactor, or withdraw the patch. +You can set your patch as a :guilabel:`work in progress` if your patch is +not ready to be merged, but you would still like some feedback from other +developers. To do this leave a review on your patch setting +:guilabel:`Workflow` to ``-1``. + +Once the gate has verified your patch, other Zaqar developers will take a look +and submit their comments. When you get two or more ``+2``'s from core +reviewers, the patch will be approved and merged. + +Don't be discouraged if a reviewer submits their comments with a ``-1``. +Patches iterate through several updates and reviews before they are ready for +merging. + +To reply to feedback save all your comments as draft, then click on the +:guilabel:`Review` button. When replying to feedback, you as the patch +author can use the score of ``0``. The only exception to using the score of +``0`` is when you discover a blocking issue and you don't want your patch to +be merged. In which case, you can review your own patch with a ``-2``, while +you decide whether to keep, refactor, or withdraw the patch. + +Professional conduct +-------------------- + +The Zaqar team holds reviewers accountable for promoting a positive, +constructive culture within our program. + +If you ever feel that a reviewer is not acting professionally or is violating +the OpenStack community code of conduct, please let the PTL know immediately +so that he or she can help resolve the issue. .. _`making a good commit message`: https://wiki.openstack.org/wiki/GitCommitMessages +.. _`squashing guide` : http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html diff --git a/doc/source/first_review.rst b/doc/source/first_review.rst index 898e6b668..29158ff7b 100644 --- a/doc/source/first_review.rst +++ b/doc/source/first_review.rst @@ -15,61 +15,101 @@ Your first review ================= -The review stage is a very important part in the development process. Following are some of the reasons this stage is important: +The review stage is a very important part in the development process. Following +are some of the reasons this stage is important: * Getting other developers feedback minimizes the risk of adding - regressions to the code base and ensures the quality of the code being merged. -* Building the community encourages everyone to review code. Everyone appreciates having their code reviewed. -* Since developers are always learning from being exposed to the points of view of others, reviews help developers to improve their coding skills. -* Providing a review is a great way to become familar with the code. + regressions to the code base and ensures the quality of the code being + merged. +* Building the community encourages everyone to review code. Everyone + appreciates having their code reviewed. +* Since developers are always learning from being exposed to the points of view + of others, reviews help developers to improve their coding skills. +* Providing a review is a great way to become familiar with the code. -Everyone is encourages to review code. You don't need to know every detail of the code base. You need to understand only what the code related to the fix does. +Everyone is encourages to review code. You don't need to know every detail of +the code base. You need to understand only what the code related to the fix +does. Step by step ------------ -Go to review.openstack.org and filter by `Open Zaqar fixes`_. Select a fix from the list to review. -Try to select an easy patch for your first review. That will help you to gain some confidence. -Download the patch to your local repository and test it:: +Go to ``review.openstack.org`` and filter by `Open Zaqar fixes`_. Select a fix +from the list to review. Try to select an easy patch for your first review. +That will help you to gain some confidence. Download the patch to your local +repository and test it: + +.. code-block:: console $ git review -d [review-id] -The review-id is the number in the URL (check the screenshot for more details). +The :samp:`{review-id}` is the number in the URL (check the screenshot for more +details). -Example:: +Example: + +.. code-block:: console $ git review -d 92979 .. image:: images/zaqar_review_id.png :alt: Zaqar review id -This git command creates a branch with the author's name and enables you to test the patch in your local environment. +This git command creates a branch with the author's name and enables you to +test the patch in your local environment. -* Inspect the code. Use all of the best programming practices you know as you review the code. -* Give code location feedback. (Do you consider that some code should be better located in another place within the file, or maybe in another file? If so, suggest this in the review comment and score with a -1 if you think that it's that important.) -* Give code-style feedback. (Do you think that the code structure could be improved? Keep the DRY, YAGNI and KISS principles in mind.) -* Give grammar and orthography feedback. Many of our contributors are not native English speakers, so it is common to find some errors of this type. +* Inspect the code. Use all of the best programming practices you know as you + review the code. +* Give code location feedback. + Do you consider that some code should be better located in another place + within the file, or maybe in another file? If so, suggest this in the + review comment and score with a ``-1`` if you think that it's that + important. +* Give code-style feedback. + Do you think that the code structure could be improved? Keep the DRY, + YAGNI and KISS principles in mind. +* Give grammar and orthography feedback. Many of our contributors are not + native English speakers, so it is common to find some errors of this type. * Make sure that: - * The commit message is formatted appropriately (check `Git Commit Messages`_ for more information on how you should write a git commit message.) - * The coding style matches guidelines given in HACKING.rst - * The patch is not too big (you might need to split some patches to improve cohesion and/or reduce size). - * The patch does what the commit message promises - * Unit and functional tests are included and/or updated -* If during the inspection you see a specific line you would like to bring up to discussion in the final review, leave feedback as an inline comment in Gerrit. This will make the review process easier. You can also see the `prefixes you can use`_ for Zaqar inline comments. -* Keep in mind the `Zaqar's Reviewer Guide`_ and be respectful when leaving feedback. -* Hit the "Review" button in the web UI to publish your comments and assign a score. -* Things to consider when leaving a score: - * You can score with a -1 if you think that there are things to fix. We have to be careful to not stall the cycle just because a few nits, so downvoting also depends on the current stage of the development cycle and the severity of the flaw you see. - * You can score with a 0 if you are the author of the fix and you want to respond to the reviewers comments, or if you are a reviewer and you want to point out some reminder for future developing (e.g. the deadline is the next day and the fix needs to be merged, but you want something to be improved). - * You can score with +1 if the fix works and you think that the code looks good, upvoting is your choice. -* Remember to leave any comment that you think is important in the comment form. When you are done, click "Publish Comments". -For more details on how to do a review, check out the Review section in the GerritWorkflow wiki. + * The commit message is formatted appropriately. + Check `Git Commit Messages`_ for more information on how you should + write a git commit message. + * The coding style matches guidelines given in ``HACKING.rst``. + * The patch is not too big. + You might need to split some patches to improve cohesion and/or reduce + size. + * The patch does what the commit message promises. + * Unit and functional tests are included and/or updated. +* If during the inspection you see a specific line you would like to bring up + to discussion in the final review, leave feedback as an inline comment in + Gerrit. This will make the review process easier. You can also use + prefixes described in :doc:`reviewer_guide` for Zaqar inline comments. +* Keep in mind the :doc:`reviewer_guide` and be respectful when leaving + feedback. +* Hit the :guilabel:`Review` button in the web UI to publish your comments + and assign a score. +* Things to consider when leaving a score: + + * You can score with a ``-1`` if you think that there are things to fix. We + have to be careful to not stall the cycle just because a few nits, so + downvoting also depends on the current stage of the development cycle + and the severity of the flaw you see. + * You can score with a "0" if you are the author of the fix and you want to + respond to the reviewers comments, or if you are a reviewer and you want + to point out some reminder for future developing (e.g. the deadline is + the next day and the fix needs to be merged, but you want something to be + improved). + * You can score with ``+1`` if the fix works and you think that the code + looks good, upvoting is your choice. +* Remember to leave any comment that you think is important in the comment + form. When you are done, click :guilabel:`Publish Comments`. + +For more details on how to do a review, check out the `Gerrit Workflow +Review section`_ document. .. _`Open Zaqar fixes`: https://review.openstack.org/#/q/status:open+zaqar,n,z -.. _`Zaqar's Reviewer Guide`: https://wiki.openstack.org/wiki/Zaqar/Reviewer_guide -.. _`prefixes you can use`: https://wiki.openstack.org/wiki/Zaqar/Reviewer_guide#Use_Prefixes .. _`Git Commit Messages`: https://wiki.openstack.org/wiki/GitCommitMessages - +.. _`Gerrit Workflow Review section`: http://docs.openstack.org/infra/manual/developers.html#code-review diff --git a/doc/source/gerrit.rst b/doc/source/gerrit.rst index d3aeac32d..99c0655d9 100644 --- a/doc/source/gerrit.rst +++ b/doc/source/gerrit.rst @@ -1,3 +1,16 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + ======================== Code reviews with Gerrit ======================== @@ -8,10 +21,8 @@ is http://review.openstack.org. Gerrit is a complete replacement for GitHub pull requests. `All GitHub pull requests to the Zaqar repository will be ignored`. -See `Gerrit Workflow Quick Reference`_ for information about how to get -started using Gerrit. See `Development Workflow`_ for more detailed -documentation on how to work with Gerrit. +See `Development Workflow with Gerrit`_ for more detailed documentation on how +to work with Gerrit. -.. _Gerrit: http://code.google.com/p/gerrit -.. _Development Workflow: http://docs.openstack.org/infra/manual/developers.html#development-workflow -.. _Gerrit Workflow Quick Reference: http://docs.openstack.org/infra/manual/developers.html#development-workflow +.. _Gerrit: https://www.gerritcodereview.com/ +.. _Development Workflow with Gerrit: http://docs.openstack.org/infra/manual/developers.html#development-workflow diff --git a/doc/source/index.rst b/doc/source/index.rst index 3bd23af51..fd852ee0a 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -12,7 +12,7 @@ under the License. ============================================= -Welcome to the Zaqar developer documentation! +Welcome to the Zaqar Developer Documentation! ============================================= Zaqar is a multi-tenant cloud messaging and notification service for web @@ -21,15 +21,17 @@ and mobile developers. The service features a REST API, which developers can use to send messages between various components of their SaaS and mobile applications, by using a variety of communication patterns. Underlying this API is an efficient -messaging engine designed with scalability and security in mind. +messaging engine designed with scalability and security in mind. The Websocket +API is also available. -Other OpenStack components can integrate with Zaqar to surface events to -end users and to communicate with guest agents that run in the -"over-cloud" layer. +Other OpenStack components can integrate with Zaqar to surface events to end +users and to communicate with guest agents that run in the "over-cloud" layer. -.. note:: This documentation is generated by the Sphinx toolkit and lives in the Zaqar project source - tree. Additional draft and project documentation regarding Zaqar and other components of OpenStack can - be found on the `OpenStack Wiki`_, as well as in the user guides found on `docs.openstack.org`_. +.. note:: This documentation is generated by the Sphinx toolkit and lives in + the Zaqar project source tree. Additional draft and project documentation + regarding Zaqar and other components of OpenStack can be found on the + `OpenStack Wiki`_, as well as in the user guides found on + `docs.openstack.org`_. .. _`OpenStack Wiki`: http://wiki.openstack.org .. _`docs.openstack.org`: http://docs.openstack.org @@ -39,26 +41,45 @@ Key features Zaqar provides the following key features: -* HTTP-based messaging API -* Multi-tenant design based on Keystone project IDs +* Choice between two communication transports. Both with Keystone support: + + * Firewall-friendly, **HTTP-based RESTful API**. Many of today's developers + prefer a more web-friendly HTTP API. They value the simplicity and + transparency of the protocol, it's firewall-friendly nature, and it's huge + ecosystem of tools, load balancers and proxies. In addition, cloud + operators appreciate the scalability aspects of the REST architectural + style. + * **Websocket-based API** for persistent connections. Websocket protocol + provides communication over persistent connections. Unlike HTTP, where + new connections are opened for each request/response pair, Websocket can + transfer multiple requests/responses over single TCP connection. It saves + much network traffic and minimizes delays. + +* Multi-tenant queues based on Keystone project IDs. * Support for several common patterns including event broadcasting, task - distribution, and point-to-point messaging -* Component-based architecture with support for custom backends and - message filters -* Efficient reference implementation with an eye toward low latency and - high throughput (dependent on backend) -* Highly-available and horizontally scalable + distribution, and point-to-point messaging. +* Component-based architecture with support for custom backends and message + filters. +* Efficient reference implementation with an eye toward low latency and high + throughput (dependent on backend). +* Highly-available and horizontally scalable. +* Support for subscriptions to queues. Several notification types are + available: + + * Email notifications. + * Webhook notifications. + * Websocket notifications. Project scope ============= The Zaqar API is data-oriented. That is, it does not provision message brokers -and expose those directly to clients. Instead, the API acts as a bridge -between the client and one or more backends. A provisioning service for -message brokers—however useful—serves a somewhat different market from what -Zaqar is targeting today. With that in mind, if users are interested in a -broker provisioning service, the community should consider starting a new -project to address that need. +and expose those directly to clients. Instead, the API acts as a bridge between +the client and one or more backends. A provisioning service for message +brokers, however useful, serves a somewhat different market from what Zaqar is +targeting today. With that in mind, if users are interested in a broker +provisioning service, the community should consider starting a new project to +address that need. Design principles ================= @@ -92,17 +113,18 @@ Zaqar is composed of two layers: storage/autoindex The **transport drivers** are responsible for interacting with Zaqar clients. -Every query made by clients is processed by the transport layer, -which is in charge of passing this information to the backend and then -returning the response in a format understandable by the client. +Every query made by clients is processed by the transport layer, which is in +charge of passing this information to the backend and then returning the +response in a format understandable by the client. -The **storage drivers** are responsible for interacting with the storage backends -and, that way, store or retrieve the data coming from the transport layer. +The **storage drivers** are responsible for interacting with the storage +backends and, that way, store or retrieve the data coming from the transport +layer. In order to keep these layers decoupled, we have established that **checks should be performed in the appropriate layer**. In other words, -transport drivers must guarantee that the incoming data is well-formed -and storage drivers must enforce their data model stays consistent. +transport drivers must guarantee that the incoming data is well-formed and +storage drivers must enforce their data model stays consistent. Setting up a development environment ==================================== @@ -112,12 +134,13 @@ Setting up a development environment devref/development.environment -Your first patch and your first review -====================================== +Welcome new contributors +======================== .. toctree:: :maxdepth: 1 + welcome first_patch first_review @@ -131,6 +154,22 @@ Running and writing tests running_tests test_suite +Reviewing +========= + +.. toctree:: + :maxdepth: 1 + + reviewer_guide + +Running benchmark +================= + +.. toctree:: + :maxdepth: 1 + + running_benchmark + Other resources =============== @@ -141,8 +180,8 @@ Other resources gerrit jenkins -API reference -============= +Internal API reference +====================== .. toctree:: :maxdepth: 3 diff --git a/doc/source/jenkins.rst b/doc/source/jenkins.rst index 4ff76a82f..ced58f8d8 100644 --- a/doc/source/jenkins.rst +++ b/doc/source/jenkins.rst @@ -1,14 +1,27 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + =================================== Continuous integration with Jenkins =================================== Zaqar uses a `Jenkins`_ server to automate development tasks. The Jenkins -front-end is at http://jenkins.openstack.org. You must have an -account on `Launchpad`_ to be able to access the OpenStack Jenkins site. +front-end is at http://jenkins.openstack.org. You must have an account on +`Launchpad`_ to be able to access the OpenStack Jenkins site. Jenkins performs tasks such as running static code analysis, running unit tests, and running functional tests. For more details on the jobs being run by -Jenkins, see the code reviews on http://review.openstack.org. Tests are run +Jenkins, see the code reviews on http://review.openstack.org. Tests are run automatically and comments are put on the reviews automatically with the results. diff --git a/doc/source/launchpad.rst b/doc/source/launchpad.rst index 35eee8f6f..e5f4a6cfd 100644 --- a/doc/source/launchpad.rst +++ b/doc/source/launchpad.rst @@ -23,7 +23,7 @@ The developers mailing list address is ``openstack-dev@lists.openstack.org``. This is a common mailing list across all OpenStack projects. To participate in the mailing list: -#. Subscribe at http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev + Subscribe at http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev The mailing list archives are at http://lists.openstack.org/pipermail/openstack-dev. @@ -41,13 +41,14 @@ https://blueprints.launchpad.net/zaqar. Technical support (Answers) --------------------------- -Zaqar uses Launchpad Answers to track Zaqar technical support questions. The Zaqar -Answers page is at https://answers.launchpad.net/zaqar. +Zaqar uses Launchpad Answers to track Zaqar technical support questions. The +Zaqar Answers page is at https://answers.launchpad.net/zaqar. -Note that `Ask OpenStack`_ (which are not hosted on Launchpad) can also -be used for technical support requests. +Note that `Ask OpenStack`_ (which is not hosted on Launchpad) can also be used +for technical support requests. -You can also reach us in #openstack-zaqar at irc.freenode.org. +You can also reach us in ``#openstack-zaqar`` IRC channel at +``irc.freenode.org``. .. _Launchpad: http://launchpad.net .. _Wiki: http://wiki.openstack.org diff --git a/doc/source/reviewer_guide.rst b/doc/source/reviewer_guide.rst new file mode 100644 index 000000000..90517c372 --- /dev/null +++ b/doc/source/reviewer_guide.rst @@ -0,0 +1,165 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +============== +Reviewer Guide +============== + +Overview +-------- + +Our program follows the usual OpenStack review process, albeit with some +important additions (see below). See also: :doc:`first_review`. + +Be Professional +--------------- +The PTL, with the support of the core reviewers, is ultimately responsible for +holding contributors accountable for creating a positive, constructive, and +productive culture. Inappropriate behavior will not be tolerated. +(`Why this is important?`_) + +Do This: + +* Act professionally. +* Treat others as friends and family. +* Seek first to understand. +* Be honest, transparent, and constructive. +* Use clear, concise language. +* Use prefixes to clarify the tone and intent of your comments. + +Don't Do This: + +* Use indecent, profane, or degrading language of any kind. +* Hold a patch hostage for an ulterior motive, political or otherwise. +* Abuse the review system to discuss big issues that would be better hashed out + on the mailing list, in IRC, or during OpenStack Summit design sessions. +* Engage in bullying behaviors, including but not limited to: + + * Belittling others' opinions + * Persistent teasing or sarcasm + * Insulting, threatening, or yelling at someone + * Accusing someone of being incompetent + * Setting someone up to fail + * Humiliating someone + * Isolating someone from others + * Withholding information to gain an advantage + * Falsely accusing someone of errors + * Sabotaging someone's work + +Reviewing Docs +-------------- + +When possible, enlist the help of a professional technical writer to help +review each doc patch. All reviewers should familiarize themselves with +`OpenStack Documentation Contributor Guide`_. When reviewing user guide +patches, please run them through Maven and proof the resulting docs before +giving your ``+1`` or ``+2``. + +Reviewing Code +-------------- + +When reviewing code patches, use your best judgment and seek to provide +constructive feedback to the author. Compliment them on things they have done +well, and highlight possible improvements. Also, dedicate as much time as +necessary in order to provide a careful analysis of the code. Don't assume that +someone else will catch any issues you yourself miss; in other words, pretend +you are the only person reviewing a given patch. Remember, "given enough +eyeballs, all bugs are shallow" ceases to be true the moment individual +reviewers become complacent. + +Some things to check when reviewing code: + +* Patch aligns with project goals, and is ideally associated with a bp or bug. +* Commit message is formatted appropriately and contains external references as + needed. +* Coding style matches guidelines given in ``HACKING.rst``. +* Patch is cohesive and not too big to be reviewed in a timely manner (some + patches may need to be split to improve cohesion and/or reduce size). +* Patch does what the commit message promises. +* Algorithms are implemented correctly, and chosen appropriately. +* Data schemas follow best practices. +* Unit and functional tests have been included and/or updated. +* Code contains no bugs (pay special attention to edge cases that tests may + have missed). + +Use Prefixes +------------ + +We encourage the use of prefixes to clarify the tone and intent of your review +comments. This is one way we try to mitigate misunderstandings that can lead to +bad designs, bad code, and bad blood. + +.. list-table:: **Prefixes** + :widths: 6 80 8 + :header-rows: 1 + + * - Prefix + - What the reviewer is saying + - Blocker? + * - KUDO + - You did a nice job here, and I wanted to point that out. Keep up the + good work! + - No + * - TEST + - I think you are missing a test for this feature, code branch, specific + data input, etc. + - Yes + * - BUG + - I don't think this code does what it was intended to do, or I think + there is a general design flaw here that we need to discuss. + - Yes + * - SEC + - This is a serious security vulnerability and we better address it before + merging the code. + - Yes + * - PERF + - I have a concern that this won't be fast enough or won't scale. Let's + discuss the issue and benchmark alternatives. + - Yes + * - DSQ + - I think there is something critical here that we need to discuss this in + IRC or on the mailing list before moving forward. + - Yes + * - STYLE + - This doesn't seem to be consistent with other code and with + ``HACKING.rst`` + - Yes + * - Q + - I don't understand something. Can you clarify? + - Yes + * - DRY + - This could be modified to reduce duplication of code, data, etc. + See also: `Wikipedia: Don't repeat yourself`_ + - Maybe + * - YAGNI + - This feature or flexibility probably isn't needed, or isn't worth the + added complexity; if it is, we can always add the feature later. See + also: `Wikipedia: You aren't gonna need it`_ + - Maybe + * - NIT + - This is a nitpick that I can live with if we want to merge without + addressing it. + - No + * - IMO + - I'm chiming in with my opinion in response to someone else's comment, or + I just wanted to share an observation. Please take what I say with a + grain of salt. + - No + * - FYI + - I just wanted to share some useful information. + - No + +.. _`Why this is important?` : https://thoughtstreams.io/kgriffs/technical-communities/5060/ +.. _`OpenStack Documentation Contributor Guide` : http://docs.openstack.org/contributor-guide/index.html +.. _`Wikipedia: Don't repeat yourself` : https://en.wikipedia.org/wiki/Don't_repeat_yourself +.. _`Wikipedia: You aren't gonna need it` : https://en.wikipedia.org/wiki/Don't_repeat_yourself \ No newline at end of file diff --git a/doc/source/running_benchmark.rst b/doc/source/running_benchmark.rst new file mode 100644 index 000000000..4a81a4f6b --- /dev/null +++ b/doc/source/running_benchmark.rst @@ -0,0 +1,115 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +================= +Running benchmark +================= + +Introduction +------------ + +This document describes how to run benchmarking tool. + +Zaqar Contributors can use this tool to test how the particular code change +affects Zaqar's performance. + +Usage +----- + +1. First install and run zaqar-server. + + For example, you can setup Zaqar in development environment. + + See :doc:`devref/development.environment`. + +2. In your terminal cd into your local Zaqar repo and install additional + requirements: + + .. code-block:: console + + $ pip install -r bench-requirements.txt + +3. Copy the configuration file to ~/.zaqar: + + .. code-block:: console + + $ cp etc/zaqar-benchmark.conf.sample ~/.zaqar/zaqar-benchmark.conf + +4. In this configuration file specify where zaqar-server can be found: + + .. code-block:: ini + + server_url = http://localhost:8888 + +5. The benchmarking tool needs a set of messages to work with. Specify the path + to the file with messages in the configuration file. Alternatively, put + it in the directory with the configuration file and name it + ``zaqar-benchmark-messages.json``. + As a starting point, you can use the sample file from the etc directory: + + .. code-block:: console + + $ cp etc/zaqar-benchmark-messages.json ~/.zaqar/ + + If the file is not found or no file is specified, a single hard-coded + message is used for all requests. + +6. Run the benchmarking tool using the following command: + + .. code-block:: console + + $ zaqar-bench + + By default, the command will run a performance test for 5 seconds, using one + producer process with 10 greenlet workers, and one observer process with 5 + workers. The consumer role is disabled by default. + + You can override these defaults in the config file or on the command line + using a variety of options. For example, the following command runs a + performance test for 30 seconds using 4 producer processes with 20 + workers each, plus 4 consumer processes with 20 workers each. + + Note that the observer role is also disabled in this example by setting its + number of workers to zero: + + .. code-block:: console + + $ zaqar-bench -pp 4 -pw 10 -cp 4 -cw 20 -ow 0 -t 30 + + By default, the results are in JSON. + For more human-readable output add the :option:`--verbose` flag. + Verbose output looks similar to the following: + + .. code-block:: console + + $ zaqar-bench --verbose + + Starting producer (pp=1 , pw=10)... + + Starting observer (op=1 , ow=5)... + + Producer + ======== + duration_sec: 5.1 + ms_per_req: 2.9 + reqs_per_sec: 344.5 + successful_reqs: 1742.0 + total_reqs: 1742.0 + + Observer + ======== + duration_sec: 5.0 + ms_per_req: 2.9 + reqs_per_sec: 339.3 + successful_reqs: 1706.0 + total_reqs: 1706.0 diff --git a/doc/source/running_tests.rst b/doc/source/running_tests.rst index 028316b7f..212debe80 100644 --- a/doc/source/running_tests.rst +++ b/doc/source/running_tests.rst @@ -1,9 +1,24 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + ============= Running tests ============= Zaqar contains a suite of tests (both unit and functional) in the -``zaqar/tests`` and ``tests`` directories. +``zaqar/tests`` directory. + + See :doc:`test_suite` for details. Any proposed code change is automatically rejected by the OpenStack Jenkins server [#f1]_ if the change causes test failures. @@ -14,53 +29,129 @@ for review. This allows to catch errors as early as possible. Preferred way to run the tests ------------------------------ -The preferred way to run the unit tests is using ``tox``. It executes tests in +The preferred way to run the unit tests is using ``tox``. It executes tests in isolated environment, by creating separate virtualenv and installing dependencies from the ``requirements.txt`` and ``test-requirements.txt`` files, -so the only package you install is ``tox`` itself:: +so the only package you install is ``tox`` itself: - pip install tox +.. code-block:: console + + $ pip install tox See `the unit testing section of the Testing wiki page`_ for more information. Following are some simple examples. -To run the Python 2.6 tests:: +To run the Python 2.7 tests: - tox -e py26 +.. code-block:: console -To run the style tests:: + $ tox -e py27 - tox -e pep8 +To run the style tests: -To run multiple tests separate items by commas:: +.. code-block:: console - tox -e py27,pep8 + $ tox -e pep8 + +To run multiple tests separate items by commas: + +.. code-block:: console + + $ tox -e py27,py34,pep8 .. _the unit testing section of the Testing wiki page: https://wiki.openstack.org/wiki/Testing#Unit_Tests Running a subset of tests -------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^ Instead of running all tests, you can specify an individual directory, file, -class, or method that contains test code. +class or method that contains test code, i.e. filter full names of tests by a +string. -To run the tests located only in the ``tests/unit/queues/storage`` directory use:: +To run the tests located only in the ``zaqar/tests/unit/queues/storage`` +directory use: - tox -e py27 tests.unit.queues.storage +.. code-block:: console -To run the tests specific to the MongoDB driver in the ``tests/unit/queues/storage/test_impl_mongodb.py`` file:: + $ tox -e py27 zaqar.tests.unit.queues.storage - tox -e py27 test_impl_mongodb +To run the tests specific to the MongoDB driver in the +``zaqar/tests/unit/queues/storage/test_impl_mongodb.py`` file: + +.. code-block:: console + + $ tox -e py27 test_impl_mongodb To run the tests in the ``MongodbMessageTests`` class in -the ``tests/unit/queues/storage/test_impl_mongodb.py`` file:: +the ``tests/unit/queues/storage/test_impl_mongodb.py`` file: - tox -e py27 test_impl_mongodb.MongodbMessageTests +.. code-block:: console -To run the `MongodbMessageTests.test_message_lifecycle` test method in -the ``tests/unit/queues/storage/test_impl_mongodb.py`` file:: + $ tox -e py27 test_impl_mongodb.MongodbMessageTests - tox -e py27 test_impl_mongodb.MongodbMessageTests.test_message_lifecycle +To run the ``MongodbMessageTests.test_message_lifecycle`` test method in +the ``tests/unit/queues/storage/test_impl_mongodb.py`` file: + +.. code-block:: console + + $ tox -e py27 test_impl_mongodb.MongodbMessageTests.test_message_lifecycle + +Running functional tests +------------------------ + +Zaqar's functional tests treat Zaqar as a black box. In other words, the API +calls attempt to simulate an actual user. Unlike unit tests, the functional +tests do not use mockendpoints. + +Functional test modes +^^^^^^^^^^^^^^^^^^^^^ + +Functional tests can run in integration mode and non-integration mode. + +Integration mode +"""""""""""""""" + +In integration mode functional tests are performed on Zaqar server instances +running as separate processes. This is real functional testing. + +To run functional tests in integration mode, execute: + +.. code-block:: console + + $ tox -e integration + +Non-integration mode +"""""""""""""""""""" + +In non-integration mode functional tests are performed on Zaqar server +instances running as python objects. This mode doesn't guarantee enough black +boxness for Zaqar, but tests run 10 times faster than in integration mode. + +To run functional tests in non-integration mode, execute: + +.. code-block:: console + + $ tox -e py27 zaqar.tests.functional + +Using a custom MongoDB instance +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you need to run functional tests against a non-default MongoDB installation, +you can set the ``ZAQAR_TEST_MONGODB_URL`` environment variable. For example: + +.. code-block:: console + + $ export ZAQAR_TEST_MONGODB_URL=mongodb://remote-server:27017 + +Using custom parameters +^^^^^^^^^^^^^^^^^^^^^^^ + +You can edit default functional test configuration file +``zaqar/tests/etc/functional-tests.conf`` according to your needs. + +For example, you want to run functional tests with keystone authentication +enabled, input a valid set of credentials to ``[auth]`` section in +configuration file and set ``auth_on`` parameter to ``True``. .. rubric:: Footnotes diff --git a/doc/source/test_suite.rst b/doc/source/test_suite.rst index 3f000c848..94c0268aa 100644 --- a/doc/source/test_suite.rst +++ b/doc/source/test_suite.rst @@ -1,7 +1,23 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + ==================== Test suite structure ==================== +Test types +---------- + There are three types of tests for Zaqar: Unit tests @@ -10,33 +26,71 @@ Unit tests Functional tests Functional tests verify that the service works as expected. In particular, - in Zaqar they exercise the API endpoints and validate that the API responses - conform to the specs. These include positive and negative tests. + in Zaqar they exercise the API endpoints and validate that the API + responses conform to the specs. These include positive and negative tests. Tempest tests Tempest tests are integration tests for OpenStack [#f1]_. - Tempest tests for Zaqar are available at https://github.com/openstack/tempest. -This document focuses on the unit and functional tests. Please refer to the -Tempest repository for details on how to run these tests. + Tempest tests for Zaqar are available in the `Tempest repository`_. + +Refer to :doc:`running_tests` document for details on how to run Unit and +Functional tests. + +Refer to the `Tempest repository`_ for details on how to run Tempest tests. Code structure -------------- -The test suite lives in two directories: +The test suite lives in ``zaqar/tests`` directory of Zaqar: -- ``zaqar/tests`` contains all base classes and defines tests for APIs (on both storage and transport levels). -- ``tests`` usually contains implementations for specific drivers and additional tests. +* ``zaqar/tests/etc`` + Contains various configuration files for Zaqar. They help to test how Zaqar + works in different configurations. -Thus base class and all common tests for storage drivers are located in the ``zaqar/tests/queues/storage/base.py`` file. -The specific instances of the base classes for any particular storage driver are located at the -``tests/unit/queues/storage/`` directory. See ``tests/unit/queues/storage/test_impl_mongodb.py`` for example. +* ``zaqar/tests/functional`` + Contains functional tests. -Similarly, unit tests for the transport layer are located in ``zaqar/tests/queues/transport`` -and are run from classes located in the ``tests/unit/queues/transport`` directory. +* ``zaqar/tests/unit`` + Contains unit tests. -All functional tests for Zaqar are located in the ``tests/functional`` directory. +The base class of all test classes is located in the ``zaqar/tests/base.py`` +file. + +Test invocation +--------------- + +When you run tests via ``tox -e py27`` command in the root directory of Zaqar: + +#. Tox program executes: + + #. Looks for ``tox.ini`` file. + #. Creates ``.tox`` directory for storing python environments. + #. Parses this file and finds parameters for py27 testing environment. + #. Sets this environment up and activates it. + #. Sets environment variables for this environment that are described in + ``tox.ini`` + #. In case of Zaqar it invokes Testr program in the environment. + + You can find more information about Tox in `Openstack Tox testing manual`_ + and in official `Tox documentation`_. + +#. Testr (Test Repository) program executes: + + #. Looks for ``testr.ini`` file. + #. Parses this file and finds parameters for executing tests. + #. Creates ``.testrepository`` directory for storing statistics of + executing tests. + #. In case of Zaqar it invokes ``Subunit`` program which finds all tests and + executes it. + + You can find more information about Testr in `Openstack Testr manual`_. .. rubric:: Footnotes .. [#f1] See http://docs.openstack.org/developer/tempest/overview.html + +.. _`Openstack Tox testing manual` : https://wiki.openstack.org/wiki/Testing#Unit_Testing_with_Tox +.. _`Tox documentation` : https://tox.readthedocs.org/en/latest/ +.. _`Openstack Testr manual` : https://wiki.openstack.org/wiki/Testr +.. _`Tempest repository` : https://github.com/openstack/tempest diff --git a/doc/source/welcome.rst b/doc/source/welcome.rst new file mode 100644 index 000000000..14cc200b9 --- /dev/null +++ b/doc/source/welcome.rst @@ -0,0 +1,187 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +======================== +Welcome new contributors +======================== + +First Steps +=========== + +It's very great that you're interested in contributing to Zaqar. + +First of all, make sure you join Zaqar communication forums: + +* Subscribe to Zaqar `mailing lists`_. +* Join Zaqar team on IRC. You can chat with us directly in the + ``#openstack-zaqar`` channel on ``irc.freenode.org``. If you don't know + how to use IRC, you can find some directions in `Openstack IRC wiki`_. +* Answer and ask questions on `Ask OpenStack`_. + +How can I contribute? +===================== + +There are many ways you can contribute to Zaqar. Of course coding is one, but +you can also join Zaqar as a tester, documenter, designer or translator. + +Coding +------ + +Bug fixing +^^^^^^^^^^ + +The first area where you can help is bug fixing. ``Confirmed`` bugs are usually +your best choice. ``Triaged`` bugs should even contain tips on how they +should be fixed. You can find both of them in +`Zaqar's Confirmed and Triaged bugs`_ web page. + +Once you selected the bug you want to work on, go ahead and assign it to +yourself, branch the code, implement the fix, and propose your change for +review. You can find information on how to do it in +:doc:`first_patch` manual. + +Some easy-to-fix bugs may be marked with the ``low-hanging-fruit`` tag: those +are good targets for a beginner. + +Bug triaging +^^^^^^^^^^^^ + +You can also help Zaqar with bug triaging. Reported bugs need care: +prioritizing them correctly, confirming them, making sure they don't go stale. +All those tasks help immensely. If you want to start contributing in coding, +but you are not a hardcore developer, consider helping in this area. + +Bugs can be marked with different tags according to their status: + +* ``New`` bugs are those bugs that have been reported by a user but haven't + been verified by the community yet. +* ``Confirmed`` bugs are those bugs that have been reproduced by someone else + than the reporter. +* ``Triaged`` bugs are those bugs that have been reproduced by a core + developer. +* ``Incomplete`` bugs are those bugs that don't have enough information to be + reproduced. +* ``In Progress`` bugs are those bugs that are being fixed by some developer. + This status is set automatically by the Gerrit review system once a fix is + proposed by a developer. You don't need to set it manually. +* ``Invalid`` bugs are those bugs that don't qualify as a bug. Usually a + support request or something unrelated to the project. + +You can learn more about this in Launchpad's `Of Bugs and Statuses`_. + +You only have to worry about ``New`` bugs. If you can reproduce them, you can +mark them as ``Confirmed``. If you cannot reproduce them, you can ask the +reported to provide more information and mark them as ``Incomplete``. If you +consider that they aren't bugs, mark them as ``Invalid`` (Be careful with this. +Asking someone else in Zaqar is always a good idea). + +Also, you can contribute instructions on how to fix a given bug. + +Check out the `Bug Triage`_ wiki for more information. + +Reviewing +^^^^^^^^^ + +Every patch submitted to OpenStack gets reviewed before it can be approved and +merged. Zaqar gets a lot of contributions and everyone can (and is encouraged +to) review Zaqar's existing patches. Pick an open review and go through +it, test it if possible, and leave a comment with a ``+1`` or ``-1`` vote +describing what you discovered. If you're planning on submitting patches of +your own, it's a great way to learn about what the community cares about and to +learn about the code base. + +Make sure you read :doc:`first_review` manual. + +Feature development +^^^^^^^^^^^^^^^^^^^ + +Once you get familiar with the code, you can start to contribute new features. +New features get implemented every 6 months in `OpenStack development cycle`_. +We use `Launchpad Blueprints`_ to track the design and implementation of +significant features, and Zaqar team uses Design Summits every 6 months to +get together and discuss things in person with the rest of the community. Code +should be proposed for inclusion before Zaqar reach the final feature milestone +of the development cycle. + +Testing +------- + +Testing efforts are highly related to coding. If you find that there are test +cases missing or that some tests could be improved, you are encouraged to +report it as a bug and then provide your fix. + +See :doc:`running_tests` and :doc:`test_suite` for information on how to run +tests and how the tests are organized in Zaqar. + +See :doc:`first_patch` for information on how to provide your fix. + + +Documenting +----------- + +You can contribute to `Zaqar's Contributor Documentation`_ which you are +currently reading and to `Zaqar's Wiki`_. + +To fix a documentation bug check the bugs marked with the ``doc`` tag in +Zaqar's bug list. In case that you want to report a documentation bug, then +don't forget to add the ``doc`` tag to it. + +`Zaqar's Contributor Documentation`_ is compiled from source files in ``.rst`` +(reStructuredText) format located in ``doc/source/`` directory in Zaqar +repository. The `"openstack-manuals" project`_ houses the documentation that is +published to ``docs.openstack.org``. + +Before contributing to `Zaqar's Contributor Documentation`_ you have to read +:doc:`first_patch` manual and `OpenStack Documentation Contributor Guide`_. + +Also, you can monitor `Ask OpenStack`_ to curate the best answers that can be +folded into the documentation. + +Designing +--------- + +Zaqar doesn't have a user interface yet. Zaqar team is working to +`integrate Zaqar to the OpenStack Dashboard (Horizon)`_. + +If you're a designer or usability professional your help will be really +appreciated. Whether it's reviewing upcoming features as a user and giving +feedback, designing features, testing designs or features with users, or +helping to build use cases and requirements, everything is useful. + +Translating +----------- + +You can translate Zaqar to language you know. +Read the `Translation wiki page`_ for more information on how OpenStack manages +translations. Zaqar has adopted Zanata, and you can use the +`OpenStack Zanata site`_ as a starting point to translate any of the OpenStack +projects, including Zaqar. It's easier to start translating directly on the +`OpenStack Zanata site`_, as there is no need to download any files or +applications to get started. + + +.. _`mailing lists` : https://wiki.openstack.org/wiki/MailingLists +.. _`Openstack IRC wiki` : https://wiki.openstack.org/wiki/IRC +.. _`Ask OpenStack` : https://ask.openstack.org/ +.. _`Zaqar's Confirmed and Triaged bugs` : https://bugs.launchpad.net/zaqar/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on +.. _`Of Bugs and Statuses` : http://blog.launchpad.net/general/of-bugs-and-statuses +.. _`Bug Triage` : https://wiki.openstack.org/wiki/BugTriage +.. _`OpenStack development cycle` : https://wiki.openstack.org/wiki/ReleaseCycle +.. _`Launchpad Blueprints` : https://wiki.openstack.org/wiki/Blueprints +.. _`OpenStack Documentation Contributor Guide` : http://docs.openstack.org/contributor-guide/index.html +.. _`Zaqar's Contributor Documentation` : http://docs.openstack.org/developer/zaqar/ +.. _`Zaqar's Wiki` : https://wiki.openstack.org/wiki/Zaqar +.. _`"openstack-manuals" project` : https://wiki.openstack.org/wiki/Documentation +.. _`integrate Zaqar to the OpenStack Dashboard (Horizon)` : https://blueprints.launchpad.net/zaqar/+spec/zaqar-horizon-integration +.. _`Translation wiki page` : https://wiki.openstack.org/wiki/Translations#Translation_.26_Management +.. _`OpenStack Zanata site` : https://translate.openstack.org/ diff --git a/zaqar/tests/functional/README.rst b/zaqar/tests/functional/README.rst deleted file mode 100644 index dc22f95bc..000000000 --- a/zaqar/tests/functional/README.rst +++ /dev/null @@ -1,68 +0,0 @@ -Zaqar Functional Tests -====================== - -Zaqar's functional tests treat Zaqar as a black box. In other -words, the API calls attempt to simulate an actual user. Unlike unit tests, -the functional tests do not use mockendpoints. - - -Running functional tests (With Tox) ------------------------------------ - -#. Setup a Zaqar server. Refer to the Zaqar `README`_ on - how to run Zaqar locally, or simply use an existing server. - -#. Run tests. :: - - $ tox - -#. Filter tests. :: - - $ tox -- zaqar.tests.functional.wsgi.v1.test_messages - -#. Run tests for specific environments. :: - - $ tox -epy27,pep8 - -Running the Functional Tests (Without Tox) ------------------------------------------- - -#. Setup a Zaqar server. Refer to the Zaqar `README`_ on - how to run Zaqar locally, or simply use an existing server. - -#. Install functional tests dependencies. :: - - pip install -r requirements.txt - pip install -r test-requirements.txt - -#. cd to the tests/etc directory - -#. If leaving keystone auth enabled, update functional-tests.conf with a - valid set of credentials. - -#. Now, to run the system tests, simply use the nosetests commands, e.g.: - - Run all test suites: :: - - nosetests -v - -Adding New Tests ----------------- - -#. Add test case to an appropriate test case file: :: - - queue/test_queue.py - messages/test_messages.py - claim/test_claims.py - -Using a custom MongoDB instance -------------------------------- - -If you need to run the tests against a non-default MongoDB installation, you -can set the ZAQAR_TEST_MONGODB_URL environemment variable. For example: :: - - export ZAQAR_TEST_MONGODB_URL=mongodb://remote-server:27017 - - -.. _README : https://github.com/openstack/zaqar/blob/master/README.rst -.. _requests : https://pypi.python.org/pypi/requests