openstack/zaqar
Code Issues Proposed changes

Refactoring of docs during Mitaka cycle

Browse Source 
This patch improves Zaqar documentation and fixes currently existing
bugs.

Bugs this patch currently addresses and solutions:

Short names for documentation locations used in this commit message:
GitRepo - https://github.com/openstack/zaqar/
Contributor Docs - http://docs.openstack.org/developer/zaqar/
Wiki - https://wiki.openstack.org/wiki/Zaqar/

1. DRY violation and spreaded information for contributors bug.
The information for Zaqar contributors is spreaded/duplicated across
GitRepo, Contributor Docs and Wiki.
Examples of DRY violation are these three articles:
https://wiki.openstack.org/wiki/Zaqar/Give_Zaqar_a_try,
https://github.com/openstack/zaqar/blob/master/README.rst,
http://docs.openstack.org/developer/zaqar/development-environment.html

Example of spreaded information is: "zaqar/tests/functional/README.rst"
Normally the contributor want to see the information from this file in
"doc/source/running_tests.rst".

Solution: move useful missing information for contributors from Wiki and
GitRepo to Contributor Docs, then replace all contributor information in
Wiki and GitRepo with links to Contributor Docs.

2. Outdated information, missing new information and broken links bug.
Example is "test_suite.rst":
a. It still states that Zaqar test suite lives in two directories -
"tests" and "zaqar/tests", but now it's not true.
b. Doesn't contain information about how test invocation is organized,
what really happens when "tox -e py27" command executes.

Solution: replace outdated information with new, fix broken links if
possible, add useful missing information.

3. Style and formatting bugs.
The reference is http://docs.openstack.org/contributor-guide/.

Many documents in Contributor Docs have wrong line wrapping - some lines
are wrapped too short and some are wrapped too long.
Lines must wrap at 79 characters, exceptions are code and links.
Example is "first_review.rst" which lines are not wrapped at all.

Enumerated lists must be written using "#. " syntax.
Example with wrong enumerated list is "development.environment.rst".

Some inline elements must be styled according to:
http://docs.openstack.org/contributor-guide/rst-conv/inline-markups.html
Example with wrong styling of inline elements is
"development.environment.rst" where many paths and file names are not
marked with `` (double backticks).

By default code inserts are implicitly styled with python syntax.
There are many places in Contributor Docs where console (bash) code is
wrongly styled with python syntax.

Also there are some failed attempts to apply a formatting in Contributor
Docs. For example there is a broken list in "first_review.rst" at line
52.

Solution: fix broken formatting, apply proper style where it is needed.

Some of these bugs fixes closes few bug reports from:
https://etherpad.openstack.org/p/zaqar-mitaka-docs

Change-Id: Id668684248bdee03eb43b537dc2c6bb2a68ed23d
This commit is contained in:
Eva Balycheva 2015-11-27 04:31:58 +03:00
parent 7ea60d0e8d
commit b53ff5d12c
14 changed files with 1198 additions and 472 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

190
README.rst
View File

@ -2,179 +2,63 @@
Zaqar
=====
Message queuing service for `OpenStack`_.
To find more information read our `wiki`_.
Zaqar is a multi-tenant cloud messaging and notification service for web
and mobile developers.
It combines the ideas pioneered by Amazon's SQS product with additional
semantics to support event broadcasting.
Running a local Zaqar server with MongoDB
-----------------------------------------
The service features a fully RESTful API, which developers can use to send
messages between various components of their SaaS and mobile applications, by
using a variety of communication patterns. Underlying this API is an efficient
messaging engine designed with scalability and security in mind.
**Note:** These instructions are for running a local instance of Zaqar and not
all of these steps are required. It is assumed you have `MongoDB`_ and `tox`
(see "Running tests" section below) installed and running.
Other OpenStack components can integrate with Zaqar to surface events to end
users and to communicate with guest agents that run in the "over-cloud" layer.
Cloud operators can leverage Zaqar to provide equivalents of SQS and SNS to
their customers.
1. Install prerequisites:
General information is available in wiki:
# Ubuntu/Debian:
sudo apt-get install gcc python-pip libxml2-dev libxslt1-dev python-dev zlib1g-dev
https://wiki.openstack.org/wiki/Zaqar
# Fedora/RHEL:
sudo yum install gcc python-pip libxml2-devel libxslt-devel python-devel
The API v1.1 (stable) specification and documentation are available at:
2. From your home folder create the ``~/.zaqar`` folder and clone the repo::
https://wiki.openstack.org/wiki/Zaqar/specs/api/v1.1
$ cd
$ mkdir ~/.zaqar
$ git clone https://git.openstack.org/openstack/zaqar.git
Zaqar Contributor Documentation, the source of which is in ``doc/source/``, is
available at:
3. Generate and copy the Zaqar config files to the directory ``~/.zaqar``::
http://docs.openstack.org/developer/zaqar/
$ pip install tox
$ cd zaqar
$ tox -e genconfig
$ cp etc/zaqar.conf.sample ~/.zaqar/zaqar.conf
$ cp etc/logging.conf.sample ~/.zaqar/logging.conf
Contributors are encouraged to join IRC (``#openstack-zaqar`` channel on
``irc.freenode.net``):
4. Find ``[drivers]`` section in ``~/.zaqar/zaqar.conf``
and specify to use mongodb storage::
https://wiki.openstack.org/wiki/IRC
message_store = mongodb
management_store = mongodb
Information on how to run unit and functional tests is available at:
Then find the ``[drivers:message_store:mongodb]`` and
``[drivers:management_store:mongodb]`` sections and
specify the URI to point to your local
mongod instance by adding this line to both the
sections::
http://docs.openstack.org/developer/zaqar/running_tests.html
uri = mongodb://$MONGODB_HOST:$MONGODB_PORT
Information on how to run benchmarking tool is available at:
By default, you will have::
http://docs.openstack.org/developer/zaqar/running_benchmark.html
uri = mongodb://127.0.0.1:27017
Using Zaqar
-----------
NOTE: If your local dev/test mongodb doesn't enable the replica set, then
you have to set below in [default] section::
If you are new to Zaqar and just want to try it, you can set up Zaqar in
the development environment.
unreliable = True
Using Zaqar in production environment:
5. For logging, find the ``[handler_file]`` section in
``~/.zaqar/logging.conf`` and modify as desired::
Coming soon!
args=('zaqar.log', 'w')
Using Zaqar in development environment:
6. Change directories back to your local copy of the repo::
The instruction is available at:
$ cd ~/zaqar
http://docs.openstack.org/developer/zaqar/devref/development.environment.html
7. Run the following so you can see the results of any changes you
make to the code without having to reinstall the package each time::
$ pip install -e .
8. Start the Zaqar server with logging level set to INFO so you can see
the port on which the server is listening::
$ zaqar-server -v
9. Test out that Zaqar is working by creating a queue::
$ ZQ_CLIENT_ID=`uuidgen`
$ curl -i -X PUT http://127.0.0.1:8888/v1.1/queues/samplequeue \
-H "Content-type: application/json" \
-H "Client-ID: $ZQ_CLIENT_ID" \
-H "X-PROJECT-ID: default"
You should get an **HTTP 201** along with some headers that will look
similar to this::
HTTP/1.0 201 Created
Date: Fri, 25 Oct 2013 15:34:37 GMT
Server: WSGIServer/0.1 Python/2.7.3
Content-Length: 0
Location: /v1.1/queues/samplequeue
Running tests
-------------
Run tests using the following command::
$ tox -e py27
You can read more about running functional tests in separate `TESTS_README`_.
Running the benchmarking tool
-----------------------------
First install and run zaqar-server (see above).
Then install additional requirements::
$ pip install -r bench-requirements.txt
Copy the configuration file to ``~/.zaqar``::
$ cp etc/zaqar-benchmark.conf.sample ~/.zaqar/zaqar-benchmark.conf
In the configuration file specify where zaqar-server can be found::
server_url = http://localhost:8888
The benchmarking tool needs a set of messages to work with. Specify the path
to the file with messages in the configuration file. Alternatively, put it in
the directory with the configuration file and name it ``zaqar-benchmark-
messages.json``. As a starting point, you can use the sample file from the
``etc`` directory::
$ cp etc/zaqar-benchmark-messages.json ~/.zaqar/
If the file is not found or no file is specified, a single hard-coded message
is used for all requests.
Run the benchmarking tool using the following command::
$ zaqar-bench
By default, the command will run a performance test for 5 seconds, using one
producer process with 10 greenlet workers, and one observer process with
5 workers. The consumer role is disabled by default.
You can override these defaults in the config file or on the command line
using a variety of options. For example, the following command runs a
performance test for 30 seconds using 4 producer processes with
20 workers each, plus 4 consumer processes with 20 workers each. Note that
the observer role is also disabled in this example by setting its number of
workers to zero::
$ zaqar-bench -pp 4 -pw 10 -cp 4 -cw 20 -ow 0 -t 30
By default, the results are in JSON. For more human-readable output add
the ``--verbose`` flag. Verbose output looks similar to the following::
$ zaqar-bench --verbose
Starting producer (pp=1 , pw=10)...
Starting observer (op=1 , ow=5)...
Producer
========
duration_sec: 5.1
ms_per_req: 2.9
reqs_per_sec: 344.5
successful_reqs: 1742.0
total_reqs: 1742.0
Observer
========
duration_sec: 5.0
ms_per_req: 2.9
reqs_per_sec: 339.3
successful_reqs: 1706.0
total_reqs: 1706.0
.. _`OpenStack` : http://openstack.org/
.. _`MongoDB` : http://docs.mongodb.org/manual/installation/
.. _`wiki` : https://wiki.openstack.org/wiki/Zaqar
.. _`TESTS_README` : https://github.com/openstack/zaqar/blob/master/zaqar/tests/functional/README.rst
This will allow you to run local Zaqar server with MongoDB as database.
This way is the easiest, quickest and most suitable for beginners.

222
doc/source/devref/development.environment.rst
View File

@ -15,10 +15,10 @@
Setting up a development environment
====================================
This section describes how to setup a working Python development
environment that you can use in developing Zaqar on Ubuntu or Fedora.
These instructions assume that you are familiar with
Git. Refer to GettingTheCode_ for additional information.
This section describes how to setup a working Python development environment
that you can use in developing Zaqar on Ubuntu or Fedora. These instructions
assume that you are familiar with Git. Refer to GettingTheCode_ for
additional information.
.. _GettingTheCode: http://wiki.openstack.org/GettingTheCode
@ -26,20 +26,18 @@ Git. Refer to GettingTheCode_ for additional information.
Virtual environments
--------------------
Use virtualenv_ to track and manage Python dependencies
for developing and testing Zaqar.
Using virtualenv_ enables you to install Python dependencies
in an isolated virtual environment, instead of installing the
packages at the system level.
Use virtualenv_ to track and manage Python dependencies for developing and
testing Zaqar.
Using virtualenv_ enables you to install Python dependencies in an isolated
virtual environment, instead of installing the packages at the system level.
.. _virtualenv: http://pypi.python.org/pypi/virtualenv
.. note::
Virtualenv is useful for development purposes, but is not
typically used for full integration testing or production usage.
If you want to learn about production best practices, check out
the `OpenStack Operations Guide`_.
Virtualenv is useful for development purposes, but is not typically used for
full integration testing or production usage. If you want to learn about
production best practices, check out the `OpenStack Operations Guide`_.
.. _`OpenStack Operations Guide`: http://docs.openstack.org/ops/
@ -48,21 +46,22 @@ Install GNU/Linux system dependencies
.. note::
This section is tested for Zaqar on Ubuntu 14.04 (Trusty) and
Fedora-based (RHEL 6.1) distributions. Feel free to add notes
and change according to your experiences or operating system.
Learn more about contributing to Zaqar documentation in the
`Write the Docs!`_ wiki.
.. _`Write the Docs!`: https://wiki.openstack.org/wiki/Write_the_Docs!_(Zaqar)
This section is tested for Zaqar on Ubuntu 14.04 (Trusty) and Fedora-based
(RHEL 6.1) distributions. Feel free to add notes and change according to your
experiences or operating system. Learn more about contributing to Zaqar
documentation in the :doc:`../welcome` manual.
Install the prerequisite packages.
On Ubuntu::
On Ubuntu:
.. code-block:: console
$ sudo apt-get install gcc python-pip libxml2-dev libxslt1-dev python-dev zlib1g-dev
On Fedora-based distributions (e.g., Fedora/RHEL/CentOS)::
On Fedora-based distributions (e.g., Fedora/RHEL/CentOS):
.. code-block:: console
$ sudo yum install gcc python-pip libxml2-devel libxslt-devel python-devel
@ -73,7 +72,8 @@ You also need to have MongoDB_ installed and running.
.. _MongoDB: http://www.mongodb.org
On Ubuntu, follow the instructions in the `MongoDB on Ubuntu Installation Guide`_.
On Ubuntu, follow the instructions in the
`MongoDB on Ubuntu Installation Guide`_.
.. _`MongoDB on Ubuntu installation guide`: http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/
@ -82,115 +82,217 @@ On Fedora-based distributions, follow the instructions in the
.. _`MongoDB on Red Hat Enterprise, CentOS, Fedora, or Amazon Linux installation guide`: http://docs.mongodb.org/manual/tutorial/install-mongodb-on-red-hat-centos-or-fedora-linux/
.. note::
If you are Contributor and plan to run Unit tests on Zaqar, you may want to
add this line to mongodb configuration file (``etc/mongod.conf`` or
``etc/mongodb.conf`` depending on distribution):
.. code-block:: ini
smallfiles = true
Many Zaqar's Unit tests do not clean up their testing databases after
executing. And database files consume much disk space even if they do not
contain any records. This behavior will be fixed soon.
Getting the code
################
Get the code from GitHub::
Get the code from GitHub to create a local repository with Zaqar:
.. code-block:: console
$ git clone https://github.com/openstack/zaqar.git
Configuration
#############
1. From your home folder create the ~/.zaqar folder. This directory holds the configuration files for Zaqar::
#. From your home folder create the ``~/.zaqar`` folder. This directory holds
the configuration files for Zaqar:
.. code-block:: console
$ mkdir ~/.zaqar
2. Generate the sample configuration file zaqar/etc/zaqar.conf.sample::
#. Generate the sample configuration file ``zaqar/etc/zaqar.conf.sample``:
.. code-block:: console
$ pip install tox
$ cd zaqar
$ tox -e genconfig
3. Copy the Zaqar configuration samples to the directory ~/.zaqar/::
#. Copy the Zaqar configuration samples to the directory ``~/.zaqar/``:
.. code-block:: console
$ cp etc/zaqar.conf.sample ~/.zaqar/zaqar.conf
$ cp etc/logging.conf.sample ~/.zaqar/logging.conf
4. Find the [drivers] section in ~/.zaqar/zaqar.conf and specify mongodb as the message store::
#. Find the ``[drivers]`` section in ``~/.zaqar/zaqar.conf`` and specify
``mongodb`` as the message store:
.. code-block:: ini
message_store = mongodb
management_store = mongodb
5. Find the [drivers:message_store:mongodb] section and modify the URI to point to your local mongod instance::
#. Then find ``[drivers:message_store:mongodb]`` and
``[drivers:management_store:mongodb]`` sections and specify the
:samp:`{URI}` to point to your local mongodb instance by adding this line
to both the sections:
uri = mongodb://$MONGODB_HOST:$MONGODB_PORT # default = mongodb://localhost:27017
.. code-block:: ini
6. For logging, find the [handler_file] section in ~/.zaqar/logging.conf and modify as desired::
uri = mongodb://$MONGODB_HOST:$MONGODB_PORT
By default you will have:
.. code-block:: ini
uri = mongodb://127.0.0.1:27017
This :samp:`{URI}` points to single mongodb node which of course is not
reliable, so you need to set in the ``[default]`` section of configuration
file:
.. code-block:: ini
unreliable = True
For your reference, you can omit this parameter or set it to False only
if the provided :samp:`{URI}` to your mongodb is actually the URI to mongodb
Replica Set or Mongos. Also it must have "Write concern" parameter set to
``majority`` or to a number more than ``1``.
For example, :samp:`{URI}` to reliable mongodb can look like this:
.. code-block:: ini
uri = mongodb://mydb0,mydb1,mydb2:27017/?replicaSet=foo&w=2
Where ``mydb0``, ``mydb1``, ``mydb2`` are addresses of the configured
mongodb Replica Set nodes, ``replicaSet`` (Replica Set name) parameter is
set to ``foo``, ``w`` (Write concern) parameter is set to ``2``.
#. For logging, find the ``[handler_file]`` section in
``~/.zaqar/logging.conf`` and modify as desired:
.. code-block:: ini
args=('zaqar.log', 'w')
Installing and using virtualenv
###############################
1. Install virtualenv by running::
#. Install virtualenv by running:
.. code-block:: console
$ pip install virtualenv
2. Create and activate a virtual environment::
#. Create and activate a virtual environment:
.. code-block:: console
$ virtualenv zaqarenv
$ source zaqarenv/bin/activate
3. Install Zaqar::
#. Install Zaqar:
.. code-block:: console
$ pip install -e .
4. Install the required Python binding for MongoDB::
#. Install the required Python binding for MongoDB:
.. code-block:: console
$ pip install pymongo
5. Start the Zaqar server::
#. Start Zaqar server in ``info`` logging mode:
.. code-block:: console
$ zaqar-server -v
6. Verify Zaqar is running by creating a queue::
Or you can start Zaqar server in ``debug`` logging mode:
.. code-block:: console
$ zaqar-server -d
#. Verify Zaqar is running by creating a queue via curl. In a separate
terminal run:
.. code-block:: console
$ curl -i -X PUT http://localhost:8888/v1/queues/samplequeue -H "Content-type: application/json"
7. Get ready to code!
#. Get ready to code!
.. note::
You can run the Zaqar server in the foreground by passing the
--nodaemon flag::
You can run the Zaqar server in the background by passing the
:option:`--daemon` flag:
$ zaqar-server -v --nodaemon
.. code-block:: console
With this method you get immediate visual feedback and it is
easier to kill and restart the process.
$ zaqar-server -v --daemon
If you do so, you have to run the cURL test (step 5) in a
separate terminal.
But with this method you will not get immediate visual feedback and it will
be harder to kill and restart the process.
Troubleshooting
^^^^^^^^^^^^^^^
No handlers found for zaqar.client (...)
""""""""""""""""""""""""""""""""""""""""
This happens because the current user cannot create the log file (for the
default configuration in ``/var/log/zaqar/server.log``). To solve it, create
the folder:
.. code-block:: console
$ sudo mkdir /var/log/zaqar
Create the file:
.. code-block:: console
$ sudo touch /var/log/zaqar/server.log
And try running the server again.
DevStack
--------
If you want to use Zaqar in an integrated OpenStack developing
environment, you can add it to your DevStack_ deployment.
If you want to use Zaqar in an integrated OpenStack developing environment, you
can add it to your DevStack_ deployment.
To do this, you first need to add the following setting
to your local.conf::
To do this, you first need to add the following setting to your ``local.conf``:
.. code-block:: ini
enable_plugin zaqar https://github.com/openstack/zaqar
Then run the stack.sh script as usual.
After running the DevStack_ script, you can start the Zaqar server
and test it by following steps 5 and 6 from the previous section.
Then run the ``stack.sh`` script as usual.
.. _DevStack: http://docs.openstack.org/developer/devstack/
Running unit tests
------------------
Running tests
-------------
See :doc:`../running_tests` for details.
Running the benchmarking tool
-----------------------------
See :doc:`../running_benchmark` for details.
Contributing your work
----------------------
Once your work is complete, you may wish to contribute it to the project.
Zaqar uses the Gerrit code review system. For information on how to submit
your branch to Gerrit, see GerritWorkflow_.
.. _GerritWorkflow: http://docs.openstack.org/infra/manual/developers.html#development-workflow
See :doc:`../welcome` and :doc:`../first_patch` for details.

208
doc/source/first_patch.rst
View File

@ -25,13 +25,16 @@ Create your contributor accounts and set up your code environment
Accounts setup
##############
You will need to create a Launchpad_ account to login to the Gerrit_ review system dashboard.
This is also useful for automatically crediting bug fixes to you when you address them
with your code commits. You will also have to sign the `Contributors License Agreement`_ and `join the OpenStack Foundation`_. It is a good idea to use the same email all of these accounts to
You will need to create a Launchpad_ account to login to the Gerrit_ review
system dashboard.
This is also useful for automatically crediting bug fixes to you when you
address them with your code commits. You will also have to sign the
`Contributors License Agreement`_ and `join the OpenStack Foundation`_.
It is a good idea to use the same email all of these accounts to
avoid hooks errors.
Visit the the `Gerrit Workflow's account setup`_ section in the wiki
to get more information on setting up your accounts.
Visit the the `Gerrit Workflow's account setup`_ section in the wiki to get
more information on setting up your accounts.
.. _Launchpad: http://launchpad.net/
.. _Gerrit: http://review.openstack.org/
@ -42,14 +45,18 @@ to get more information on setting up your accounts.
SSH setup
#########
You are going to need to create and upload an SSH key to Gerrit to be
able to commit changes for review. To create an SSH key::
You are going to need to create and upload an SSH key to Gerrit to be able to
commit changes for review. To create an SSH key:
.. code-block:: console
$ ssh-keygen –t rsa
You can optionally enter a password to enchance security.
You can optionally enter a password to enhance security.
View and copy your SSH key::
View and copy your SSH key:
.. code-block:: console
$ less ~/.ssh/id_rsa.pub
@ -60,23 +67,31 @@ Now you can `upload the SSH key to Gerrit`_.
Git Review installation
#######################
Before you start working, make sure you have git-review installed on your system.
You can install it with the following command::
Before you start working, make sure you have ``git-review`` installed on your
system.
You can install it with the following command:
.. code-block:: console
$ pip install git-review
Git-review checks if you can authenticate to Gerrit with your SSH key.
It will ask you for your username. You can configure your Gerrit username so you don't have to
keep re-entering it every time you want to use git-review::
``Git-review`` checks if you can authenticate to Gerrit with your SSH key.
It will ask you for your username. You can configure your Gerrit username so
you don't have to keep re-entering it every time you want to use
``git-review``:
.. code-block:: console
$ git config --global gitreview.username yourgerritusername
You can also save some time by entering your email and your name::
You can also save some time by entering your email and your name:
.. code-block:: console
$ git config --global gitreview.email "yourgerritemail"
$ git config --global gitreview.name "Firstname Lastname"
You can view your Gerrit user name in the `settings page`_.
.. _`settings page`: https://review.openstack.org/#/settings/
@ -84,29 +99,39 @@ You can view your Gerrit user name in the `settings page`_.
Project setup
#############
Clone the Zaqar repository with the following git command::
Clone the Zaqar repository with the following git command:
.. code-block:: console
$ git clone git://git.openstack.org/openstack/zaqar.git
For information on how to set up the Zaqar development environment
see :doc:`devref/development.environment`.
Before writing code, you will have to do some configurations
to connect your local repository with Gerrit. You will only need to do this
your first time setting up the development environment.
Before writing code, you will have to do some configurations to connect your
local repository with Gerrit. You will only need to do this your first time
setting up the development environment.
You can set ``git-review`` to configure the project and install the Gerrit
change-id commit hook with the following command:
.. code-block:: console
You can set git-review to configure the project and install the gerrit change-id commit hook with the following command::
$ cd zaqar
$ git review -s
If you get the error "We don't know where your Gerrit is", you will need to
add a new git remote. The URL should be in the error message. Copy that and
create the new remote. It looks something like::
If you get the error "We don't know where your Gerrit is", you will need to add
a new git remote. The URL should be in the error message. Copy that and create
the new remote. It looks something like:
.. code-block:: console
$ git remote add gerrit ssh://<username>@review.openstack.org:29418/openstack/zaqar.git
In the project directory you have a hidden .git directory and a
.gitreview file. You can view them with the following command::
In the project directory you have a hidden ``.git`` directory and a
``.gitreview`` file. You can view them with the following command:
.. code-block:: console
$ ls -la
@ -117,15 +142,18 @@ Pick or report a bug
####################
You can start tackling some bugs from the `bugs list in Launchpad`_.
If you find a bug you want to work on, assign yourself. Make sure to read
the bug report. If you need more information, ask the reporter to provide
more details through a comment on Launchpad or through IRC or email.
If you find a bug you want to work on, assign yourself. Make sure to read the
bug report. If you need more information, ask the reporter to provide more
details through a comment on Launchpad or through IRC or email.
If you find a bug, look through Launchpad to see if it has been reported. If it hasn't, report the bug, and
ask for another developer to confirm it. You can start working on it if another developer confirms the bug.
If you find a bug, look through Launchpad to see if it has been reported. If it
hasn't, report the bug, and ask for another developer to confirm it. You can
start working on it if another developer confirms the bug.
Here are some details you might want to include when filling out a bug report:
* The release, or milestone, or commit ID corresponding to the software that you are running
* The release, or milestone, or commit ID corresponding to the software that
you are running
* The operating system and version where you've identified the bug
* Steps to reproduce the bug, including what went wrong
* Description of the expected results instead of what you saw
@ -142,25 +170,68 @@ You can read more about `Launchpad bugs`_ in the wiki.
Workflow
########
Make sure your repo is up to date. You can update it with the following git commands::
Make sure your repo is up to date. You can update it with the following git
commands:
.. code-block:: console
$ git remote update
$ git checkout master
$ git pull --ff-only origin master
Create a topic branch. You can create one with the following git command::
Create a topic branch. You can create one with the following git command:
.. code-block:: console
$ git checkout -b TOPIC-BRANCH
If you are working on a blueprint, name your topic branch bp/BLUEPRINT where BLUEPRINT
is the name of a blueprint in Launchpad (for example, "bp/authentication"). The general
convention when working on bugs is to name the branch bug/BUG-NUMBER
(for example, "bug/1234567").
If you are working on a blueprint, name your :samp:`{TOPIC-BRANCH}`
``bp/BLUEPRINT`` where :samp:`{BLUEPRINT}` is the name of a blueprint in
Launchpad (for example, "bp/authentication"). The general convention when
working on bugs is to name the branch ``bug/BUG-NUMBER`` (for example,
"bug/1234567").
Read more about the commit syntax in the `Gerrit workflow`_ wiki.
.. _`Gerrit workflow`: http://docs.openstack.org/infra/manual/developers.html#development-workflow
Common problems
^^^^^^^^^^^^^^^
#. You realized that you were working in master and you haven't made any
commits. Solution:
.. code-block:: console
$ git checkout -b newbranch
$ git commit -a -m "Edited"
If you already created the branch, omit the :option:`-b`.
You put all your changes to :samp:`{newbranch}`. Problem solved.
#. You realized that you were working in master and you have made commits to
master. Solution:
.. code-block:: console
$ git branch newbranch
$ git reset --hard HEAD~x
$ git checkout newbranch
Where ``x`` is the number of commits you have made to master.
And remember, you will lose any uncommitted work.
You put your commits in :samp:`{newbranch}`. Problem solved.
#. You made multiple commits and realized that Gerrit requires one commit per
patch. Solution:
* You need to squash your previous commits. Make sure you are in your
branch and follow `squashing guide`_. Then fill commit message properly.
You squashed your commits. Problem solved.
Design principles
#################
@ -192,37 +263,58 @@ Your commit message should:
* Insert a single blank line after the first line.
* Provide a detailed description of the change in the following lines,
breaking paragraphs where needed.
* The first line should be limited to 50 characters and should not end with a period.
* The first line should be limited to 50 characters and should not end with a
period.
* Subsequent lines should be wrapped at 72 characters.
* Put the 'Change-id', 'Closes-Bug #NNNNN' and 'blueprint NNNNNNNNNNN'
lines at the very end.
Read more about `making a good commit message`_.
To submit it for review use the following git command::
To submit it for review use the following git command:
.. code-block:: console
$ git review
You will see the URL of your review page once it is successfully sent. You can also see your
reviews in "My Changes" in Gerrit. The first thing to watch for is a +1 in the "Verified"
column next to your patch in the server and/or client list of pending patches.
If the "Jenkins" user gives you a -1, you'll need to check the log it posts to
find out what gate test failed, update your patch, and resubmit.
You will see the URL of your review page once it is successfully sent.
You can set your patch as a "work in progress" if your patch is not ready to be merged,
but you would still like some feedback from other developers. To do this
leave a review on your patch setting "Workflow" to -1.
You can also see your reviews in :guilabel:`My Changes` in Gerrit. The first
thing to watch for is a ``+1`` in the :guilabel:`Verified` column next to your
patch in the server and/or client list of pending patches.
Once the gate has verified your patch, other Zaqar devs will take a look and submit
their comments. When you get two or more +2's from core reviewers,
the patch will be approved and merged.
If the "Jenkins" user gives you a ``-1``, you'll need to check the log it posts
to find out what gate test failed, update your patch, and resubmit.
Don't be discouraged if a reviewer submits their comments with a "-1".
Patches iterate through several updates and reviews before they are ready for merging.
To reply to feedback save all your comments as draft, then click on the "Review" button.
When replying to feedback, you as the patch author can use the score of "0".
The only exception to using the score of "0" is when you discover a blocking issue
and you don't want your patch to be merged. In which case, you can review your own
patch with a "-2", while you decide whether to keep, refactor, or withdraw the patch.
You can set your patch as a :guilabel:`work in progress` if your patch is
not ready to be merged, but you would still like some feedback from other
developers. To do this leave a review on your patch setting
:guilabel:`Workflow` to ``-1``.
Once the gate has verified your patch, other Zaqar developers will take a look
and submit their comments. When you get two or more ``+2``'s from core
reviewers, the patch will be approved and merged.
Don't be discouraged if a reviewer submits their comments with a ``-1``.
Patches iterate through several updates and reviews before they are ready for
merging.
To reply to feedback save all your comments as draft, then click on the
:guilabel:`Review` button. When replying to feedback, you as the patch
author can use the score of ``0``. The only exception to using the score of
``0`` is when you discover a blocking issue and you don't want your patch to
be merged. In which case, you can review your own patch with a ``-2``, while
you decide whether to keep, refactor, or withdraw the patch.
Professional conduct
--------------------
The Zaqar team holds reviewers accountable for promoting a positive,
constructive culture within our program.
If you ever feel that a reviewer is not acting professionally or is violating
the OpenStack community code of conduct, please let the PTL know immediately
so that he or she can help resolve the issue.
.. _`making a good commit message`: https://wiki.openstack.org/wiki/GitCommitMessages
.. _`squashing guide` : http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html

106
doc/source/first_review.rst
View File

@ -15,61 +15,101 @@
Your first review
=================
The review stage is a very important part in the development process. Following are some of the reasons this stage is important:
The review stage is a very important part in the development process. Following
are some of the reasons this stage is important:
* Getting other developers feedback minimizes the risk of adding
regressions to the code base and ensures the quality of the code being merged.
* Building the community encourages everyone to review code. Everyone appreciates having their code reviewed.
* Since developers are always learning from being exposed to the points of view of others, reviews help developers to improve their coding skills.
* Providing a review is a great way to become familar with the code.
regressions to the code base and ensures the quality of the code being
merged.
* Building the community encourages everyone to review code. Everyone
appreciates having their code reviewed.
* Since developers are always learning from being exposed to the points of view
of others, reviews help developers to improve their coding skills.
* Providing a review is a great way to become familiar with the code.
Everyone is encourages to review code. You don't need to know every detail of the code base. You need to understand only what the code related to the fix does.
Everyone is encourages to review code. You don't need to know every detail of
the code base. You need to understand only what the code related to the fix
does.
Step by step
------------
Go to review.openstack.org and filter by `Open Zaqar fixes`_. Select a fix from the list to review.
Try to select an easy patch for your first review. That will help you to gain some confidence.
Download the patch to your local repository and test it::
Go to ``review.openstack.org`` and filter by `Open Zaqar fixes`_. Select a fix
from the list to review. Try to select an easy patch for your first review.
That will help you to gain some confidence. Download the patch to your local
repository and test it:
.. code-block:: console
$ git review -d [review-id]
The review-id is the number in the URL (check the screenshot for more details).
The :samp:`{review-id}` is the number in the URL (check the screenshot for more
details).
Example::
Example:
.. code-block:: console
$ git review -d 92979
.. image:: images/zaqar_review_id.png
:alt: Zaqar review id
This git command creates a branch with the author's name and enables you to test the patch in your local environment.
This git command creates a branch with the author's name and enables you to
test the patch in your local environment.
* Inspect the code. Use all of the best programming practices you know as you review the code.
* Give code location feedback. (Do you consider that some code should be better located in another place within the file, or maybe in another file? If so, suggest this in the review comment and score with a -1 if you think that it's that important.)
* Give code-style feedback. (Do you think that the code structure could be improved? Keep the DRY, YAGNI and KISS principles in mind.)
* Give grammar and orthography feedback. Many of our contributors are not native English speakers, so it is common to find some errors of this type.
* Inspect the code. Use all of the best programming practices you know as you
review the code.
* Give code location feedback.
Do you consider that some code should be better located in another place
within the file, or maybe in another file? If so, suggest this in the
review comment and score with a ``-1`` if you think that it's that
important.
* Give code-style feedback.
Do you think that the code structure could be improved? Keep the DRY,
YAGNI and KISS principles in mind.
* Give grammar and orthography feedback. Many of our contributors are not
native English speakers, so it is common to find some errors of this type.
* Make sure that:
* The commit message is formatted appropriately (check `Git Commit Messages`_ for more information on how you should write a git commit message.)
* The coding style matches guidelines given in HACKING.rst
* The patch is not too big (you might need to split some patches to improve cohesion and/or reduce size).
* The patch does what the commit message promises
* Unit and functional tests are included and/or updated
* If during the inspection you see a specific line you would like to bring up to discussion in the final review, leave feedback as an inline comment in Gerrit. This will make the review process easier. You can also see the `prefixes you can use`_ for Zaqar inline comments.
* Keep in mind the `Zaqar's Reviewer Guide`_ and be respectful when leaving feedback.
* Hit the "Review" button in the web UI to publish your comments and assign a score.
* Things to consider when leaving a score:
* You can score with a -1 if you think that there are things to fix. We have to be careful to not stall the cycle just because a few nits, so downvoting also depends on the current stage of the development cycle and the severity of the flaw you see.
* You can score with a 0 if you are the author of the fix and you want to respond to the reviewers comments, or if you are a reviewer and you want to point out some reminder for future developing (e.g. the deadline is the next day and the fix needs to be merged, but you want something to be improved).
* You can score with +1 if the fix works and you think that the code looks good, upvoting is your choice.
* Remember to leave any comment that you think is important in the comment form. When you are done, click "Publish Comments".
For more details on how to do a review, check out the Review section in the GerritWorkflow wiki.
* The commit message is formatted appropriately.
Check `Git Commit Messages`_ for more information on how you should
write a git commit message.
* The coding style matches guidelines given in ``HACKING.rst``.
* The patch is not too big.
You might need to split some patches to improve cohesion and/or reduce
size.
* The patch does what the commit message promises.
* Unit and functional tests are included and/or updated.
* If during the inspection you see a specific line you would like to bring up
to discussion in the final review, leave feedback as an inline comment in
Gerrit. This will make the review process easier. You can also use
prefixes described in :doc:`reviewer_guide` for Zaqar inline comments.
* Keep in mind the :doc:`reviewer_guide` and be respectful when leaving
feedback.
* Hit the :guilabel:`Review` button in the web UI to publish your comments
and assign a score.
* Things to consider when leaving a score:
* You can score with a ``-1`` if you think that there are things to fix. We
have to be careful to not stall the cycle just because a few nits, so
downvoting also depends on the current stage of the development cycle
and the severity of the flaw you see.
* You can score with a "0" if you are the author of the fix and you want to
respond to the reviewers comments, or if you are a reviewer and you want
to point out some reminder for future developing (e.g. the deadline is
the next day and the fix needs to be merged, but you want something to be
improved).
* You can score with ``+1`` if the fix works and you think that the code
looks good, upvoting is your choice.
* Remember to leave any comment that you think is important in the comment
form. When you are done, click :guilabel:`Publish Comments`.
For more details on how to do a review, check out the `Gerrit Workflow
Review section`_ document.
.. _`Open Zaqar fixes`: https://review.openstack.org/#/q/status:open+zaqar,n,z
.. _`Zaqar's Reviewer Guide`: https://wiki.openstack.org/wiki/Zaqar/Reviewer_guide
.. _`prefixes you can use`: https://wiki.openstack.org/wiki/Zaqar/Reviewer_guide#Use_Prefixes
.. _`Git Commit Messages`: https://wiki.openstack.org/wiki/GitCommitMessages
.. _`Gerrit Workflow Review section`: http://docs.openstack.org/infra/manual/developers.html#code-review

23
doc/source/gerrit.rst
View File

@ -1,3 +1,16 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
========================
Code reviews with Gerrit
========================
@ -8,10 +21,8 @@ is http://review.openstack.org.
Gerrit is a complete replacement for GitHub pull requests. `All GitHub pull
requests to the Zaqar repository will be ignored`.
See `Gerrit Workflow Quick Reference`_ for information about how to get
started using Gerrit. See `Development Workflow`_ for more detailed
documentation on how to work with Gerrit.
See `Development Workflow with Gerrit`_ for more detailed documentation on how
to work with Gerrit.
.. _Gerrit: http://code.google.com/p/gerrit
.. _Development Workflow: http://docs.openstack.org/infra/manual/developers.html#development-workflow
.. _Gerrit Workflow Quick Reference: http://docs.openstack.org/infra/manual/developers.html#development-workflow
.. _Gerrit: https://www.gerritcodereview.com/
.. _Development Workflow with Gerrit: http://docs.openstack.org/infra/manual/developers.html#development-workflow

105
doc/source/index.rst
View File

@ -12,7 +12,7 @@
under the License.
=============================================
Welcome to the Zaqar developer documentation!
Welcome to the Zaqar Developer Documentation!
=============================================
Zaqar is a multi-tenant cloud messaging and notification service for web
@ -21,15 +21,17 @@ and mobile developers.
The service features a REST API, which developers can use to send messages
between various components of their SaaS and mobile applications, by using a
variety of communication patterns. Underlying this API is an efficient
messaging engine designed with scalability and security in mind.
messaging engine designed with scalability and security in mind. The Websocket
API is also available.
Other OpenStack components can integrate with Zaqar to surface events to
end users and to communicate with guest agents that run in the
"over-cloud" layer.
Other OpenStack components can integrate with Zaqar to surface events to end
users and to communicate with guest agents that run in the "over-cloud" layer.
.. note:: This documentation is generated by the Sphinx toolkit and lives in the Zaqar project source
tree. Additional draft and project documentation regarding Zaqar and other components of OpenStack can
be found on the `OpenStack Wiki`_, as well as in the user guides found on `docs.openstack.org`_.
.. note:: This documentation is generated by the Sphinx toolkit and lives in
the Zaqar project source tree. Additional draft and project documentation
regarding Zaqar and other components of OpenStack can be found on the
`OpenStack Wiki`_, as well as in the user guides found on
`docs.openstack.org`_.
.. _`OpenStack Wiki`: http://wiki.openstack.org
.. _`docs.openstack.org`: http://docs.openstack.org
@ -39,26 +41,45 @@ Key features
Zaqar provides the following key features:
* HTTP-based messaging API
* Multi-tenant design based on Keystone project IDs
* Choice between two communication transports. Both with Keystone support:
* Firewall-friendly, **HTTP-based RESTful API**. Many of today's developers
prefer a more web-friendly HTTP API. They value the simplicity and
transparency of the protocol, it's firewall-friendly nature, and it's huge
ecosystem of tools, load balancers and proxies. In addition, cloud
operators appreciate the scalability aspects of the REST architectural
style.
* **Websocket-based API** for persistent connections. Websocket protocol
provides communication over persistent connections. Unlike HTTP, where
new connections are opened for each request/response pair, Websocket can
transfer multiple requests/responses over single TCP connection. It saves
much network traffic and minimizes delays.
* Multi-tenant queues based on Keystone project IDs.
* Support for several common patterns including event broadcasting, task
distribution, and point-to-point messaging
* Component-based architecture with support for custom backends and
message filters
* Efficient reference implementation with an eye toward low latency and
high throughput (dependent on backend)
* Highly-available and horizontally scalable
distribution, and point-to-point messaging.
* Component-based architecture with support for custom backends and message
filters.
* Efficient reference implementation with an eye toward low latency and high
throughput (dependent on backend).
* Highly-available and horizontally scalable.
* Support for subscriptions to queues. Several notification types are
available:
* Email notifications.
* Webhook notifications.
* Websocket notifications.
Project scope
=============
The Zaqar API is data-oriented. That is, it does not provision message brokers
and expose those directly to clients. Instead, the API acts as a bridge
between the client and one or more backends. A provisioning service for
message brokers—however useful—serves a somewhat different market from what
Zaqar is targeting today. With that in mind, if users are interested in a
broker provisioning service, the community should consider starting a new
project to address that need.
and expose those directly to clients. Instead, the API acts as a bridge between
the client and one or more backends. A provisioning service for message
brokers, however useful, serves a somewhat different market from what Zaqar is
targeting today. With that in mind, if users are interested in a broker
provisioning service, the community should consider starting a new project to
address that need.
Design principles
=================
@ -92,17 +113,18 @@ Zaqar is composed of two layers:
storage/autoindex
The **transport drivers** are responsible for interacting with Zaqar clients.
Every query made by clients is processed by the transport layer,
which is in charge of passing this information to the backend and then
returning the response in a format understandable by the client.
Every query made by clients is processed by the transport layer, which is in
charge of passing this information to the backend and then returning the
response in a format understandable by the client.
The **storage drivers** are responsible for interacting with the storage backends
and, that way, store or retrieve the data coming from the transport layer.
The **storage drivers** are responsible for interacting with the storage
backends and, that way, store or retrieve the data coming from the transport
layer.
In order to keep these layers decoupled, we have established that
**checks should be performed in the appropriate layer**. In other words,
transport drivers must guarantee that the incoming data is well-formed
and storage drivers must enforce their data model stays consistent.
transport drivers must guarantee that the incoming data is well-formed and
storage drivers must enforce their data model stays consistent.
Setting up a development environment
====================================
@ -112,12 +134,13 @@ Setting up a development environment
devref/development.environment
Your first patch and your first review
======================================
Welcome new contributors
========================
.. toctree::
:maxdepth: 1
welcome
first_patch
first_review
@ -131,6 +154,22 @@ Running and writing tests
running_tests
test_suite
Reviewing
=========
.. toctree::
:maxdepth: 1
reviewer_guide
Running benchmark
=================
.. toctree::
:maxdepth: 1
running_benchmark
Other resources
===============
@ -141,8 +180,8 @@ Other resources
gerrit
jenkins
API reference
=============
Internal API reference
======================
.. toctree::
:maxdepth: 3

17
doc/source/jenkins.rst
View File

@ -1,10 +1,23 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
===================================
Continuous integration with Jenkins
===================================
Zaqar uses a `Jenkins`_ server to automate development tasks. The Jenkins
front-end is at http://jenkins.openstack.org. You must have an
account on `Launchpad`_ to be able to access the OpenStack Jenkins site.
front-end is at http://jenkins.openstack.org. You must have an account on
`Launchpad`_ to be able to access the OpenStack Jenkins site.
Jenkins performs tasks such as running static code analysis, running unit
tests, and running functional tests. For more details on the jobs being run by

13
doc/source/launchpad.rst
View File

@ -23,7 +23,7 @@ The developers mailing list address is ``openstack-dev@lists.openstack.org``.
This is a common mailing list across all OpenStack projects.
To participate in the mailing list:
#. Subscribe at http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Subscribe at http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
The mailing list archives are at http://lists.openstack.org/pipermail/openstack-dev.
@ -41,13 +41,14 @@ https://blueprints.launchpad.net/zaqar.
Technical support (Answers)
---------------------------
Zaqar uses Launchpad Answers to track Zaqar technical support questions. The Zaqar
Answers page is at https://answers.launchpad.net/zaqar.
Zaqar uses Launchpad Answers to track Zaqar technical support questions. The
Zaqar Answers page is at https://answers.launchpad.net/zaqar.
Note that `Ask OpenStack`_ (which are not hosted on Launchpad) can also
be used for technical support requests.
Note that `Ask OpenStack`_ (which is not hosted on Launchpad) can also be used
for technical support requests.
You can also reach us in #openstack-zaqar at irc.freenode.org.
You can also reach us in ``#openstack-zaqar`` IRC channel at
``irc.freenode.org``.
.. _Launchpad: http://launchpad.net
.. _Wiki: http://wiki.openstack.org

165
doc/source/reviewer_guide.rst Normal file
View File

@ -0,0 +1,165 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
==============
Reviewer Guide
==============
Overview
--------
Our program follows the usual OpenStack review process, albeit with some
important additions (see below). See also: :doc:`first_review`.
Be Professional
---------------
The PTL, with the support of the core reviewers, is ultimately responsible for
holding contributors accountable for creating a positive, constructive, and
productive culture. Inappropriate behavior will not be tolerated.
(`Why this is important?`_)
Do This:
* Act professionally.
* Treat others as friends and family.
* Seek first to understand.
* Be honest, transparent, and constructive.
* Use clear, concise language.
* Use prefixes to clarify the tone and intent of your comments.
Don't Do This:
* Use indecent, profane, or degrading language of any kind.
* Hold a patch hostage for an ulterior motive, political or otherwise.
* Abuse the review system to discuss big issues that would be better hashed out
on the mailing list, in IRC, or during OpenStack Summit design sessions.
* Engage in bullying behaviors, including but not limited to:
* Belittling others' opinions
* Persistent teasing or sarcasm
* Insulting, threatening, or yelling at someone
* Accusing someone of being incompetent
* Setting someone up to fail
* Humiliating someone
* Isolating someone from others
* Withholding information to gain an advantage
* Falsely accusing someone of errors
* Sabotaging someone's work
Reviewing Docs
--------------
When possible, enlist the help of a professional technical writer to help
review each doc patch. All reviewers should familiarize themselves with
`OpenStack Documentation Contributor Guide`_. When reviewing user guide
patches, please run them through Maven and proof the resulting docs before
giving your ``+1`` or ``+2``.
Reviewing Code
--------------
When reviewing code patches, use your best judgment and seek to provide
constructive feedback to the author. Compliment them on things they have done
well, and highlight possible improvements. Also, dedicate as much time as
necessary in order to provide a careful analysis of the code. Don't assume that
someone else will catch any issues you yourself miss; in other words, pretend
you are the only person reviewing a given patch. Remember, "given enough
eyeballs, all bugs are shallow" ceases to be true the moment individual
reviewers become complacent.
Some things to check when reviewing code:
* Patch aligns with project goals, and is ideally associated with a bp or bug.
* Commit message is formatted appropriately and contains external references as
needed.
* Coding style matches guidelines given in ``HACKING.rst``.
* Patch is cohesive and not too big to be reviewed in a timely manner (some
patches may need to be split to improve cohesion and/or reduce size).
* Patch does what the commit message promises.
* Algorithms are implemented correctly, and chosen appropriately.
* Data schemas follow best practices.
* Unit and functional tests have been included and/or updated.
* Code contains no bugs (pay special attention to edge cases that tests may
have missed).
Use Prefixes
------------
We encourage the use of prefixes to clarify the tone and intent of your review
comments. This is one way we try to mitigate misunderstandings that can lead to
bad designs, bad code, and bad blood.
.. list-table:: **Prefixes**
:widths: 6 80 8
:header-rows: 1
* - Prefix
- What the reviewer is saying
- Blocker?
* - KUDO
- You did a nice job here, and I wanted to point that out. Keep up the
good work!
- No
* - TEST
- I think you are missing a test for this feature, code branch, specific
data input, etc.
- Yes
* - BUG
- I don't think this code does what it was intended to do, or I think
there is a general design flaw here that we need to discuss.
- Yes
* - SEC
- This is a serious security vulnerability and we better address it before
merging the code.
- Yes
* - PERF
- I have a concern that this won't be fast enough or won't scale. Let's
discuss the issue and benchmark alternatives.
- Yes
* - DSQ
- I think there is something critical here that we need to discuss this in
IRC or on the mailing list before moving forward.
- Yes
* - STYLE
- This doesn't seem to be consistent with other code and with
``HACKING.rst``
- Yes
* - Q
- I don't understand something. Can you clarify?
- Yes
* - DRY
- This could be modified to reduce duplication of code, data, etc.
See also: `Wikipedia: Don't repeat yourself`_
- Maybe
* - YAGNI
- This feature or flexibility probably isn't needed, or isn't worth the
added complexity; if it is, we can always add the feature later. See
also: `Wikipedia: You aren't gonna need it`_
- Maybe
* - NIT
- This is a nitpick that I can live with if we want to merge without
addressing it.
- No
* - IMO
- I'm chiming in with my opinion in response to someone else's comment, or
I just wanted to share an observation. Please take what I say with a
grain of salt.
- No
* - FYI
- I just wanted to share some useful information.
- No
.. _`Why this is important?` : https://thoughtstreams.io/kgriffs/technical-communities/5060/
.. _`OpenStack Documentation Contributor Guide` : http://docs.openstack.org/contributor-guide/index.html
.. _`Wikipedia: Don't repeat yourself` : https://en.wikipedia.org/wiki/Don't_repeat_yourself
.. _`Wikipedia: You aren't gonna need it` : https://en.wikipedia.org/wiki/Don't_repeat_yourself

115
doc/source/running_benchmark.rst Normal file
View File

@ -0,0 +1,115 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
=================
Running benchmark
=================
Introduction
------------
This document describes how to run benchmarking tool.
Zaqar Contributors can use this tool to test how the particular code change
affects Zaqar's performance.
Usage
-----
1. First install and run zaqar-server.
For example, you can setup Zaqar in development environment.
See :doc:`devref/development.environment`.
2. In your terminal cd into your local Zaqar repo and install additional
requirements:
.. code-block:: console
$ pip install -r bench-requirements.txt
3. Copy the configuration file to ~/.zaqar:
.. code-block:: console
$ cp etc/zaqar-benchmark.conf.sample ~/.zaqar/zaqar-benchmark.conf
4. In this configuration file specify where zaqar-server can be found:
.. code-block:: ini
server_url = http://localhost:8888
5. The benchmarking tool needs a set of messages to work with. Specify the path
to the file with messages in the configuration file. Alternatively, put
it in the directory with the configuration file and name it
``zaqar-benchmark-messages.json``.
As a starting point, you can use the sample file from the etc directory:
.. code-block:: console
$ cp etc/zaqar-benchmark-messages.json ~/.zaqar/
If the file is not found or no file is specified, a single hard-coded
message is used for all requests.
6. Run the benchmarking tool using the following command:
.. code-block:: console
$ zaqar-bench
By default, the command will run a performance test for 5 seconds, using one
producer process with 10 greenlet workers, and one observer process with 5
workers. The consumer role is disabled by default.
You can override these defaults in the config file or on the command line
using a variety of options. For example, the following command runs a
performance test for 30 seconds using 4 producer processes with 20
workers each, plus 4 consumer processes with 20 workers each.
Note that the observer role is also disabled in this example by setting its
number of workers to zero:
.. code-block:: console
$ zaqar-bench -pp 4 -pw 10 -cp 4 -cw 20 -ow 0 -t 30
By default, the results are in JSON.
For more human-readable output add the :option:`--verbose` flag.
Verbose output looks similar to the following:
.. code-block:: console
$ zaqar-bench --verbose
Starting producer (pp=1 , pw=10)...
Starting observer (op=1 , ow=5)...
Producer
========
duration_sec: 5.1
ms_per_req: 2.9
reqs_per_sec: 344.5
successful_reqs: 1742.0
total_reqs: 1742.0
Observer
========
duration_sec: 5.0
ms_per_req: 2.9
reqs_per_sec: 339.3
successful_reqs: 1706.0
total_reqs: 1706.0

131
doc/source/running_tests.rst
View File

@ -1,9 +1,24 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
=============
Running tests
=============
Zaqar contains a suite of tests (both unit and functional) in the
``zaqar/tests`` and ``tests`` directories.
``zaqar/tests`` directory.
See :doc:`test_suite` for details.
Any proposed code change is automatically rejected by the OpenStack Jenkins
server [#f1]_ if the change causes test failures.
@ -17,50 +32,126 @@ Preferred way to run the tests
The preferred way to run the unit tests is using ``tox``. It executes tests in
isolated environment, by creating separate virtualenv and installing
dependencies from the ``requirements.txt`` and ``test-requirements.txt`` files,
so the only package you install is ``tox`` itself::
so the only package you install is ``tox`` itself:
pip install tox
.. code-block:: console
$ pip install tox
See `the unit testing section of the Testing wiki page`_ for more information.
Following are some simple examples.
To run the Python 2.6 tests::
To run the Python 2.7 tests:
tox -e py26
.. code-block:: console
To run the style tests::
$ tox -e py27
tox -e pep8
To run the style tests:
To run multiple tests separate items by commas::
.. code-block:: console
tox -e py27,pep8
$ tox -e pep8
To run multiple tests separate items by commas:
.. code-block:: console
$ tox -e py27,py34,pep8
.. _the unit testing section of the Testing wiki page: https://wiki.openstack.org/wiki/Testing#Unit_Tests
Running a subset of tests
-------------------------
^^^^^^^^^^^^^^^^^^^^^^^^^
Instead of running all tests, you can specify an individual directory, file,
class, or method that contains test code.
class or method that contains test code, i.e. filter full names of tests by a
string.
To run the tests located only in the ``tests/unit/queues/storage`` directory use::
To run the tests located only in the ``zaqar/tests/unit/queues/storage``
directory use:
tox -e py27 tests.unit.queues.storage
.. code-block:: console
To run the tests specific to the MongoDB driver in the ``tests/unit/queues/storage/test_impl_mongodb.py`` file::
$ tox -e py27 zaqar.tests.unit.queues.storage
tox -e py27 test_impl_mongodb
To run the tests specific to the MongoDB driver in the
``zaqar/tests/unit/queues/storage/test_impl_mongodb.py`` file:
.. code-block:: console
$ tox -e py27 test_impl_mongodb
To run the tests in the ``MongodbMessageTests`` class in
the ``tests/unit/queues/storage/test_impl_mongodb.py`` file::
the ``tests/unit/queues/storage/test_impl_mongodb.py`` file:
tox -e py27 test_impl_mongodb.MongodbMessageTests
.. code-block:: console
To run the `MongodbMessageTests.test_message_lifecycle` test method in
the ``tests/unit/queues/storage/test_impl_mongodb.py`` file::
$ tox -e py27 test_impl_mongodb.MongodbMessageTests
tox -e py27 test_impl_mongodb.MongodbMessageTests.test_message_lifecycle
To run the ``MongodbMessageTests.test_message_lifecycle`` test method in
the ``tests/unit/queues/storage/test_impl_mongodb.py`` file:
.. code-block:: console
$ tox -e py27 test_impl_mongodb.MongodbMessageTests.test_message_lifecycle
Running functional tests
------------------------
Zaqar's functional tests treat Zaqar as a black box. In other words, the API
calls attempt to simulate an actual user. Unlike unit tests, the functional
tests do not use mockendpoints.
Functional test modes
^^^^^^^^^^^^^^^^^^^^^
Functional tests can run in integration mode and non-integration mode.
Integration mode
""""""""""""""""
In integration mode functional tests are performed on Zaqar server instances
running as separate processes. This is real functional testing.
To run functional tests in integration mode, execute:
.. code-block:: console
$ tox -e integration
Non-integration mode
""""""""""""""""""""
In non-integration mode functional tests are performed on Zaqar server
instances running as python objects. This mode doesn't guarantee enough black
boxness for Zaqar, but tests run 10 times faster than in integration mode.
To run functional tests in non-integration mode, execute:
.. code-block:: console
$ tox -e py27 zaqar.tests.functional
Using a custom MongoDB instance
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you need to run functional tests against a non-default MongoDB installation,
you can set the ``ZAQAR_TEST_MONGODB_URL`` environment variable. For example:
.. code-block:: console
$ export ZAQAR_TEST_MONGODB_URL=mongodb://remote-server:27017
Using custom parameters
^^^^^^^^^^^^^^^^^^^^^^^
You can edit default functional test configuration file
``zaqar/tests/etc/functional-tests.conf`` according to your needs.
For example, you want to run functional tests with keystone authentication
enabled, input a valid set of credentials to ``[auth]`` section in
configuration file and set ``auth_on`` parameter to ``True``.
.. rubric:: Footnotes

82
doc/source/test_suite.rst
View File

@ -1,7 +1,23 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
====================
Test suite structure
====================
Test types
----------
There are three types of tests for Zaqar:
Unit tests
@ -10,33 +26,71 @@ Unit tests
Functional tests
Functional tests verify that the service works as expected. In particular,
in Zaqar they exercise the API endpoints and validate that the API responses
conform to the specs. These include positive and negative tests.
in Zaqar they exercise the API endpoints and validate that the API
responses conform to the specs. These include positive and negative tests.
Tempest tests
Tempest tests are integration tests for OpenStack [#f1]_.
Tempest tests for Zaqar are available at https://github.com/openstack/tempest.
This document focuses on the unit and functional tests. Please refer to the
Tempest repository for details on how to run these tests.
Tempest tests for Zaqar are available in the `Tempest repository`_.
Refer to :doc:`running_tests` document for details on how to run Unit and
Functional tests.
Refer to the `Tempest repository`_ for details on how to run Tempest tests.
Code structure
--------------
The test suite lives in two directories:
The test suite lives in ``zaqar/tests`` directory of Zaqar:
- ``zaqar/tests`` contains all base classes and defines tests for APIs (on both storage and transport levels).
- ``tests`` usually contains implementations for specific drivers and additional tests.
* ``zaqar/tests/etc``
Contains various configuration files for Zaqar. They help to test how Zaqar
works in different configurations.
Thus base class and all common tests for storage drivers are located in the ``zaqar/tests/queues/storage/base.py`` file.
The specific instances of the base classes for any particular storage driver are located at the
``tests/unit/queues/storage/`` directory. See ``tests/unit/queues/storage/test_impl_mongodb.py`` for example.
* ``zaqar/tests/functional``
Contains functional tests.
Similarly, unit tests for the transport layer are located in ``zaqar/tests/queues/transport``
and are run from classes located in the ``tests/unit/queues/transport`` directory.
* ``zaqar/tests/unit``
Contains unit tests.
All functional tests for Zaqar are located in the ``tests/functional`` directory.
The base class of all test classes is located in the ``zaqar/tests/base.py``
file.
Test invocation
---------------
When you run tests via ``tox -e py27`` command in the root directory of Zaqar:
#. Tox program executes:
#. Looks for ``tox.ini`` file.
#. Creates ``.tox`` directory for storing python environments.
#. Parses this file and finds parameters for py27 testing environment.
#. Sets this environment up and activates it.
#. Sets environment variables for this environment that are described in
``tox.ini``
#. In case of Zaqar it invokes Testr program in the environment.
You can find more information about Tox in `Openstack Tox testing manual`_
and in official `Tox documentation`_.
#. Testr (Test Repository) program executes:
#. Looks for ``testr.ini`` file.
#. Parses this file and finds parameters for executing tests.
#. Creates ``.testrepository`` directory for storing statistics of
executing tests.
#. In case of Zaqar it invokes ``Subunit`` program which finds all tests and
executes it.
You can find more information about Testr in `Openstack Testr manual`_.
.. rubric:: Footnotes
.. [#f1] See http://docs.openstack.org/developer/tempest/overview.html
.. _`Openstack Tox testing manual` : https://wiki.openstack.org/wiki/Testing#Unit_Testing_with_Tox
.. _`Tox documentation` : https://tox.readthedocs.org/en/latest/
.. _`Openstack Testr manual` : https://wiki.openstack.org/wiki/Testr
.. _`Tempest repository` : https://github.com/openstack/tempest

187
doc/source/welcome.rst Normal file
View File

@ -0,0 +1,187 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
========================
Welcome new contributors
========================
First Steps
===========
It's very great that you're interested in contributing to Zaqar.
First of all, make sure you join Zaqar communication forums:
* Subscribe to Zaqar `mailing lists`_.
* Join Zaqar team on IRC. You can chat with us directly in the
``#openstack-zaqar`` channel on ``irc.freenode.org``. If you don't know
how to use IRC, you can find some directions in `Openstack IRC wiki`_.
* Answer and ask questions on `Ask OpenStack`_.
How can I contribute?
=====================
There are many ways you can contribute to Zaqar. Of course coding is one, but
you can also join Zaqar as a tester, documenter, designer or translator.
Coding
------
Bug fixing
^^^^^^^^^^
The first area where you can help is bug fixing. ``Confirmed`` bugs are usually
your best choice. ``Triaged`` bugs should even contain tips on how they
should be fixed. You can find both of them in
`Zaqar's Confirmed and Triaged bugs`_ web page.
Once you selected the bug you want to work on, go ahead and assign it to
yourself, branch the code, implement the fix, and propose your change for
review. You can find information on how to do it in
:doc:`first_patch` manual.
Some easy-to-fix bugs may be marked with the ``low-hanging-fruit`` tag: those
are good targets for a beginner.
Bug triaging
^^^^^^^^^^^^
You can also help Zaqar with bug triaging. Reported bugs need care:
prioritizing them correctly, confirming them, making sure they don't go stale.
All those tasks help immensely. If you want to start contributing in coding,
but you are not a hardcore developer, consider helping in this area.
Bugs can be marked with different tags according to their status:
* ``New`` bugs are those bugs that have been reported by a user but haven't
been verified by the community yet.
* ``Confirmed`` bugs are those bugs that have been reproduced by someone else
than the reporter.
* ``Triaged`` bugs are those bugs that have been reproduced by a core
developer.
* ``Incomplete`` bugs are those bugs that don't have enough information to be
reproduced.
* ``In Progress`` bugs are those bugs that are being fixed by some developer.
This status is set automatically by the Gerrit review system once a fix is
proposed by a developer. You don't need to set it manually.
* ``Invalid`` bugs are those bugs that don't qualify as a bug. Usually a
support request or something unrelated to the project.
You can learn more about this in Launchpad's `Of Bugs and Statuses`_.
You only have to worry about ``New`` bugs. If you can reproduce them, you can
mark them as ``Confirmed``. If you cannot reproduce them, you can ask the
reported to provide more information and mark them as ``Incomplete``. If you
consider that they aren't bugs, mark them as ``Invalid`` (Be careful with this.
Asking someone else in Zaqar is always a good idea).
Also, you can contribute instructions on how to fix a given bug.
Check out the `Bug Triage`_ wiki for more information.
Reviewing
^^^^^^^^^
Every patch submitted to OpenStack gets reviewed before it can be approved and
merged. Zaqar gets a lot of contributions and everyone can (and is encouraged
to) review Zaqar's existing patches. Pick an open review and go through
it, test it if possible, and leave a comment with a ``+1`` or ``-1`` vote
describing what you discovered. If you're planning on submitting patches of
your own, it's a great way to learn about what the community cares about and to
learn about the code base.
Make sure you read :doc:`first_review` manual.
Feature development
^^^^^^^^^^^^^^^^^^^
Once you get familiar with the code, you can start to contribute new features.
New features get implemented every 6 months in `OpenStack development cycle`_.
We use `Launchpad Blueprints`_ to track the design and implementation of
significant features, and Zaqar team uses Design Summits every 6 months to
get together and discuss things in person with the rest of the community. Code
should be proposed for inclusion before Zaqar reach the final feature milestone
of the development cycle.
Testing
-------
Testing efforts are highly related to coding. If you find that there are test
cases missing or that some tests could be improved, you are encouraged to
report it as a bug and then provide your fix.
See :doc:`running_tests` and :doc:`test_suite` for information on how to run
tests and how the tests are organized in Zaqar.
See :doc:`first_patch` for information on how to provide your fix.
Documenting
-----------
You can contribute to `Zaqar's Contributor Documentation`_ which you are
currently reading and to `Zaqar's Wiki`_.
To fix a documentation bug check the bugs marked with the ``doc`` tag in
Zaqar's bug list. In case that you want to report a documentation bug, then
don't forget to add the ``doc`` tag to it.
`Zaqar's Contributor Documentation`_ is compiled from source files in ``.rst``
(reStructuredText) format located in ``doc/source/`` directory in Zaqar
repository. The `"openstack-manuals" project`_ houses the documentation that is
published to ``docs.openstack.org``.
Before contributing to `Zaqar's Contributor Documentation`_ you have to read
:doc:`first_patch` manual and `OpenStack Documentation Contributor Guide`_.
Also, you can monitor `Ask OpenStack`_ to curate the best answers that can be
folded into the documentation.
Designing
---------
Zaqar doesn't have a user interface yet. Zaqar team is working to
`integrate Zaqar to the OpenStack Dashboard (Horizon)`_.
If you're a designer or usability professional your help will be really
appreciated. Whether it's reviewing upcoming features as a user and giving
feedback, designing features, testing designs or features with users, or
helping to build use cases and requirements, everything is useful.
Translating
-----------
You can translate Zaqar to language you know.
Read the `Translation wiki page`_ for more information on how OpenStack manages
translations. Zaqar has adopted Zanata, and you can use the
`OpenStack Zanata site`_ as a starting point to translate any of the OpenStack
projects, including Zaqar. It's easier to start translating directly on the
`OpenStack Zanata site`_, as there is no need to download any files or
applications to get started.
.. _`mailing lists` : https://wiki.openstack.org/wiki/MailingLists
.. _`Openstack IRC wiki` : https://wiki.openstack.org/wiki/IRC
.. _`Ask OpenStack` : https://ask.openstack.org/
.. _`Zaqar's Confirmed and Triaged bugs` : https://bugs.launchpad.net/zaqar/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on
.. _`Of Bugs and Statuses` : http://blog.launchpad.net/general/of-bugs-and-statuses
.. _`Bug Triage` : https://wiki.openstack.org/wiki/BugTriage
.. _`OpenStack development cycle` : https://wiki.openstack.org/wiki/ReleaseCycle
.. _`Launchpad Blueprints` : https://wiki.openstack.org/wiki/Blueprints
.. _`OpenStack Documentation Contributor Guide` : http://docs.openstack.org/contributor-guide/index.html
.. _`Zaqar's Contributor Documentation` : http://docs.openstack.org/developer/zaqar/
.. _`Zaqar's Wiki` : https://wiki.openstack.org/wiki/Zaqar
.. _`"openstack-manuals" project` : https://wiki.openstack.org/wiki/Documentation
.. _`integrate Zaqar to the OpenStack Dashboard (Horizon)` : https://blueprints.launchpad.net/zaqar/+spec/zaqar-horizon-integration
.. _`Translation wiki page` : https://wiki.openstack.org/wiki/Translations#Translation_.26_Management
.. _`OpenStack Zanata site` : https://translate.openstack.org/

68
zaqar/tests/functional/README.rst
View File

@ -1,68 +0,0 @@
Zaqar Functional Tests
======================
Zaqar's functional tests treat Zaqar as a black box. In other
words, the API calls attempt to simulate an actual user. Unlike unit tests,
the functional tests do not use mockendpoints.
Running functional tests (With Tox)
-----------------------------------
#. Setup a Zaqar server. Refer to the Zaqar `README`_ on
how to run Zaqar locally, or simply use an existing server.
#. Run tests. ::
$ tox
#. Filter tests. ::
$ tox -- zaqar.tests.functional.wsgi.v1.test_messages
#. Run tests for specific environments. ::
$ tox -epy27,pep8
Running the Functional Tests (Without Tox)
------------------------------------------
#. Setup a Zaqar server. Refer to the Zaqar `README`_ on
how to run Zaqar locally, or simply use an existing server.
#. Install functional tests dependencies. ::
pip install -r requirements.txt
pip install -r test-requirements.txt
#. cd to the tests/etc directory
#. If leaving keystone auth enabled, update functional-tests.conf with a
valid set of credentials.
#. Now, to run the system tests, simply use the nosetests commands, e.g.:
Run all test suites: ::
nosetests -v
Adding New Tests
----------------
#. Add test case to an appropriate test case file: ::
queue/test_queue.py
messages/test_messages.py
claim/test_claims.py
Using a custom MongoDB instance
-------------------------------
If you need to run the tests against a non-default MongoDB installation, you
can set the ZAQAR_TEST_MONGODB_URL environemment variable. For example: ::
export ZAQAR_TEST_MONGODB_URL=mongodb://remote-server:27017
.. _README : https://github.com/openstack/zaqar/blob/master/README.rst
.. _requests : https://pypi.python.org/pypi/requests