x/turbo-hipster
Code Issues Proposed changes

Restructured documentation

Browse Source 
Moved information out of intro.rst and into more specific doc files.
Reorganised index.rst accordingly, and deleted unused files.

Change-Id: I660c5e5e597baddbd1d2d5f60871687b5306834c
This commit is contained in:
Lana Brindley 2014-01-17 13:32:17 +10:00
parent c972d2d1d7
commit ec7637a767
6 changed files with 240 additions and 410 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
doc/source/configuration.rst
View File

@ -1,10 +0,0 @@
:title: Configuration
Configuration
=============
It is required that debug logging be configured for Turbo Hispter, as we use
the python logging framework to capture log messages from the task plugin
code. Configuration is as simple as setting the debug_log configuration
setting in config.json.

2
doc/source/index.rst
View File

@ -12,9 +12,7 @@ Contents:
:maxdepth: 2
intro
structure
installation
configuration
running

209
doc/source/installation.rst
View File

@ -6,142 +6,111 @@
Installation
============
Download
--------
Turbo-hipster is installed directly into your Python ``site-packages``
directory, and is then run as a service. It is managed using a configuration
file, which is in .json format.
Grab a copy from https://github.com/rcbau/turbo-hipster
Installing turbo-hipster
------------------------
1. Turbo-Hipster can be installed directly to your Python ``site-packages``
directory::
Install
-------
$ sudo python setup.py install
turbo-hipster is configured to use setup tools for installation if
you would like to install it to your site-packages use::
2. Copy the configuration file to a convenient location. By default,
turbo-hipster will look in ``/etc/turbo-hipster/config.json`` ::
sudo python setup.py install
$ cp -R etc/turbo-hipster /etc/
3. The Turbo-Hipster configuration file is in .json format. Open the
``config.json`` configuration file in your preferred editor and modify it
for your environment::
Copy config
-----------
**zuul_server**
A dictionary containing details about how to communicate
with zuul
**git_url**
The publicly accessible protocol and URI from where
to clone projects and zuul_ references from. For
example::
http://review.openstack.org/p/
or::
git://review.example.org
**gearman_host**
The host of gearman_. zuul talks to its workers via
the gearman protocol and while it comes with a built-
in gearman server you can use a separate one.
**gearman_port**
The port that gearman listens on.
**debug_log**
A path to the debug log. Turbo-hipster will attempt to create
the file but must have write permissions.
**jobs_working_dir**
Each job will likely need to write out log and debug
information. This defines where turbo-hipster will do that.
**git_working_dir**
turbo-hipster needs to take a copy of the git tree of a
project to work from. This is the path it'll clone into and
work from (if needed).
**pip_download_cache**
Some of turbo-hipsters task plugins download requirements
for projects. This is the cache directory used by pip.
**plugins**
A list of enabled plugins and their settings in a dictionary.
The only required parameters are *name*, which should be the
same as the folder containing the plugin module, and
*function*, which is the function registered with zuul.
Any other parameters are specified by the plugin themselves
as required.
**publish_logs**
Log results from plugins can be published using multiple
methods. Currently only a local copy is fully implemented.
**type**
The type of protocol to copy the log to. eg 'local'
**path**
A type specific parameter defining the local location
destination.
**prepend_url**
What to prepend to the path when sending the result
URL back to zuul. This can be useful as you may want
to use a script to authenticate against a swift
account or to use *laughing_spice* to format the logs
etc.
Place the configuration where you are comfortable managing it. For
example::
4. Create a turbo-hipster user:
cp -R etc/turbo-hipster /etc/
$ useradd turbo-hipster
5. Create the directories listed in the configuration file, and give the
``turbo-hipster`` user write access::
Edit config
-----------
$ mkdir -p /var/log/turbo-hipster/
$ chown turbo-hipster:turbo-hipster /var/log/turbo-hipster/
Turbo-hipsters configuration is currently stored in json format.
Modify the config.json appropriately::
$ mkdir -p /var/lib/turbo-hipster/jobs
$ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/jobs
**zuul_server**
A dictionary containing details about how to communicate
with zuul
**git_url**
The publicly accessible protocol and URI from where
to clone projects and zuul_ references from. For
example::
http://review.openstack.org/p/
or::
git://review.example.org
**gearman_host**
The host of gearman_. zuul talks to its workers via
the gearman protocol and while it comes with a built-
in gearman server you can use a separate one.
**gearman_port**
The port that gearman listens on.
**debug_log**
A path to the debug log. Turbo-hipster will attempt to create
the file but must have write permissions.
**jobs_working_dir**
Each job will likely need to write out log and debug
information. This defines where turbo-hipster will do that.
**git_working_dir**
turbo-hipster needs to take a copy of the git tree of a
project to work from. This is the path it'll clone into and
work from (if needed).
**pip_download_cache**
Some of turbo-hipsters task plugins download requirements
for projects. This is the cache directory used by pip.
**plugins**
A list of enabled plugins and their settings in a dictionary.
The only required parameters are *name*, which should be the
same as the folder containing the plugin module, and
*function*, which is the function registered with zuul.
Any other parameters are specified by the plugin themselves
as required.
**publish_logs**
Log results from plugins can be published using multiple
methods. Currently only a local copy is fully implemented.
**type**
The type of protocol to copy the log to. eg 'local'
**path**
A type specific parameter defining the local location
destination.
**prepend_url**
What to prepend to the path when sending the result
URL back to zuul. This can be useful as you may want
to use a script to authenticate against a swift
account or to use *laughing_spice* to format the logs
etc.
$ mkdir -p /var/lib/turbo-hipster/git
$ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/git
$ mkdir -p /var/cache/pip
$ chown turbo-hipster:turbo-hipster /var/cache/pip
Set up
------
6. Open the MySQL log rotation configuration file in your preferred text
editor, and edit it to ensure it is writable by ``other``::
You probably want to create a user to run turbo-hipster as. This user
will need to write to all of the directories specified in
config.json.
$ vim /etc/logrotate.d/mysql-server
# edit create 640 to 644.
Make sure the required directories as defined by the config.json
exist and are writeable by your turbo-hipster user::
.. note::
The turbo-hipster source code is also available for download from
the `turbo-hipster github page <https://github.com/rcbau/turbo-hipster/>`_
mkdir -p /var/log/turbo-hipster/
chown turbo-hipster:turbo-hipster /var/log/turbo-hipster/
$ git clone https://github.com/rcbau/turbo-hipster
mkdir -p /var/lib/turbo-hipster/jobs
chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/jobs
mkdir -p /var/lib/turbo-hipster/git
chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/git
mkdir -p /var/cache/pip
chown turbo-hipster:turbo-hipster /var/cache/pip
Edit MySQL's log rotate to ensure it is other writable::
vim /etc/logrotate.d/mysql-server
# edit create 640 to 644.
Start turbo-hipster
-------------------
turbo hipster can be ran by executing::
./turbo-hipster/worker_server.py
and optionally takes the following parameters::
./turbo_hipster/worker_server.py --help
usage: worker_server.py [-h] [-c CONFIG] [-b] [-p PIDFILE]
optional arguments:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
Path to json config file.
-b, --background Run as a daemon in the background.
-p PIDFILE, --pidfile PIDFILE
PID file to lock during daemonization.
By default turbo-hipster will look for
*/etc/turbo-hipster/config.json*
Alternatively turbo-hipster can be launched by init.d using the
included etc/init.d/turbo-hipster script::
sudo cp etc/init.d/turbo-hipster /etc/init.d/
sudo update-rc.d turbo-hipster defaults
sudo service turbo-hipster start
.. note::
Debug logging must be configured for turbo-hipster, as it uses the Python
logging framework to capture log messages from the task plugin code.
To configure debug logging, set the ``debug_log`` configuration
setting in the ``config.json`` configuration file.

256
doc/source/intro.rst
View File

@ -92,260 +92,4 @@ Typical workflow diagram
}
Installation
============
Turbo-hipster is installed directly into your Python ``site-packages``
directory, and is then run as a service. It is managed using a configuration
file, which is in .json format.
Installing turbo-hipster
------------------------
1. Turbo-Hipster can be installed directly to your Python ``site-packages``
directory::
$ sudo python setup.py install
2. Copy the configuration file to a convenient location. By default,
turbo-hipster will look in ``/etc/turbo-hipster/config.json`` ::
$ cp -R etc/turbo-hipster /etc/
3. The Turbo-Hipster configuration file is in .json format. Open the
``config.json`` configuration file in your preferred editor and modify it
for your environment::
**zuul_server**
A dictionary containing details about how to communicate
with zuul
**git_url**
The publicly accessible protocol and URI from where
to clone projects and zuul_ references from. For
example::
http://review.openstack.org/p/
or::
git://review.example.org
**gearman_host**
The host of gearman_. zuul talks to its workers via
the gearman protocol and while it comes with a built-
in gearman server you can use a separate one.
**gearman_port**
The port that gearman listens on.
**debug_log**
A path to the debug log. Turbo-hipster will attempt to create
the file but must have write permissions.
**jobs_working_dir**
Each job will likely need to write out log and debug
information. This defines where turbo-hipster will do that.
**git_working_dir**
turbo-hipster needs to take a copy of the git tree of a
project to work from. This is the path it'll clone into and
work from (if needed).
**pip_download_cache**
Some of turbo-hipsters task plugins download requirements
for projects. This is the cache directory used by pip.
**plugins**
A list of enabled plugins and their settings in a dictionary.
The only required parameters are *name*, which should be the
same as the folder containing the plugin module, and
*function*, which is the function registered with zuul.
Any other parameters are specified by the plugin themselves
as required.
**publish_logs**
Log results from plugins can be published using multiple
methods. Currently only a local copy is fully implemented.
**type**
The type of protocol to copy the log to. eg 'local'
**path**
A type specific parameter defining the local location
destination.
**prepend_url**
What to prepend to the path when sending the result
URL back to zuul. This can be useful as you may want
to use a script to authenticate against a swift
account or to use *laughing_spice* to format the logs
etc.
4. Create a turbo-hipster user:
$ useradd turbo-hipster
5. Create the directories listed in the configuration file, and give the
``turbo-hipster`` user write access::
$ mkdir -p /var/log/turbo-hipster/
$ chown turbo-hipster:turbo-hipster /var/log/turbo-hipster/
$ mkdir -p /var/lib/turbo-hipster/jobs
$ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/jobs
$ mkdir -p /var/lib/turbo-hipster/git
$ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/git
$ mkdir -p /var/cache/pip
$ chown turbo-hipster:turbo-hipster /var/cache/pip
6. Open the MySQL log rotation configuration file in your preferred text
editor, and edit it to ensure it is writable by ``other``::
$ vim /etc/logrotate.d/mysql-server
# edit create 640 to 644.
.. note::
The turbo-hipster source code is also available for download from
the `turbo-hipster github page <https://github.com/rcbau/turbo-hipster/>`_
$ git clone https://github.com/rcbau/turbo-hipster
Starting turbo-hipster
----------------------
Turbo-hipster can be run from the command line::
$ ./turbo-hipster/worker_server.py
This option allows you to pass parameters to turbo-hipster. Use the --help
parameter to see a full list.
+-------+--------------+--------------------------------------------------------+
| Short | Long | Description |
+=======+==============+========================================================+
| -c | --config | Print the path to the configuration file and exit |
+-------+--------------+--------------------------------------------------------+
| -b | --background | Run as a daemon in the background |
+-------+--------------+--------------------------------------------------------+
| -p | --pidfile | Specify the PID file to lock while running as a daemon |
+-------+--------------+--------------------------------------------------------+
Alternatively, you can start turbo-hipster as a service.
1. Copy the turbo-hipster init.d script to /etc/init.d/:
$ sudo cp etc/init.d/turbo-hipster /etc/init.d/
2. Reload the script with the default configuration:
$ sudo update-rc.d turbo-hipster defaults
3. Start the service:
$ sudo service turbo-hipster start
Plugins
=======
Plugins can be used to extend turbo-hipster's capabilities.
.. note::
Currently, the only available plugin for turbo-hipster is the
Database Migration plugin, ``gate_real_db_upgrade``, which tests code
against a variety of real-world databases.
Installing plugins
------------------
Turbo-hipster plugins are responsible for handling the jobs that are passed
to it. They must successfully build reports and publish them according to
their configuration. They must also be able to communicate test results back
to Zuul using Gearman.
Plugins must take a standard format in order to be able to work correctly
with turbo-hipster. They must contain a ``task.py`` file with a ``Runner``
class.
Once you have created a turbo-hipster plugin, you need to configure it in
the ``config.json`` configuration file.
.. FIXME More config information required here
Plugin: Database migration with ``gate_real_db_upgrade``
--------------------------------------------------------
The database migration plugin, ``gate_real_db_upgrade``, is used to test
datasets against real-world, anonymized, databases.
Migrating a database
--------------------
In order to use turbo-hipster with the ``gate_real_db_upgrade`` plugin, you
need to set up the databases to test against, and point to the plugin in
turbo-hipster's configuration file.
1. Create a directory for the datasets:
$ mkdir -p /var/lib/turbo-hipster/datasets
2. Copy the json dataset to the directory you created:
$ cp /my/dataset.json /var/lib/turbo-hipster/datasets/
3. Open the ``/etc/turbo-hipster/config.json`` file in your preferred
editor, locate the plugins section, and add this line::
**plugins**
gate_real_db_upgrade
Testing with turbo-hipster
==========================
When turbo-hipster completes a test, it sends the result of the test back to
Gearman. These results contain a link to a compiled logfile for the test.
If the test fails, or takes too long to complete, turbo-hipster will add a
review to your patchset that looks like this:
.. image:: ../images/THTestResult.png
Reading test reports
--------------------
An example of a standard log file:
http://thw01.rcbops.com/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log
An example of the same logfile, using the javascript logviewer:
http://thw01.rcbops.com/logviewer/?q=/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log
Test failure codes
------------------
This section gives a list of failure codes, including some steps you can
take for troubleshooting errors:
FAILURE - Did not find the end of a migration after a start
If you look at the log you should find that a migration began but never
finished. Hopefully there'll be a traceroute for you to follow through to
get some hints about why it failed.
WARNING - Migration %s took too long
In this case your migration took a long time to run against one of the test
datasets. You should reconsider what operations your migration is performing
and see if there are any optimizations you can make, or if it is really
necessary. If there is no way to speed up your migration you can email us at
rcbau@rcbops.com for an exception.
FAILURE - Final schema version does not match expectation
Somewhere along the line the migrations stopped and did not reach the
expected version. Our datasets start at previous releases and have to
upgrade all the way through to the most current release. If you see this,
inspect the log for traceroutes or other hints about the failure.
FAILURE - Could not setup seed database.
FAILURE - Could not find seed database.
These errors are internal errors. If you see either of these, contact us at
rcbau@rcbops.com to let us know so we can fix and rerun the tests for you.
FAILURE - Could not import required module.
This error probably shouldn't happen as Jenkins should catch it in the unit
tests before Turbo-Hipster launches. If you see this, please contact us at
rcbau@rcbops.com and let us know.
If you receive an error that you think is a false positive, leave a comment
on the review with the sole contents of "recheck migrations".
If you have any questions/problems please contact us at rcbau@rcbops.com.

152
doc/source/running.rst
View File

@ -4,4 +4,154 @@
Running
=======
todo...
Starting turbo-hipster
----------------------
Turbo-hipster can be run from the command line::
$ ./turbo-hipster/worker_server.py
This option allows you to pass parameters to turbo-hipster. Use the --help
parameter to see a full list.
+-------+--------------+--------------------------------------------------------+
| Short | Long | Description |
+=======+==============+========================================================+
| -c | --config | Print the path to the configuration file and exit |
+-------+--------------+--------------------------------------------------------+
| -b | --background | Run as a daemon in the background |
+-------+--------------+--------------------------------------------------------+
| -p | --pidfile | Specify the PID file to lock while running as a daemon |
+-------+--------------+--------------------------------------------------------+
Alternatively, you can start turbo-hipster as a service.
1. Copy the turbo-hipster init.d script to /etc/init.d/:
$ sudo cp etc/init.d/turbo-hipster /etc/init.d/
2. Reload the script with the default configuration:
$ sudo update-rc.d turbo-hipster defaults
3. Start the service:
$ sudo service turbo-hipster start
Plugins
=======
Plugins can be used to extend turbo-hipster's capabilities.
.. note::
Currently, the only available plugin for turbo-hipster is the
Database Migration plugin, ``gate_real_db_upgrade``, which tests code
against a variety of real-world databases.
Installing plugins
------------------
Turbo-hipster plugins are responsible for handling the jobs that are passed
to it. They must successfully build reports and publish them according to
their configuration. They must also be able to communicate test results back
to Zuul using Gearman.
Plugins must take a standard format in order to be able to work correctly
with turbo-hipster. They must contain a ``task.py`` file with a ``Runner``
class.
Once you have created a turbo-hipster plugin, you need to configure it in
the ``config.json`` configuration file.
.. FIXME More config information required here
Plugin: Database migration with ``gate_real_db_upgrade``
--------------------------------------------------------
The database migration plugin, ``gate_real_db_upgrade``, is used to test
datasets against real-world, anonymized, databases.
Migrating a database
--------------------
In order to use turbo-hipster with the ``gate_real_db_upgrade`` plugin, you
need to set up the databases to test against, and point to the plugin in
turbo-hipster's configuration file.
1. Create a directory for the datasets:
$ mkdir -p /var/lib/turbo-hipster/datasets
2. Copy the json dataset to the directory you created:
$ cp /my/dataset.json /var/lib/turbo-hipster/datasets/
3. Open the ``/etc/turbo-hipster/config.json`` file in your preferred
editor, locate the plugins section, and add this line::
**plugins**
gate_real_db_upgrade
Testing with turbo-hipster
==========================
When turbo-hipster completes a test, it sends the result of the test back to
Gearman. These results contain a link to a compiled logfile for the test.
If the test fails, or takes too long to complete, turbo-hipster will add a
review to your patchset that looks like this:
.. image:: ../images/THTestResult.png
Reading test reports
--------------------
An example of a standard log file:
http://thw01.rcbops.com/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log
An example of the same logfile, using the javascript logviewer:
http://thw01.rcbops.com/logviewer/?q=/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log
Test failure codes
------------------
This section gives a list of failure codes, including some steps you can
take for troubleshooting errors:
FAILURE - Did not find the end of a migration after a start
If you look at the log you should find that a migration began but never
finished. Hopefully there'll be a traceroute for you to follow through to
get some hints about why it failed.
WARNING - Migration %s took too long
In this case your migration took a long time to run against one of the test
datasets. You should reconsider what operations your migration is performing
and see if there are any optimizations you can make, or if it is really
necessary. If there is no way to speed up your migration you can email us at
rcbau@rcbops.com for an exception.
FAILURE - Final schema version does not match expectation
Somewhere along the line the migrations stopped and did not reach the
expected version. Our datasets start at previous releases and have to
upgrade all the way through to the most current release. If you see this,
inspect the log for traceroutes or other hints about the failure.
FAILURE - Could not setup seed database.
FAILURE - Could not find seed database.
These errors are internal errors. If you see either of these, contact us at
rcbau@rcbops.com to let us know so we can fix and rerun the tests for you.
FAILURE - Could not import required module.
This error probably shouldn't happen as Jenkins should catch it in the unit
tests before Turbo-Hipster launches. If you see this, please contact us at
rcbau@rcbops.com and let us know.
If you receive an error that you think is a false positive, leave a comment
on the review with the sole contents of "recheck migrations".
If you have any questions/problems please contact us at rcbau@rcbops.com.

21
doc/source/structure.rst
View File

@ -1,21 +0,0 @@
:title:Turbo Hipster Structure
Structure
=======================
Plugins
-------
A little bit about plugins to come here soon.
- You need to configure it in config.json
- Folder name must be the same as 'name'
- You probably want to specify a gate in the config
- see something (introduction?) about gates and how turbo-hipster
uses them
- Each plugin has a task.py with a Runner class
- Responsible for registering functions
- handling jobs
- checking for errors
- building reports
- publishing them according to configuration