Added documentation
Added initial draft of turbo-hipster docs. Change-Id: I3820f3fc9ede52218b4c055678b14267b6c92e92
This commit is contained in:
Lana Brindley
parent
5efa05ccc4
commit
5a11bb3244
BIN
doc/images/THTestResult.png
Normal file
BIN
doc/images/THTestResult.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 45 KiB |
@ -1,21 +1,55 @@
|
||||
:title:Introduction
|
||||
|
||||
Turbo Hipster
|
||||
Turbo-hipster
|
||||
=============
|
||||
|
||||
Turbo Hipster is a gearman worker designed to run tests using zuul
|
||||
as the gearman client. It is primarily aimed at running openstack
|
||||
continuous integration tests against pre-existing datasets but can
|
||||
be used to automate any tests with zuul.
|
||||
Turbo-hipster works with the existing OpenStack code review system to
|
||||
implement testing-related plugins. Historically, whenever code has been
|
||||
written for Nova it has been tested against trivial datasets rather than
|
||||
real data. This can mean that when users run the updated code on their
|
||||
databases they can run into issues that were not found during testing. A
|
||||
variety of real-world databases have been collected, anonymized, and added
|
||||
to the database migration plugin used by turbo-hipster. Turbo-hipster is
|
||||
integrated into the existing code review system, and automatically runs
|
||||
tests against these larger test datasets. Turbo-hipster is specifically
|
||||
designed to flag issues where changes to the database schema may not work
|
||||
due to outliers in real datasets, and to identify situations where a
|
||||
migration may take an unreasonable amount of time against a large database.
|
||||
|
||||
Overview
|
||||
--------
|
||||
.. note::
|
||||
Database anonymity is important, and can be very time consuming.
|
||||
The databases used by turbo-hipster to test against are real-world databases
|
||||
that have been anonymized with a database anonymization tool called Fuzzy
|
||||
Happiness. Fuzzy Happiness takes markup in the sqlalchemy models file and
|
||||
uses that to decide what values to anonymize, and how to do so. This feature
|
||||
is still in development, and until it is complete turbo-hipster will not
|
||||
report back to Zuul automatically. See the Release Notes for more detail.
|
||||
|
||||
The zuul server receives jobs from a trigger requesting particular
|
||||
jobs/tests to be ran. Turbo Hipster is able to provide a worker for
|
||||
each of those jobs or a subset and report the success/failure back to
|
||||
zuul. zuul will then collate responses from multiple workers and
|
||||
build a report.
|
||||
Additionally, turbo-hipster has been designed to be extensible, so it is
|
||||
possible to write other plugins to expand its capabilities.
|
||||
|
||||
Turbo-hipster and Zuul
|
||||
----------------------
|
||||
|
||||
Turbo-hipster is a Gearman worker. Zuul provides arguments that turbo-
|
||||
hipster uses to check out the patch, perform the database testing, and then
|
||||
report back with success or failure. Zuul allows you to specify which jobs
|
||||
should be run against which projects. You can create a rule in Zuul for it
|
||||
to select jobs that require testing against a database. Turbo-hipster will
|
||||
then register as being able to complete that type of job. Gearman handles
|
||||
the connection between Zuul and Turbo-Hipster, recognizing when a job
|
||||
matches the rule, and passing it to turbo-hipster for testing. When turbo-
|
||||
hipster receives the patchset for the job, it creates a virtual environment
|
||||
to test it. The result of the test is sent back to Gearman as a json string,
|
||||
which contains links to compiled logfiles.
|
||||
|
||||
The simplified workflow for Turbo-Hipster:
|
||||
|
||||
1. Registers as a worker against Zuul's Gearman server
|
||||
2. Receives jobs from Zuul as they arrive
|
||||
3. Checks out the patchset
|
||||
4. Sets up a new virtual environment for testing
|
||||
5. Loads in a representative subset of the available datasets
|
||||
6. Runs the migration against each dataset, and checks the result
|
||||
7. Reports the results to Zuul, using the Gearman protocol
|
||||
|
||||
Typical workflow diagram
|
||||
------------------------
|
||||
@ -111,9 +145,260 @@ is mostly there... If you know graphviz please help!**
|
||||
|
||||
}
|
||||
|
||||
zuul integration
|
||||
----------------
|
||||
Installation
|
||||
============
|
||||
|
||||
Explain how zuul triggers builds and gates etc and how turbo-hipster
|
||||
responds to them. Most of this is in the zuul documentation so don't
|
||||
duplicate.
|
||||
Turbo-hipster is installed directly into your Python ``site-packages``
|
||||
directory, and is then run as a service. It is managed using a configuration
|
||||
file, which is in .json format.
|
||||
|
||||
Installing turbo-hipster
|
||||
------------------------
|
||||
|
||||
1. Turbo-Hipster can be installed directly to your Python ``site-packages``
|
||||
directory::
|
||||
|
||||
$ sudo python setup.py install
|
||||
|
||||
2. Copy the configuration file to a convenient location. By default,
|
||||
turbo-hipster will look in ``/etc/turbo-hipster/config.json`` ::
|
||||
|
||||
$ cp -R etc/turbo-hipster /etc/
|
||||
|
||||
3. The Turbo-Hipster configuration file is in .json format. Open the
|
||||
``config.json`` configuration file in your preferred editor and modify it
|
||||
for your environment::
|
||||
|
||||
**zuul_server**
|
||||
A dictionary containing details about how to communicate
|
||||
with zuul
|
||||
**git_url**
|
||||
The publicly accessible protocol and URI from where
|
||||
to clone projects and zuul_ references from. For
|
||||
example::
|
||||
http://review.openstack.org/p/
|
||||
or::
|
||||
git://review.example.org
|
||||
**gearman_host**
|
||||
The host of gearman_. zuul talks to its workers via
|
||||
the gearman protocol and while it comes with a built-
|
||||
in gearman server you can use a separate one.
|
||||
**gearman_port**
|
||||
The port that gearman listens on.
|
||||
**debug_log**
|
||||
A path to the debug log. Turbo-hipster will attempt to create
|
||||
the file but must have write permissions.
|
||||
**jobs_working_dir**
|
||||
Each job will likely need to write out log and debug
|
||||
information. This defines where turbo-hipster will do that.
|
||||
**git_working_dir**
|
||||
turbo-hipster needs to take a copy of the git tree of a
|
||||
project to work from. This is the path it'll clone into and
|
||||
work from (if needed).
|
||||
**pip_download_cache**
|
||||
Some of turbo-hipsters task plugins download requirements
|
||||
for projects. This is the cache directory used by pip.
|
||||
**plugins**
|
||||
A list of enabled plugins and their settings in a dictionary.
|
||||
The only required parameters are *name*, which should be the
|
||||
same as the folder containing the plugin module, and
|
||||
*function*, which is the function registered with zuul.
|
||||
Any other parameters are specified by the plugin themselves
|
||||
as required.
|
||||
**publish_logs**
|
||||
Log results from plugins can be published using multiple
|
||||
methods. Currently only a local copy is fully implemented.
|
||||
**type**
|
||||
The type of protocol to copy the log to. eg 'local'
|
||||
**path**
|
||||
A type specific parameter defining the local location
|
||||
destination.
|
||||
**prepend_url**
|
||||
What to prepend to the path when sending the result
|
||||
URL back to zuul. This can be useful as you may want
|
||||
to use a script to authenticate against a swift
|
||||
account or to use *laughing_spice* to format the logs
|
||||
etc.
|
||||
|
||||
4. Create a turbo-hipster user:
|
||||
|
||||
$ useradd turbo-hipster
|
||||
|
||||
5. Create the directories listed in the configuration file, and give the
|
||||
``turbo-hipster`` user write access::
|
||||
|
||||
$ mkdir -p /var/log/turbo-hipster/
|
||||
$ chown turbo-hipster:turbo-hipster /var/log/turbo-hipster/
|
||||
|
||||
$ mkdir -p /var/lib/turbo-hipster/jobs
|
||||
$ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/jobs
|
||||
|
||||
$ mkdir -p /var/lib/turbo-hipster/git
|
||||
$ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/git
|
||||
|
||||
$ mkdir -p /var/cache/pip
|
||||
$ chown turbo-hipster:turbo-hipster /var/cache/pip
|
||||
|
||||
6. Open the MySQL log rotation configuration file in your preferred text
|
||||
editor, and edit it to ensure it is writable by ``other``::
|
||||
|
||||
$ vim /etc/logrotate.d/mysql-server
|
||||
# edit create 640 to 644.
|
||||
|
||||
.. note::
|
||||
The turbo-hipster source code is also available for download from
|
||||
the `turbo-hipster github page <https://github.com/rcbau/turbo-hipster/>`_
|
||||
|
||||
$ git clone https://github.com/rcbau/turbo-hipster
|
||||
|
||||
Starting turbo-hipster
|
||||
----------------------
|
||||
|
||||
Turbo-hipster can be run from the command line::
|
||||
|
||||
$ ./turbo-hipster/worker_server.py
|
||||
|
||||
This option allows you to pass parameters to turbo-hipster. Use the --help
|
||||
parameter to see a full list.
|
||||
|
||||
+-------+--------------+--------------------------------------------------------+
|
||||
| Short | Long | Description |
|
||||
+=======+==============+========================================================+
|
||||
| -c | --config | Print the path to the configuration file and exit |
|
||||
+-------+--------------+--------------------------------------------------------+
|
||||
| -b | --background | Run as a daemon in the background |
|
||||
+-------+--------------+--------------------------------------------------------+
|
||||
| -p | --pidfile | Specify the PID file to lock while running as a daemon |
|
||||
+-------+--------------+--------------------------------------------------------+
|
||||
|
||||
Alternatively, you can start turbo-hipster as a service.
|
||||
|
||||
1. Copy the turbo-hipster init.d script to /etc/init.d/:
|
||||
|
||||
$ sudo cp etc/init.d/turbo-hipster /etc/init.d/
|
||||
|
||||
2. Reload the script with the default configuration:
|
||||
|
||||
$ sudo update-rc.d turbo-hipster defaults
|
||||
|
||||
3. Start the service:
|
||||
|
||||
$ sudo service turbo-hipster start
|
||||
|
||||
Plugins
|
||||
=======
|
||||
|
||||
Plugins can be used to extend turbo-hipster's capabilities.
|
||||
|
||||
.. note::
|
||||
Currently, the only available plugin for turbo-hipster is the
|
||||
Database Migration plugin, ``gate_real_db_upgrade``, which tests code
|
||||
against a variety of real-world databases.
|
||||
|
||||
Installing plugins
|
||||
------------------
|
||||
|
||||
Turbo-hipster plugins are responsible for handling the jobs that are passed
|
||||
to it. They must successfully build reports and publish them according to
|
||||
their configuration. They must also be able to communicate test results back
|
||||
to Zuul using Gearman.
|
||||
|
||||
Plugins must take a standard format in order to be able to work correctly
|
||||
with turbo-hipster. They must contain a ``task.py`` file with a ``Runner``
|
||||
class.
|
||||
|
||||
Once you have created a turbo-hipster plugin, you need to configure it in
|
||||
the ``config.json`` configuration file.
|
||||
|
||||
.. FIXME More config information required here
|
||||
|
||||
Plugin: Database migration with ``gate_real_db_upgrade``
|
||||
--------------------------------------------------------
|
||||
|
||||
The database migration plugin, ``gate_real_db_upgrade``, is used to test
|
||||
datasets against real-world, anonymized, databases.
|
||||
|
||||
Migrating a database
|
||||
--------------------
|
||||
|
||||
In order to use turbo-hipster with the ``gate_real_db_upgrade`` plugin, you
|
||||
need to set up the databases to test against, and point to the plugin in
|
||||
turbo-hipster's configuration file.
|
||||
|
||||
1. Create a directory for the datasets:
|
||||
|
||||
$ mkdir -p /var/lib/turbo-hipster/datasets
|
||||
|
||||
2. Copy the json dataset to the directory you created:
|
||||
|
||||
$ cp /my/dataset.json /var/lib/turbo-hipster/datasets/
|
||||
|
||||
3. Open the ``/etc/turbo-hipster/config.json`` file in your preferred
|
||||
editor, locate the plugins section, and add this line::
|
||||
|
||||
**plugins**
|
||||
gate_real_db_upgrade
|
||||
|
||||
Testing with turbo-hipster
|
||||
==========================
|
||||
|
||||
When turbo-hipster completes a test, it sends the result of the test back to
|
||||
Gearman. These results contain a link to a compiled logfile for the test.
|
||||
|
||||
If the test fails, or takes too long to complete, turbo-hipster will add a
|
||||
review to your patchset that looks like this:
|
||||
|
||||
.. image:: ../images/THTestResult.png
|
||||
|
||||
Reading test reports
|
||||
--------------------
|
||||
|
||||
An example of a standard log file:
|
||||
http://thw01.rcbops.com/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log
|
||||
|
||||
An example of the same logfile, using the javascript logviewer:
|
||||
http://thw01.rcbops.com/logviewer/?q=/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log
|
||||
|
||||
Test failure codes
|
||||
------------------
|
||||
|
||||
This section gives a list of failure codes, including some steps you can
|
||||
take for troubleshooting errors:
|
||||
|
||||
FAILURE - Did not find the end of a migration after a start
|
||||
|
||||
If you look at the log you should find that a migration began but never
|
||||
finished. Hopefully there'll be a traceroute for you to follow through to
|
||||
get some hints about why it failed.
|
||||
|
||||
WARNING - Migration %s took too long
|
||||
|
||||
In this case your migration took a long time to run against one of the test
|
||||
datasets. You should reconsider what operations your migration is performing
|
||||
and see if there are any optimizations you can make, or if it is really
|
||||
necessary. If there is no way to speed up your migration you can email us at
|
||||
rcbau@rcbops.com for an exception.
|
||||
|
||||
FAILURE - Final schema version does not match expectation
|
||||
|
||||
Somewhere along the line the migrations stopped and did not reach the
|
||||
expected version. Our datasets start at previous releases and have to
|
||||
upgrade all the way through to the most current release. If you see this,
|
||||
inspect the log for traceroutes or other hints about the failure.
|
||||
|
||||
FAILURE - Could not setup seed database.
|
||||
FAILURE - Could not find seed database.
|
||||
|
||||
These errors are internal errors. If you see either of these, contact us at
|
||||
rcbau@rcbops.com to let us know so we can fix and rerun the tests for you.
|
||||
|
||||
FAILURE - Could not import required module.
|
||||
|
||||
This error probably shouldn't happen as Jenkins should catch it in the unit
|
||||
tests before Turbo-Hipster launches. If you see this, please contact us at
|
||||
rcbau@rcbops.com and let us know.
|
||||
|
||||
If you receive an error that you think is a false positive, leave a comment
|
||||
on the review with the sole contents of "recheck migrations".
|
||||
|
||||
If you have any questions/problems please contact us at rcbau@rcbops.com.
|
||||
Loading…
x
Reference in New Issue
Block a user