From ec7637a767ed959283f017949188c32e04ac749a Mon Sep 17 00:00:00 2001 From: Lana Brindley Date: Fri, 17 Jan 2014 13:32:17 +1000 Subject: [PATCH] Restructured documentation Moved information out of intro.rst and into more specific doc files. Reorganised index.rst accordingly, and deleted unused files. Change-Id: I660c5e5e597baddbd1d2d5f60871687b5306834c --- doc/source/configuration.rst | 10 -- doc/source/index.rst | 2 - doc/source/installation.rst | 209 ++++++++++++---------------- doc/source/intro.rst | 256 ----------------------------------- doc/source/running.rst | 152 ++++++++++++++++++++- doc/source/structure.rst | 21 --- 6 files changed, 240 insertions(+), 410 deletions(-) delete mode 100644 doc/source/configuration.rst delete mode 100644 doc/source/structure.rst diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst deleted file mode 100644 index 5bba790..0000000 --- a/doc/source/configuration.rst +++ /dev/null @@ -1,10 +0,0 @@ -:title: Configuration - - -Configuration -============= - -It is required that debug logging be configured for Turbo Hispter, as we use -the python logging framework to capture log messages from the task plugin -code. Configuration is as simple as setting the debug_log configuration -setting in config.json. diff --git a/doc/source/index.rst b/doc/source/index.rst index 25c0f30..c9ebbd0 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -12,9 +12,7 @@ Contents: :maxdepth: 2 intro - structure installation - configuration running diff --git a/doc/source/installation.rst b/doc/source/installation.rst index a0a0732..7c3de7f 100644 --- a/doc/source/installation.rst +++ b/doc/source/installation.rst @@ -6,142 +6,111 @@ Installation ============ -Download --------- +Turbo-hipster is installed directly into your Python ``site-packages`` +directory, and is then run as a service. It is managed using a configuration +file, which is in .json format. -Grab a copy from https://github.com/rcbau/turbo-hipster +Installing turbo-hipster +------------------------ +1. Turbo-Hipster can be installed directly to your Python ``site-packages`` +directory:: -Install -------- + $ sudo python setup.py install -turbo-hipster is configured to use setup tools for installation if -you would like to install it to your site-packages use:: +2. Copy the configuration file to a convenient location. By default, +turbo-hipster will look in ``/etc/turbo-hipster/config.json`` :: - sudo python setup.py install + $ cp -R etc/turbo-hipster /etc/ +3. The Turbo-Hipster configuration file is in .json format. Open the +``config.json`` configuration file in your preferred editor and modify it +for your environment:: -Copy config ------------ + **zuul_server** + A dictionary containing details about how to communicate + with zuul + **git_url** + The publicly accessible protocol and URI from where + to clone projects and zuul_ references from. For + example:: + http://review.openstack.org/p/ + or:: + git://review.example.org + **gearman_host** + The host of gearman_. zuul talks to its workers via + the gearman protocol and while it comes with a built- + in gearman server you can use a separate one. + **gearman_port** + The port that gearman listens on. + **debug_log** + A path to the debug log. Turbo-hipster will attempt to create + the file but must have write permissions. + **jobs_working_dir** + Each job will likely need to write out log and debug + information. This defines where turbo-hipster will do that. + **git_working_dir** + turbo-hipster needs to take a copy of the git tree of a + project to work from. This is the path it'll clone into and + work from (if needed). + **pip_download_cache** + Some of turbo-hipsters task plugins download requirements + for projects. This is the cache directory used by pip. + **plugins** + A list of enabled plugins and their settings in a dictionary. + The only required parameters are *name*, which should be the + same as the folder containing the plugin module, and + *function*, which is the function registered with zuul. + Any other parameters are specified by the plugin themselves + as required. + **publish_logs** + Log results from plugins can be published using multiple + methods. Currently only a local copy is fully implemented. + **type** + The type of protocol to copy the log to. eg 'local' + **path** + A type specific parameter defining the local location + destination. + **prepend_url** + What to prepend to the path when sending the result + URL back to zuul. This can be useful as you may want + to use a script to authenticate against a swift + account or to use *laughing_spice* to format the logs + etc. -Place the configuration where you are comfortable managing it. For -example:: +4. Create a turbo-hipster user: - cp -R etc/turbo-hipster /etc/ + $ useradd turbo-hipster +5. Create the directories listed in the configuration file, and give the +``turbo-hipster`` user write access:: -Edit config ------------ + $ mkdir -p /var/log/turbo-hipster/ + $ chown turbo-hipster:turbo-hipster /var/log/turbo-hipster/ -Turbo-hipsters configuration is currently stored in json format. -Modify the config.json appropriately:: + $ mkdir -p /var/lib/turbo-hipster/jobs + $ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/jobs - **zuul_server** - A dictionary containing details about how to communicate - with zuul - **git_url** - The publicly accessible protocol and URI from where - to clone projects and zuul_ references from. For - example:: - http://review.openstack.org/p/ - or:: - git://review.example.org - **gearman_host** - The host of gearman_. zuul talks to its workers via - the gearman protocol and while it comes with a built- - in gearman server you can use a separate one. - **gearman_port** - The port that gearman listens on. - **debug_log** - A path to the debug log. Turbo-hipster will attempt to create - the file but must have write permissions. - **jobs_working_dir** - Each job will likely need to write out log and debug - information. This defines where turbo-hipster will do that. - **git_working_dir** - turbo-hipster needs to take a copy of the git tree of a - project to work from. This is the path it'll clone into and - work from (if needed). - **pip_download_cache** - Some of turbo-hipsters task plugins download requirements - for projects. This is the cache directory used by pip. - **plugins** - A list of enabled plugins and their settings in a dictionary. - The only required parameters are *name*, which should be the - same as the folder containing the plugin module, and - *function*, which is the function registered with zuul. - Any other parameters are specified by the plugin themselves - as required. - **publish_logs** - Log results from plugins can be published using multiple - methods. Currently only a local copy is fully implemented. - **type** - The type of protocol to copy the log to. eg 'local' - **path** - A type specific parameter defining the local location - destination. - **prepend_url** - What to prepend to the path when sending the result - URL back to zuul. This can be useful as you may want - to use a script to authenticate against a swift - account or to use *laughing_spice* to format the logs - etc. + $ mkdir -p /var/lib/turbo-hipster/git + $ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/git + $ mkdir -p /var/cache/pip + $ chown turbo-hipster:turbo-hipster /var/cache/pip -Set up ------- +6. Open the MySQL log rotation configuration file in your preferred text +editor, and edit it to ensure it is writable by ``other``:: -You probably want to create a user to run turbo-hipster as. This user -will need to write to all of the directories specified in -config.json. + $ vim /etc/logrotate.d/mysql-server + # edit create 640 to 644. -Make sure the required directories as defined by the config.json -exist and are writeable by your turbo-hipster user:: +.. note:: + The turbo-hipster source code is also available for download from + the `turbo-hipster github page `_ - mkdir -p /var/log/turbo-hipster/ - chown turbo-hipster:turbo-hipster /var/log/turbo-hipster/ + $ git clone https://github.com/rcbau/turbo-hipster - mkdir -p /var/lib/turbo-hipster/jobs - chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/jobs - - mkdir -p /var/lib/turbo-hipster/git - chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/git - - mkdir -p /var/cache/pip - chown turbo-hipster:turbo-hipster /var/cache/pip - -Edit MySQL's log rotate to ensure it is other writable:: - - vim /etc/logrotate.d/mysql-server - # edit create 640 to 644. - - -Start turbo-hipster -------------------- - -turbo hipster can be ran by executing:: - - ./turbo-hipster/worker_server.py - -and optionally takes the following parameters:: - - ./turbo_hipster/worker_server.py --help - usage: worker_server.py [-h] [-c CONFIG] [-b] [-p PIDFILE] - - optional arguments: - -h, --help show this help message and exit - -c CONFIG, --config CONFIG - Path to json config file. - -b, --background Run as a daemon in the background. - -p PIDFILE, --pidfile PIDFILE - PID file to lock during daemonization. - -By default turbo-hipster will look for -*/etc/turbo-hipster/config.json* - -Alternatively turbo-hipster can be launched by init.d using the -included etc/init.d/turbo-hipster script:: - - sudo cp etc/init.d/turbo-hipster /etc/init.d/ - sudo update-rc.d turbo-hipster defaults - sudo service turbo-hipster start +.. note:: + Debug logging must be configured for turbo-hipster, as it uses the Python + logging framework to capture log messages from the task plugin code. + To configure debug logging, set the ``debug_log`` configuration + setting in the ``config.json`` configuration file. \ No newline at end of file diff --git a/doc/source/intro.rst b/doc/source/intro.rst index 5ec60bc..d2c0045 100644 --- a/doc/source/intro.rst +++ b/doc/source/intro.rst @@ -92,260 +92,4 @@ Typical workflow diagram } -Installation -============ -Turbo-hipster is installed directly into your Python ``site-packages`` -directory, and is then run as a service. It is managed using a configuration -file, which is in .json format. - -Installing turbo-hipster ------------------------- - -1. Turbo-Hipster can be installed directly to your Python ``site-packages`` -directory:: - - $ sudo python setup.py install - -2. Copy the configuration file to a convenient location. By default, -turbo-hipster will look in ``/etc/turbo-hipster/config.json`` :: - - $ cp -R etc/turbo-hipster /etc/ - -3. The Turbo-Hipster configuration file is in .json format. Open the -``config.json`` configuration file in your preferred editor and modify it -for your environment:: - - **zuul_server** - A dictionary containing details about how to communicate - with zuul - **git_url** - The publicly accessible protocol and URI from where - to clone projects and zuul_ references from. For - example:: - http://review.openstack.org/p/ - or:: - git://review.example.org - **gearman_host** - The host of gearman_. zuul talks to its workers via - the gearman protocol and while it comes with a built- - in gearman server you can use a separate one. - **gearman_port** - The port that gearman listens on. - **debug_log** - A path to the debug log. Turbo-hipster will attempt to create - the file but must have write permissions. - **jobs_working_dir** - Each job will likely need to write out log and debug - information. This defines where turbo-hipster will do that. - **git_working_dir** - turbo-hipster needs to take a copy of the git tree of a - project to work from. This is the path it'll clone into and - work from (if needed). - **pip_download_cache** - Some of turbo-hipsters task plugins download requirements - for projects. This is the cache directory used by pip. - **plugins** - A list of enabled plugins and their settings in a dictionary. - The only required parameters are *name*, which should be the - same as the folder containing the plugin module, and - *function*, which is the function registered with zuul. - Any other parameters are specified by the plugin themselves - as required. - **publish_logs** - Log results from plugins can be published using multiple - methods. Currently only a local copy is fully implemented. - **type** - The type of protocol to copy the log to. eg 'local' - **path** - A type specific parameter defining the local location - destination. - **prepend_url** - What to prepend to the path when sending the result - URL back to zuul. This can be useful as you may want - to use a script to authenticate against a swift - account or to use *laughing_spice* to format the logs - etc. - -4. Create a turbo-hipster user: - - $ useradd turbo-hipster - -5. Create the directories listed in the configuration file, and give the -``turbo-hipster`` user write access:: - - $ mkdir -p /var/log/turbo-hipster/ - $ chown turbo-hipster:turbo-hipster /var/log/turbo-hipster/ - - $ mkdir -p /var/lib/turbo-hipster/jobs - $ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/jobs - - $ mkdir -p /var/lib/turbo-hipster/git - $ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/git - - $ mkdir -p /var/cache/pip - $ chown turbo-hipster:turbo-hipster /var/cache/pip - -6. Open the MySQL log rotation configuration file in your preferred text -editor, and edit it to ensure it is writable by ``other``:: - - $ vim /etc/logrotate.d/mysql-server - # edit create 640 to 644. - -.. note:: - The turbo-hipster source code is also available for download from - the `turbo-hipster github page `_ - - $ git clone https://github.com/rcbau/turbo-hipster - -Starting turbo-hipster ----------------------- - -Turbo-hipster can be run from the command line:: - - $ ./turbo-hipster/worker_server.py - -This option allows you to pass parameters to turbo-hipster. Use the --help -parameter to see a full list. - -+-------+--------------+--------------------------------------------------------+ -| Short | Long | Description | -+=======+==============+========================================================+ -| -c | --config | Print the path to the configuration file and exit | -+-------+--------------+--------------------------------------------------------+ -| -b | --background | Run as a daemon in the background | -+-------+--------------+--------------------------------------------------------+ -| -p | --pidfile | Specify the PID file to lock while running as a daemon | -+-------+--------------+--------------------------------------------------------+ - -Alternatively, you can start turbo-hipster as a service. - -1. Copy the turbo-hipster init.d script to /etc/init.d/: - - $ sudo cp etc/init.d/turbo-hipster /etc/init.d/ - -2. Reload the script with the default configuration: - - $ sudo update-rc.d turbo-hipster defaults - -3. Start the service: - - $ sudo service turbo-hipster start - -Plugins -======= - -Plugins can be used to extend turbo-hipster's capabilities. - -.. note:: - Currently, the only available plugin for turbo-hipster is the - Database Migration plugin, ``gate_real_db_upgrade``, which tests code - against a variety of real-world databases. - -Installing plugins ------------------- - -Turbo-hipster plugins are responsible for handling the jobs that are passed -to it. They must successfully build reports and publish them according to -their configuration. They must also be able to communicate test results back -to Zuul using Gearman. - -Plugins must take a standard format in order to be able to work correctly -with turbo-hipster. They must contain a ``task.py`` file with a ``Runner`` -class. - -Once you have created a turbo-hipster plugin, you need to configure it in -the ``config.json`` configuration file. - -.. FIXME More config information required here - -Plugin: Database migration with ``gate_real_db_upgrade`` --------------------------------------------------------- - -The database migration plugin, ``gate_real_db_upgrade``, is used to test -datasets against real-world, anonymized, databases. - -Migrating a database --------------------- - -In order to use turbo-hipster with the ``gate_real_db_upgrade`` plugin, you -need to set up the databases to test against, and point to the plugin in -turbo-hipster's configuration file. - -1. Create a directory for the datasets: - - $ mkdir -p /var/lib/turbo-hipster/datasets - -2. Copy the json dataset to the directory you created: - - $ cp /my/dataset.json /var/lib/turbo-hipster/datasets/ - -3. Open the ``/etc/turbo-hipster/config.json`` file in your preferred -editor, locate the plugins section, and add this line:: - - **plugins** - gate_real_db_upgrade - -Testing with turbo-hipster -========================== - -When turbo-hipster completes a test, it sends the result of the test back to -Gearman. These results contain a link to a compiled logfile for the test. - -If the test fails, or takes too long to complete, turbo-hipster will add a -review to your patchset that looks like this: - -.. image:: ../images/THTestResult.png - -Reading test reports --------------------- - -An example of a standard log file: -http://thw01.rcbops.com/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log - -An example of the same logfile, using the javascript logviewer: -http://thw01.rcbops.com/logviewer/?q=/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log - -Test failure codes ------------------- - -This section gives a list of failure codes, including some steps you can -take for troubleshooting errors: - - FAILURE - Did not find the end of a migration after a start - -If you look at the log you should find that a migration began but never -finished. Hopefully there'll be a traceroute for you to follow through to -get some hints about why it failed. - - WARNING - Migration %s took too long - -In this case your migration took a long time to run against one of the test -datasets. You should reconsider what operations your migration is performing -and see if there are any optimizations you can make, or if it is really -necessary. If there is no way to speed up your migration you can email us at -rcbau@rcbops.com for an exception. - - FAILURE - Final schema version does not match expectation - -Somewhere along the line the migrations stopped and did not reach the -expected version. Our datasets start at previous releases and have to -upgrade all the way through to the most current release. If you see this, -inspect the log for traceroutes or other hints about the failure. - - FAILURE - Could not setup seed database. - FAILURE - Could not find seed database. - -These errors are internal errors. If you see either of these, contact us at -rcbau@rcbops.com to let us know so we can fix and rerun the tests for you. - - FAILURE - Could not import required module. - -This error probably shouldn't happen as Jenkins should catch it in the unit -tests before Turbo-Hipster launches. If you see this, please contact us at -rcbau@rcbops.com and let us know. - -If you receive an error that you think is a false positive, leave a comment -on the review with the sole contents of "recheck migrations". - -If you have any questions/problems please contact us at rcbau@rcbops.com. diff --git a/doc/source/running.rst b/doc/source/running.rst index b0903db..9a60686 100644 --- a/doc/source/running.rst +++ b/doc/source/running.rst @@ -4,4 +4,154 @@ Running ======= -todo... \ No newline at end of file +Starting turbo-hipster +---------------------- + +Turbo-hipster can be run from the command line:: + + $ ./turbo-hipster/worker_server.py + +This option allows you to pass parameters to turbo-hipster. Use the --help +parameter to see a full list. + ++-------+--------------+--------------------------------------------------------+ +| Short | Long | Description | ++=======+==============+========================================================+ +| -c | --config | Print the path to the configuration file and exit | ++-------+--------------+--------------------------------------------------------+ +| -b | --background | Run as a daemon in the background | ++-------+--------------+--------------------------------------------------------+ +| -p | --pidfile | Specify the PID file to lock while running as a daemon | ++-------+--------------+--------------------------------------------------------+ + +Alternatively, you can start turbo-hipster as a service. + +1. Copy the turbo-hipster init.d script to /etc/init.d/: + + $ sudo cp etc/init.d/turbo-hipster /etc/init.d/ + +2. Reload the script with the default configuration: + + $ sudo update-rc.d turbo-hipster defaults + +3. Start the service: + + $ sudo service turbo-hipster start + +Plugins +======= + +Plugins can be used to extend turbo-hipster's capabilities. + +.. note:: + Currently, the only available plugin for turbo-hipster is the + Database Migration plugin, ``gate_real_db_upgrade``, which tests code + against a variety of real-world databases. + +Installing plugins +------------------ + +Turbo-hipster plugins are responsible for handling the jobs that are passed +to it. They must successfully build reports and publish them according to +their configuration. They must also be able to communicate test results back +to Zuul using Gearman. + +Plugins must take a standard format in order to be able to work correctly +with turbo-hipster. They must contain a ``task.py`` file with a ``Runner`` +class. + +Once you have created a turbo-hipster plugin, you need to configure it in +the ``config.json`` configuration file. + +.. FIXME More config information required here + +Plugin: Database migration with ``gate_real_db_upgrade`` +-------------------------------------------------------- + +The database migration plugin, ``gate_real_db_upgrade``, is used to test +datasets against real-world, anonymized, databases. + +Migrating a database +-------------------- + +In order to use turbo-hipster with the ``gate_real_db_upgrade`` plugin, you +need to set up the databases to test against, and point to the plugin in +turbo-hipster's configuration file. + +1. Create a directory for the datasets: + + $ mkdir -p /var/lib/turbo-hipster/datasets + +2. Copy the json dataset to the directory you created: + + $ cp /my/dataset.json /var/lib/turbo-hipster/datasets/ + +3. Open the ``/etc/turbo-hipster/config.json`` file in your preferred +editor, locate the plugins section, and add this line:: + + **plugins** + gate_real_db_upgrade + +Testing with turbo-hipster +========================== + +When turbo-hipster completes a test, it sends the result of the test back to +Gearman. These results contain a link to a compiled logfile for the test. + +If the test fails, or takes too long to complete, turbo-hipster will add a +review to your patchset that looks like this: + +.. image:: ../images/THTestResult.png + +Reading test reports +-------------------- + +An example of a standard log file: +http://thw01.rcbops.com/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log + +An example of the same logfile, using the javascript logviewer: +http://thw01.rcbops.com/logviewer/?q=/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log + +Test failure codes +------------------ + +This section gives a list of failure codes, including some steps you can +take for troubleshooting errors: + + FAILURE - Did not find the end of a migration after a start + +If you look at the log you should find that a migration began but never +finished. Hopefully there'll be a traceroute for you to follow through to +get some hints about why it failed. + + WARNING - Migration %s took too long + +In this case your migration took a long time to run against one of the test +datasets. You should reconsider what operations your migration is performing +and see if there are any optimizations you can make, or if it is really +necessary. If there is no way to speed up your migration you can email us at +rcbau@rcbops.com for an exception. + + FAILURE - Final schema version does not match expectation + +Somewhere along the line the migrations stopped and did not reach the +expected version. Our datasets start at previous releases and have to +upgrade all the way through to the most current release. If you see this, +inspect the log for traceroutes or other hints about the failure. + + FAILURE - Could not setup seed database. + FAILURE - Could not find seed database. + +These errors are internal errors. If you see either of these, contact us at +rcbau@rcbops.com to let us know so we can fix and rerun the tests for you. + + FAILURE - Could not import required module. + +This error probably shouldn't happen as Jenkins should catch it in the unit +tests before Turbo-Hipster launches. If you see this, please contact us at +rcbau@rcbops.com and let us know. + +If you receive an error that you think is a false positive, leave a comment +on the review with the sole contents of "recheck migrations". + +If you have any questions/problems please contact us at rcbau@rcbops.com. diff --git a/doc/source/structure.rst b/doc/source/structure.rst deleted file mode 100644 index c203466..0000000 --- a/doc/source/structure.rst +++ /dev/null @@ -1,21 +0,0 @@ -:title:Turbo Hipster Structure - -Structure -======================= - -Plugins -------- - -A little bit about plugins to come here soon. - - - You need to configure it in config.json - - Folder name must be the same as 'name' - - You probably want to specify a gate in the config - - see something (introduction?) about gates and how turbo-hipster - uses them - - Each plugin has a task.py with a Runner class - - Responsible for registering functions - - handling jobs - - checking for errors - - building reports - - publishing them according to configuration