x/turbo-hipster
Code Issues Proposed changes
A generic zuul worker using the gearman protocol.
189 Commits 1 Branch 0 Tags 1.9 MiB
Python 83%
Shell 17%
ffe7a2cf34
Go to file
Michael Still ffe7a2cf34 Better cost metrics. 
This review attempts to provide better cost metrics for the upgrade
steps. I do a few things:

 - restart mysql before the upgrades to flush innodb buffers and
   caches
 - run each migration as a separate job
 - log the innodb counters from mysql after each run

This will provide some first order metrics which are better than
wall time -- for example the number of written rows is probably
a more interesting metric than the number of seconds on a loaded
instance with a noisy neighbour.

We also need to do git pulls on stable branches, as zuul doesn't
do that for us.

Change-Id: I39e964a9bbe43717b5a5b5dfd236ee2fa2c62a00
2014-01-14 11:45:49 +11:00
datasets Regenerate historical data, add auto generated data config 2014-01-01 18:36:44 +11:00
doc Add unit tests for execute_to_log. 2014-01-10 14:09:10 +11:00
etc Better cost metrics. 2014-01-14 11:45:49 +11:00
tests Add unit tests for execute_to_log. 2014-01-10 14:09:10 +11:00
tools Script to upload old logs to swift 2013-12-12 14:21:20 +11:00
turbo_hipster Better cost metrics. 2014-01-14 11:45:49 +11:00
.gitignore Check the logs for migrations that take too long 2013-12-12 13:39:07 +11:00
.gitreview Add .gitreview file and fix documentation build 2013-11-21 10:31:33 +11:00
.testr.conf Move tests to root folder 2013-08-12 10:40:27 +10:00
LICENSE add in license 2013-08-12 23:25:35 +10:00
README.md More worker build documentation. 2013-12-29 22:02:20 +11:00
requirements.txt Update requirements for swift/keystone 2013-12-16 12:30:39 +11:00
results.json Regenerate historical data, add auto generated data config 2014-01-01 18:36:44 +11:00
setup.cfg Move extra commands into folder 2014-01-09 13:56:07 +08:00
setup.py update install requirements 2013-11-21 10:37:27 +11:00
test-requirements.txt Check the logs for migrations that take too long 2013-12-12 13:39:07 +11:00
tox.ini Add .gitreview file and fix documentation build 2013-11-21 10:31:33 +11:00

README.md

turbo-hipster

A set of CI tools.

worker_server.py is a worker server that loads and runs task_plugins.

Each task_plugin is a zuul gearman worker that implements, handles, executes a job, uploads/post-processes the logs and sends back the results to zuul.

Plugins

gate_real_db_upgrade: Runs the db_sync migrations on each dataset available in the datasets subdir.

Installation

  • boot a fresh Ubuntu image
  • setup ssh authentication for your admin team
  • apt-get update; apt-get dist-upgrade
  • adduser th
  • apt-get install vim git python-pip python-setuptools python-keystoneclient virtualenvwrapper python-eventlet python-numpy python-mysqldb python-git python-gitdb python-netaddr python-pkg-resources libxml2-dev libxml2-utils libxslt-dev git-review libxml2-dev libxml2-utils libxslt-dev libmysqlclient-dev pep8 postgresql-server-dev-9.1 python2.7-dev python-coverage python-netaddr
  • pip install -U pip
  • apt-get purge python-pip
  • cd /home/th; git clone http://github.com/stackforge/turbo-hipster
  • apply any patches you need
  • python setup.py install
  • cp turbo_hipster/task_plugins/gate_real_db_upgrade/*.sh /usr/local/lib/python2.7/dist-packages/turbo_hipster/task_plugins/gate_real_db_upgrade/
  • cp -R etc/* /etc/
  • mkdir /var/lib/turbo-hipster
  • chown -R th.th /var/lib/turbo-hipster
  • mkdir /var/log/turbo-hipster
  • chown -R th.th /var/log/turbo-hipster
  • install your chosen MySQL-like database engine (percona, maria, mysql)
  • mysql -u root --password=$1 -e "create user 'nova'@'localhost' identified by 'tester';"
  • mysql -u root --password=$1 -e "create user 'nova'@'172.16.0.2' identified by 'tester';"
  • mysql -u root --password=$1 -e "grant all privileges on . to 'nova'@'localhost' with grant option;"
  • mysql -u root --password=$1 -e "grant all privileges on . to 'nova'@'172.16.0.2' with grant option;"
  • /etc/rc.local
  • rsync the datasets from the master
  • logrotate -f /etc/logrotate.conf
  • chmod -R ugo+r /var/log/*
  • chmod ugo+rx /var/log/mysql
  • mkdir /var/cache/pip
  • chmod -R ugo+rwx /var/cache/pip
  • touch /var/log/mysql/slow-queries.log
  • /etc/init.d/turbo-hipster start