x/turbo-hipster
Code Issues Proposed changes
A generic zuul worker using the gearman protocol.
195 Commits 1 Branch 0 Tags 1.9 MiB
Python 83%
Shell 17%
c972d2d1d7
Go to file
Michael Still c972d2d1d7 Set benchmarks for innodb counters. 
Use the currently collected innodb counters to decide what we
expect from existing migrations. I've also moved metrics collection
to a cloud db because json was a poor scaling choice.

Change-Id: I2ff0e4b58ac54f53a569ec3e67ad2fa753748dbf
2014-01-16 16:12:46 +11:00
datasets Set benchmarks for innodb counters. 2014-01-16 16:12:46 +11:00
doc Update docs diagram with sequence 2014-01-16 02:41:20 +00:00
etc Better cost metrics. 2014-01-14 11:45:49 +11:00
tests Handle the new format for migrations from the LogParser. 2014-01-15 13:26:50 +11:00
tools Script to upload old logs to swift 2013-12-12 14:21:20 +11:00
turbo_hipster Set benchmarks for innodb counters. 2014-01-16 16:12:46 +11:00
.gitignore Check the logs for migrations that take too long 2013-12-12 13:39:07 +11:00
.gitreview Add .gitreview file and fix documentation build 2013-11-21 10:31:33 +11:00
.testr.conf Move tests to root folder 2013-08-12 10:40:27 +10:00
LICENSE add in license 2013-08-12 23:25:35 +10:00
README.md More worker build documentation. 2013-12-29 22:02:20 +11:00
requirements.txt Update docs diagram with sequence 2014-01-16 02:41:20 +00:00
setup.cfg Move extra commands into folder 2014-01-09 13:56:07 +08:00
setup.py update install requirements 2013-11-21 10:37:27 +11:00
test-requirements.txt Check the logs for migrations that take too long 2013-12-12 13:39:07 +11:00
tox.ini Add .gitreview file and fix documentation build 2013-11-21 10:31:33 +11:00

README.md

turbo-hipster

A set of CI tools.

worker_server.py is a worker server that loads and runs task_plugins.

Each task_plugin is a zuul gearman worker that implements, handles, executes a job, uploads/post-processes the logs and sends back the results to zuul.

Plugins

gate_real_db_upgrade: Runs the db_sync migrations on each dataset available in the datasets subdir.

Installation

  • boot a fresh Ubuntu image
  • setup ssh authentication for your admin team
  • apt-get update; apt-get dist-upgrade
  • adduser th
  • apt-get install vim git python-pip python-setuptools python-keystoneclient virtualenvwrapper python-eventlet python-numpy python-mysqldb python-git python-gitdb python-netaddr python-pkg-resources libxml2-dev libxml2-utils libxslt-dev git-review libxml2-dev libxml2-utils libxslt-dev libmysqlclient-dev pep8 postgresql-server-dev-9.1 python2.7-dev python-coverage python-netaddr
  • pip install -U pip
  • apt-get purge python-pip
  • cd /home/th; git clone http://github.com/stackforge/turbo-hipster
  • apply any patches you need
  • python setup.py install
  • cp turbo_hipster/task_plugins/gate_real_db_upgrade/*.sh /usr/local/lib/python2.7/dist-packages/turbo_hipster/task_plugins/gate_real_db_upgrade/
  • cp -R etc/* /etc/
  • mkdir /var/lib/turbo-hipster
  • chown -R th.th /var/lib/turbo-hipster
  • mkdir /var/log/turbo-hipster
  • chown -R th.th /var/log/turbo-hipster
  • install your chosen MySQL-like database engine (percona, maria, mysql)
  • mysql -u root --password=$1 -e "create user 'nova'@'localhost' identified by 'tester';"
  • mysql -u root --password=$1 -e "create user 'nova'@'172.16.0.2' identified by 'tester';"
  • mysql -u root --password=$1 -e "grant all privileges on . to 'nova'@'localhost' with grant option;"
  • mysql -u root --password=$1 -e "grant all privileges on . to 'nova'@'172.16.0.2' with grant option;"
  • /etc/rc.local
  • rsync the datasets from the master
  • logrotate -f /etc/logrotate.conf
  • chmod -R ugo+r /var/log/*
  • chmod ugo+rx /var/log/mysql
  • mkdir /var/cache/pip
  • chmod -R ugo+rwx /var/cache/pip
  • touch /var/log/mysql/slow-queries.log
  • /etc/init.d/turbo-hipster start