x/stacktach-sandbox
Code Issues Proposed changes

Fix up build script for stackforge repo rename.

Browse Source 
Change repo paths for stv3 compionents in build script.
Also fix docs for same.

Change-Id: If846f6c097b28f23c697e17bb50dbc33f7c390a8
This commit is contained in:
Monsyne Dragon 2015-10-29 18:43:52 +00:00
parent a069d95942
commit df47784421
7 changed files with 55 additions and 50 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -14,7 +14,7 @@ Prerequisites:
TL;DR:
handle the prerequisites above.
git clone https://github.com/stackforge/stacktach-sandbox
git clone https://github.com/openstack/stacktach-sandbox
cd sandbox
create a mysql database to use
set the database url appropriately in winchester.yaml

2
build.sh
View File

@ -60,7 +60,7 @@ for file in shoebox simport notigen notification-utils \
stackdistiller quincy quince timex \
klugman winchester
do
git clone http://git.openstack.org/stackforge/stacktach-$file
git clone http://git.openstack.org/openstack/stacktach-$file
done
# We still have some stragglers ...
for file in StackTach/notabene rackerlabs/yagi

4
docs/about.html
View File

@ -152,7 +152,7 @@ max_messages = 100
</pre>
<p>The important part of this configuration is the <code>[event_worker]</code> section. This says we want to use the RabbitMQ data source. The RabbitMQ connectivity information is stored in the <code>[rabbit_broker]</code> section. The name of each RabbitMQ queue to consume from is specified in the <code>[consumers]</code> section. For every queue you define there, you will need a <code>[consumer:&lt;queue_name&gt;]</code> section. This last section is where there real magic happens. Beyond defining the exchange, routing_key and durability characteristics, it defines the chain of <code>Yagi Handlers</code> that will run on every notification that gets consumed. </p>
<p>You can write your own Yagi handlers if you like, but there are a number that ship with StackTach.v3 to do some interesting things. The most important of these is the <a href='https://github.com/stackforge/stacktach-winchester/blob/4875e419a66974e416dbe3b43ed286017bad1ec4/winchester/yagi_handler.py#L18'>winchester.yagi_handler:WinchesterHandler</a>. This handler is your entry point into StackTach.v3 stream processing. But first, we need to convert those messy notifications into events ...</p>
<p>You can write your own Yagi handlers if you like, but there are a number that ship with StackTach.v3 to do some interesting things. The most important of these is the <a href='https://github.com/openstack/stacktach-winchester/blob/4875e419a66974e416dbe3b43ed286017bad1ec4/winchester/yagi_handler.py#L18'>winchester.yagi_handler:WinchesterHandler</a>. This handler is your entry point into StackTach.v3 stream processing. But first, we need to convert those messy notifications into events ...</p>
<h3><a id='distill'>Distilling Notifications to Events</a></h3>
<p>Now we have notifications coming into Winchester. But, as we hinted at above, we need to take the larger notification and <i>distill</i> it down into a, more manageable, event. The stack-distiller module makes this happen. Within StackTach.v3, this is part of <code>winchester.yagi_handler:WinchesterHandler</code>.</p>
<p>A notification is a large, nested JSON data structure. But we don't need all of that data for stream processing. In fact, we generally only require a few <a href='glossary.html#trait'>Traits</a> from the notification. That's what distilling does. It pulls out the important traits, scrubs the data and uses that. Distillations are done via the distillation configuration file (specified in winchester.conf). </p>
@ -231,7 +231,7 @@ max_messages = 100
- exception.kwargs.uuid
- instance.uuid
</pre>
<p>For details on the distiller configuration grammar, see the associated <a href='https://github.com/stackforge/stacktach-stackdistiller/blob/master/doc/event_definitions_config.md'>docs</a>. Note that the distiller config file is in YAML format.</p>
<p>For details on the distiller configuration grammar, see the associated <a href='https://github.com/openstack/stacktach-stackdistiller/blob/master/doc/event_definitions_config.md'>docs</a>. Note that the distiller config file is in YAML format.</p>
<p>When processed, the notification would be converted into an event that looks like this:</p>
<span class="label label-default">A sample event</span>

4
docs/contribute.html
View File

@ -58,9 +58,9 @@
<div class="col-lg-12">
<h3>Contributing to StackTach.v3</h3>
<p>StackTach.v3 is licensed under the Apache 2.0 license</p>
<p>All the source repos for StackTach.v3 (and .v2) are available on <a href='https://github.com/stackforge?query=stacktach'>StackForge</a>. Details on contributing to StackForge projects are available <a href='https://wiki.openstack.org/wiki/How_To_Contribute'>here</a></p>
<p>All the source repos for StackTach.v3 (and .v2) are available on <a href='https://github.com/openstack?query=stacktach'>StackForge</a>. Details on contributing to StackForge projects are available <a href='https://wiki.openstack.org/wiki/How_To_Contribute'>here</a></p>
<p>The core developers are available on Freenode IRC in the <code>#stacktach</code> channel</p>
<p>These docs are available in the <a href='https://github.com/stackforge/stacktach-sandbox/tree/master/docs'>Sandbox</a> repo. Patches welcome!</p>
<p>These docs are available in the <a href='https://github.com/openstack/stacktach-sandbox/tree/master/docs'>Sandbox</a> repo. Patches welcome!</p>
<footer class="footer">
<p>&copy; Dark Secret Software Inc. 2014</p>

6
docs/install.html
View File

@ -65,7 +65,7 @@
<li>a yagi-event process to read the notifications and get them into winchester</li>
<li>a winchester pipeline-worker process to do processing on ready streams</li>
</ul>
<p>If you have a machine with MySQL and RabbitMQ on it, the <a href='https://github.com/stackforge/stacktach-sandbox' target='_blank'>stacktach-sandbox</a> repo does the rest. The sandbox will set up a python virtual environment, install all the required components and run everything under a <a href="http://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/" target="_blank">screen</a> session.</p>
<p>If you have a machine with MySQL and RabbitMQ on it, the <a href='https://github.com/openstack/stacktach-sandbox' target='_blank'>stacktach-sandbox</a> repo does the rest. The sandbox will set up a python virtual environment, install all the required components and run everything under a <a href="http://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/" target="_blank">screen</a> session.</p>
<img src='sandbox-1.gif' class="img-rounded"/>
<p>The sandbox sets up seven screen pages that contain:</p>
<ol type="a">
@ -106,7 +106,7 @@
<p>In this deployment scenario, we have a set of "worker" boxes that run the <code>yagi-event</code> workers (to consume notifications from RabbitMQ). We run one yagi-event worker per OpenStack cell. We also run the Winchester pipeline-workers on this box. The pipeline-worker manages pipeline processing for ready streams. We run as many pipeline-workers as needed to keep each core busy. Both the yagi-event and pipeline-worker processes are single threaded so we spawn more to keep the host busy. We add more servers like these to handle the load in the pipeline processing and/or the source queues.</p>
<p>Additionally, we have servers for the API nodes and the MySQL database. Load balancing the API servers and making MySQL highly available is left to the deployer.</p>
<h3>The Ansible deployment scripts</h3>
<p>In the <a href='https://github.com/stackforge/stacktach-sandbox/tree/master/ansible'>StackTach Sandbox</a> repo, there is an ansible directory that contains roles for the workers, the API nodes and the database. There is a fourth "common" role that is used by the other roles.</p>
<p>In the <a href='https://github.com/openstack/stacktach-sandbox/tree/master/ansible'>StackTach Sandbox</a> repo, there is an ansible directory that contains roles for the workers, the API nodes and the database. There is a fourth "common" role that is used by the other roles.</p>
<p>These scripts assume a multi-cell deployment where there is one StackTach.v3 deployment per region (each with it's own API servers and database). If you are not running with cells, pretend it's a single cell.</p>
<p>The servers are deployed in the following way:</p>
@ -138,7 +138,7 @@ cd /opt/stv3/bin
./yagi-event --config /etc/stv3/yagi-cell-1.conf
</pre>
<h3>The Build script</h3>
<p>The <a href='https://github.com/stackforge/stacktach-sandbox'>StackTach Sandbox</a> repo is the umbrella repo for all the other StackTach modules. It:</p>
<p>The <a href='https://github.com/openstack/stacktach-sandbox'>StackTach Sandbox</a> repo is the umbrella repo for all the other StackTach modules. It:</p>
<ul>
<li>can create a local dev environment for anyone wanting to contribute to StackTach or just play around with it
<li>can package StackTach into a python virtual environment and tarball it up for larger scale deployment. The ansible scripts use this tarball approach.

85
pip_freeze_versions.txt
View File

@ -1,58 +1,63 @@
BeautifulSoup==3.2.1
Mako==1.0.1
MarkupSafe==0.23
MySQL-python==1.2.5
PubSubHubbub-Publisher==1.0
PyYAML==3.11
Pygments==2.0.2
Routes==2.1
SQLAlchemy==1.0.2
WebOb==1.4.1
alembic==0.7.5.post2
amqp==1.4.6
alembic==0.8.3
amqp==1.4.7
amqplib==1.0.2
anyjson==0.3.3
argparse==1.2.1
BeautifulSoup==3.2.1
carrot==0.10.7
dateutils==0.6.6
decorator==3.4.2
decorator==4.0.4
docopt==0.6.2
docutils==0.12
enum34==1.0.4
eventlet==0.17.3
falcon==0.2
eventlet==0.17.4
falcon==0.3.0
feedgenerator==1.7
greenlet==0.4.5
flake8==2.2.4
greenlet==0.4.9
gunicorn==19.3.0
hacking==0.10.2
httpie==0.9.2
httplib2==0.9.1
httplib2==0.9.2
iso8601==0.1.10
jsonpath-rw==1.4.0
klugman==0.5.dev24
kombu==3.0.26
klugman==0.5.0.dev25
kombu==3.0.29
librabbitmq==1.6.1
lockfile==0.10.2
mysql-connector-python==2.0.3
notabene==0.2.dev31
notification-utils==0.3.dev6
notigen==0.3.dev36
pbr==0.10.8
ply==3.6
lockfile==0.11.0
Mako==1.0.3
MarkupSafe==0.23
mccabe==0.2.1
mysql-connector-python==2.0.4
MySQL-python==1.2.5
notabene==0.2.0.dev30
notification-utils==0.3.0.dev7
notigen==0.3.0.dev35
pbr==0.11.0
pep8==1.5.7
ply==3.8
prettytable==0.7.2
python-daemon==2.0.5
PubSubHubbub-Publisher==1.0
pyflakes==0.8.1
Pygments==2.0.2
python-daemon==2.0.6
python-dateutil==2.4.2
python-editor==0.4
python-mimeparse==0.1.4
pytz==2015.2
quince==0.3.dev10
quincy==0.3.dev24
pytz==2015.7
PyYAML==3.11
quince==0.3.0.dev12
quincy==0.3.0.dev25
redis==2.10.3
repoze.lru==0.6
requests==2.6.2
shoebox==0.7.dev53
simport==0.2.dev16
six==1.9.0
stackdistiller==0.11
timex==0.20.0
winchester==0.55.dev34
wsgiref==0.1.2
yagi==0.23
requests==2.8.1
Routes==2.2
shoebox==0.7.0.dev52
simport==0.3.0.dev18
six==1.10.0
SQLAlchemy==1.0.9
stackdistiller==0.13.0.dev10
timex==0.22.0.dev6
WebOb==1.5.0
wheel==0.24.0
winchester==0.62.0.dev42
yagi==0.24

2
vagrant/Vagrantfile vendored
View File

@ -4,5 +4,5 @@
Vagrant.configure(2) do |config|
config.vm.box = "ubuntu/trusty64"
config.vm.provision "shell", path: "bootstrap.sh"
config.vm.provision "shell", inline: "git clone https://github.com/stackforge/stacktach-sandbox.git", privileged: false
config.vm.provision "shell", inline: "git clone https://github.com/openstack/stacktach-sandbox.git", privileged: false
end